irelephant [he/him]🍭@lemm.ee to Showerthoughts@lemmy.worldEnglish · 6 months agoRobots.txt is just a dni for scrapers.message-squaremessage-square6fedilinkarrow-up126arrow-down16
arrow-up120arrow-down1message-squareRobots.txt is just a dni for scrapers.irelephant [he/him]🍭@lemm.ee to Showerthoughts@lemmy.worldEnglish · 6 months agomessage-square6fedilink
Do not interact.