Disclaimer: I don’t have a background in computer science
I recently heard about a lightweight opensource software called Anubis.
Anubis is designed to stop AI crawlers that download a lot of data to train artificial intelligence models
Several websites have deployed Anubis:
https://gitlab.gnome.org/GNOME
10 websites have deployed. 10 websites. Out of millions of websites.
My question is extremely simple.
If this software is so damn great, why isn’t it everywhere?
Seriously. Why isn’t it used on Lemmy? On Wikipedia? On CBC?
Yeah, that’s the dilemma. The more popular it gets, the more crawlers will be designed to circumvent it.