I’d link to some blog posts about this an example, but the site they’re from went down a while ago.
At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host’s resources beyond being an ⊛ to webmasters?
Edit: Let me elaborate. A lot of the answers i’m seeing are just restating the problem without explaining why LLM scrapers are apparently all either coded by idiots or assholes. I know that they’re hitting sites with unreasonable numbers of requests, wasting bandwidth, and making tools like Anubis too important. I know that’s what’s happening. I’m asking why. What they gain from not being even a little intelligent about this.
With all the money and effort (and maybe even brainpower) going into this, surely there’s some explanation beyond incompetence.
I think many people tend to mistake any ddos for an AI scraper.
People are shitty. And some will just ddos you not for AI but trying to find vulnerabilities or just because they don’t like your site.
Deeper analysis of the attacks should be given to determine if they were scrapers in the sense that they were trying to get the information from your site, or they were just a ddos trying to take the site down, or maybe they were proving trying to find a vulnerability.
The latest is the one I get the most. I get petitions for over 400 paths in less than a second trying to find a vulnerable endpoint.
In these cases they do it so quick, that might provoke a ddos in some servers, because they know that they will get ban as soon as the log get processed, so they need to act quick.