My hypothesis is that those scrappers run on botnets hosted on infected servers, so they want to get the info as fast as possible, since you don’t know how long it will last before they are detected and removed.
Another option is that so many servers are scrapping the same web that one server may send only a request every a few seconds, but the whole fleet could be sending thousands of requests per second, some big websites can handle the load, but most small to medium size sites get overwhelmed.
My hypothesis is that those scrappers run on botnets hosted on infected servers, so they want to get the info as fast as possible, since you don’t know how long it will last before they are detected and removed.
Another option is that so many servers are scrapping the same web that one server may send only a request every a few seconds, but the whole fleet could be sending thousands of requests per second, some big websites can handle the load, but most small to medium size sites get overwhelmed.