• 0 Posts
  • 5 Comments
Joined 3 years ago
cake
Cake day: June 20th, 2023

help-circle
  • The unfortunate answer is because it’s cheaper to not give a shit. Sending a request and waiting for a timeout costs next to nothing, and scales linearly in terms of compute cost. The overwhelmed server on the other end slows exponentially with each concurrent request. The crawlers are set to maximize the efficiency of local resources, which include both wall-clock time and developer time. Why send one request at a time when your server can handle tens of thousands?

    Try x; wait 60 seconds, if fail: put on a list to try again later.

    Costs nothing to write and nothing to run. And if you own the hardware, and are paying for power already, may as well extract maximum dollar per watt.