Sorry about the long downtime everyone. What happened is sometime around 20:00 yesterday the instance went down. I wouldn’t be able to ssh into the server until the morning, so I tried restarting the server through our hosting’s web interface, but this didn’t work.
When I got home I found the reason Lemmy wasn’t working was because the database had suffered some corruption. Thankfully, the only lost data was 3 posts, but some data got duplicated in random tables that shouldn’t be and this wasn’t the easiest thing to track down.
Why did the server go down in the first place?
I don’t actually know the real reason, what I suspect happened is that the frontend got taken down by the rampant scraping we’ve been subject to. To put it into prospective, just look at the jump in the size of our web logs:
These are compressed plain text, like it’s unreal.
But back to the server, I think when I restarted the server to get everything back up, docker killed the database before it had shutdown properly hence the corruption.
Anyway, it’s sorted now. It’s going to be a bit quiet until we’ve caught up with the activities we’ve missed, so enjoy the quiet while you can.
Anubis doesn’t really do anything if the bot knows about it and doesn’t send the bits Anubis looks for. There are a few other heuristics that you can enable in the Anubis config, but it is a principal issue with how Anubis works. The general idea is that if a bot doesn’t send the things Anubis looks for, then it should be possible to distinguish it from regular browsers by that, but then you are back at a manual whack a mole game and given the extensive use of residential proxies these days it is very much a losing battle.
Anubis recently added a honeypot with automatic IP logging though that might help in combination with fail2ban. But in the end if scraper bot authors are already circumventing Anubis intentionally, it is probably also only a temporary respite.