Sorry about the long downtime everyone. What happened is sometime around 20:00 yesterday the instance went down. I wouldn’t be able to ssh into the server until the morning, so I tried restarting the server through our hosting’s web interface, but this didn’t work.
When I got home I found the reason Lemmy wasn’t working was because the database had suffered some corruption. Thankfully, the only lost data was 3 posts, but some data got duplicated in random tables that shouldn’t be and this wasn’t the easiest thing to track down.
Why did the server go down in the first place?
I don’t actually know the real reason, what I suspect happened is that the frontend got taken down by the rampant scraping we’ve been subject to. To put it into prospective, just look at the jump in the size of our web logs:
These are compressed plain text, like it’s unreal.
But back to the server, I think when I restarted the server to get everything back up, docker killed the database before it had shutdown properly hence the corruption.
Anyway, it’s sorted now. It’s going to be a bit quiet until we’ve caught up with the activities we’ve missed, so enjoy the quiet while you can.
Glad to see we are back. Those access logs are crazy.
I’m sure you will fine a lot of Chrome/12X, Chrome/13X and Chrome/14X user agents there. Seems to be the favourites of attackers/scrapers recently.
On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.
Thank you so much for the work you do!
Someone did actually reach out to my personal Mastodon account to ask what was happening.
Like florge said, there is a Matrix space that mostly serves this function, but that does require you to have a Matrix account so isn’t the best for public notices. It’d be nice to have a page (eg status.feddit.uk) where I could put messages about what’s happening. I’ll need to look to see if something like that exists.
The trouble with the matrix space is that when the site is down it’s pretty much impossible to find the details for it.
There’s a matrix space here