Sorry about the long downtime everyone. What happened is sometime around 20:00 yesterday the instance went down. I wouldn’t be able to ssh into the server until the morning, so I tried restarting the server through our hosting’s web interface, but this didn’t work.
When I got home I found the reason Lemmy wasn’t working was because the database had suffered some corruption. Thankfully, the only lost data was 3 posts, but some data got duplicated in random tables that shouldn’t be and this wasn’t the easiest thing to track down.
Why did the server go down in the first place?
I don’t actually know the real reason, what I suspect happened is that the frontend got taken down by the rampant scraping we’ve been subject to. To put it into prospective, just look at the jump in the size of our web logs:
These are compressed plain text, like it’s unreal.
But back to the server, I think when I restarted the server to get everything back up, docker killed the database before it had shutdown properly hence the corruption.
Anyway, it’s sorted now. It’s going to be a bit quiet until we’ve caught up with the activities we’ve missed, so enjoy the quiet while you can.
See this is why I hate working in the IT department, because it involves using computers, and they suck.
Yikes sounds like a nightmare. I hope you managed to get some sleep!
I did and it was surprisingly pretty good.
Glad to see we are back. Those access logs are crazy.
I’m sure you will fine a lot of Chrome/12X, Chrome/13X and Chrome/14X user agents there. Seems to be the favourites of attackers/scrapers recently.
On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.
Thank you so much for the work you do!
On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.
Someone did actually reach out to my personal Mastodon account to ask what was happening.
Like florge said, there is a Matrix space that mostly serves this function, but that does require you to have a Matrix account so isn’t the best for public notices. It’d be nice to have a page (eg status.feddit.uk) where I could put messages about what’s happening. I’ll need to look to see if something like that exists.
There’s a matrix space here
Thank you for your efforts to fix it!
Just wondering, is there a backup place where things like this can be communicated? Blue sky or mastodon for example?
There’s a matrix channel here: https://matrix.to/#/#feddit.uk:matrix.org
