Content warning: Sucidality (click or activate to expand post)
So… TL;DR: We lost all of your data created after 28th November 2021 at midnight… No, we probably can't get it back for you… Yes, I am terribly sorry, this affected us as well… a lot… anyway… Matrix data should be mostly recoverable, we hope, except for password resets and new accounts, I suppose… that would have to be recreated… also anything that isn't federated… and perhaps some things that are… Invidious and Plik data from after the date above are definitely gone… static webserver data is still there… and no, I don't think I need any lectures on data safety right now; we've learned our lesson…
What happened
We wanted to do some setup changes… move things off our RockPro64…
T.: Turns out Invidious n Synapse on the same machine's a bit much for one passively cooled RK3399, lmao.
So we got another RockPro64, just with eMMC, no SSD… so we would keep storage-intensive things (i.e. Synapse and Plik) on that machine… and move all the containers (Invidious, Libreddit, Cryptpad, and soon Nitter) to a different one… and move the static webserver to the 4GB Rock64, using that as a reverse proxy, with mkcert for internal network TLS certificates… well… we were at that… already had the static webserver running… then Synapse in between… then realized the SSD on the RockPro64 had a partition for /srv/www we didn't need anymore now… so… we deleted that… which wasn't a problem… what was a problem was later on trying resize.f2fs on the other (unmounted, accessed from a live system) partition… that ended up completely destroying the file system, fsck.f2fs didn't help either… well… wouldn't be much of an issue, thanks to our backups, we thought… if it hadn't turned out the last automatically run backups had happened end of November… at which point I got pretty suicidal, and, uh…
T.: I saved her (n us), yet again. Happens, Ig.
There were some more issues actually fixing the system in the end, getting the backups where we wanted them, and so on… hence why we're still up at this time… now… need to fix some inconsistencies in the Synapse DB, resulting from using that old DB and a lot of upgrades in between… once that's done… syncing will probably take ages; it already took maybe half an hour or so when it had been down just a few hours… this time it was effectively about two months… and while Synapse is doing that, we should also be able to bring up Invidious and Cryptpad again (Plik and Libreddit are already up)… Bots will take some more time; guess I'll post the link to this one manually… Also, nevermind, fixed the DB while writing this, so… Synapse is up, I guess?
What about the future?
Well… let's hope Matrix is actually robust enough to bring back messages again… apart from that… terribly sorry for your loss… but life does go on… We'll make sure backups are actually run regularly… and always do at least one manually when we mess around with storage… Mumble (on the 2GB Rock64) we shut down for good… might offer Jitsi instead, in the future… and as I said, Nitter might be a thing soon… but first of all, got to set up the Matrix bots again… See you…