Fyi the site has been back online for ~1h without issues. However we won't know if the lightweight fix is viable for another few more hours.
Posted Mar 24, 2024 - 10:02 UTC
Update
As a follow-up to the issues we had 2 weeks ago, we applied multiple fixes which got things mostly back to stability.
However, as we mentioned at the time, some more maintenance steps would be necessary to completely fix issues on our side. The last one we must do is however extremely time-consuming. So we are first attempting a more lightweight fix, hoping that it will be sufficient.
ETA is ~1h from the time of publication of this status update.
Posted Mar 24, 2024 - 07:06 UTC
Monitoring
We have now completed one of the most important steps and the site should be reasonably stable again. We still have a lot of work to do behind the scenes, but it should not be user impacting for a little while.
Posted Mar 11, 2024 - 00:23 UTC
Update
We were successful with the operation in our testing environment, so we will now begin the first step requiring a maintenance (again, yes). We expect it to be fairly short (<1h) but will give a more precise estimate when it is ongoing.
Posted Mar 10, 2024 - 22:34 UTC
Identified
We have decided on a course of action, and are currently trialing it in our testing environment. If all goes well, we will apply it to the live website, which will likely require at least 3 downtime windows. We will share more details once we begin that process.
Posted Mar 10, 2024 - 20:16 UTC
Update
We are tentatively bringing the site back online everywhere, however we do so without the forums, to help alleviate some of the current limitations. We are still working on the issue overall, but it will take longer for the complete solution.
Posted Mar 10, 2024 - 18:49 UTC
Update
While we work on the issue, some partial outage is necessary to avoid the site completely dying. We're therefore cutting off parts of the world in alternance to maintain things as we work on the problem.
Posted Mar 10, 2024 - 15:05 UTC
Investigating
Due to an ongoing problem with our database software, the site is experiencing degraded performance. We are investigating longer-term fixes, as the root cause is complex and in third-party software which we cannot reliably fix ourselves.