Resolved -
This incident has been resolved.
May 17, 05:36 UTC
Monitoring -
A proper fix has been implemented and we will closely monitor the status of the site over the coming hours.
Short postmortem:
- Computing title ratings site-wide is very expensive (mainly because of the usage of bayesian rating in this case),
- So this computation is done ahead of time, once per hour, and its result is stored in memory for quick access in the meantime,
- However the space we allocate in memory for this (& similar things) has grown and grown over time, and just today passed the maximum allocated total size for it, causing automatic eviction of some values from memory (Redis maxmem + lru eviction),
- When this value is missing, the site attempts to initialize it, and when it tries to do so multiple times per second it causes extreme strain on our databases, and in turn an outage
May 16, 19:04 UTC
Update -
We have identified the likely cause of the issue. It has been mitigated for now and the site was reopened.
May 16, 17:40 UTC
Identified -
We have identified the likely cause of the issue. It has been mitigated for now and the site was reopened.
May 16, 17:40 UTC
Investigating -
MangaDex is experiencing significant performance issues, we are investigating to find the root cause.
May 16, 16:34 UTC