Here is a quick recap of what has happened during Dec 18th-22nd:
During Saturday, the stats database was getting overloaded for PC. This was both due to increased player counts and 3rd party stats websites pulling out stats.
This manifested itself in several ways:
* Sometimes the game would spend a really long time at the "updating stats" stage.
* Sometimes when people logged in, the game claimed that they were at rank 1.
* Sometimes the game would claim that they hadn't achieved anything (no pins/insignias etc).
One thing which is important to note here is that the stats for players have not been modified in any way - it is merely the game client that fails at retrieving them from the database.
We have blocked the main 3rd party stats websites, and are having a dialogue with their developers. Also, we have done modifications to the master server code to reduce the load that game clients cause on the stats DB. These changes were deployed during two master server restarts during Tuesday.
A bit later some networking issues caused something which we refer to as "ghost games" appearing in the server browser. What happened here was that 5-8% of all game servers fell offline, yet parts of the backend was under the impression that the servers were still connected. When the servers tried to reconnect, the backend would deny with the reason "there is already an identical server connected".
The swiftest way of resolving that is also the crudest - restarting the backend. We did so during Wednesday (a bit before lunchtime GMT).
At the time of writing, the backend is doing reasonably well and we are about to surpass 40k players online in less than an hour.
EDIT:
During evening Dec 22nd there are still reports about problems. These are likely due to packet loss that was occurring between two backbone locations in the US (one destination being near the master servers). The ranking DB itself held up fairly well during the night.
