Server downtime, Sunday December 20th

Dear Generals,

many of you will have noticed, that the games were unreachable for several hours on Sunday morning.

Put shortly, the main problem was a database server which practically died, required a restart but did not get restarted in a timely manner. As you may have noticed, we had some similar stability issues during the past couple of weeks which resulted in sometimes bad performance and, sadly more than usual, even unplanned downtimes.

In the context of readying our servers for the release of the Supremacy 1914 500 player map and to cope with the added load of Call of War, we have significantly upgraded our hardware and worked a lot on the performance of the complete server backend. From this side, the servers are therefore now faster than ever.

The big and in the past time very noticeable problem we still have is that some core-parts of the system need hands-on work, when they go down. During normal office hours, we normally recognise problems like this within minutes and react very fast. For all other times, we have a contract with a service provider which is supposed to take care of this type of problems.

Sadly, this does not yet work reliably. We have tried to improve the situation with a very high priority since the beginning of October working with them to remove all false positives from our alerting so that it is very clear when there is the need to react to problems. But, as yesterday’s incident showed, we did not succeed in getting our cooperation with them to a level that is applicable for our games.

Our goal is of course to provide a reliable and fun gaming experience to all of you and that is why all devs in the team have now volunteered to step in 24/7 if a problem like this emerges again to get it resolved quicker. We will change to a new professional service provider for monitoring and maintaining our servers as quickly as possible and in the meantime we will try to react ourselves as fast as possible to any problems.

Here we would like to thank all our hard working moderators, admins and game operators. It is thanks to them that we were able to detect and handle many of the issues - it is them who inform us when something goes wrong and take the courage to wake up developers in the middle of the night to get stuff fixed. In return we will work on improving the tools they have to smoothen out their daily work.

We apologize for the problems you had to experience and want to let you know that we are giving our best to get a grip on them.