Several of you asked us what happened on Friday evening. Here is the full account, hour by hour.

Down
Fri. 9:40 PM
Back
Sat. 9:30 PM
Duration
23 h 50

6 PM: the planned maintenance

We’re always working to reduce lag and connection delays. To that end, a one-hour maintenance was scheduled for Friday at 6 PM. The goal: move our connection and security servers, along with our IP addresses, from the host OVH to a private host. That would cut our monthly costs while giving us better control over the infrastructure.

The maintenance ended at 7:05 PM, and the server’s ping improved noticeably. We were delighted.

9:40 PM: chaos

After several hours without a hitch, the evening RP session had just started and we were nearing peak player count. At exactly 9:40 PM, the routing infrastructure (the system that redirects your connections to our game address) failed. Its owner couldn’t be reached for a quick fix.

Brokeos, our technician and the one who keeps the server running, was also away for personal reasons. The rest of the team had neither the networking skills nor access to the source of the problem: we were stuck. When the routing server went down, it took all our remote management access with it. No way to roll back the configuration or switch to our backup systems.

At 10:30 PM, we decided to wait for Brokeos and our new provider to be back.

Saturday 8 PM: recovery

After some thought, we took back control and decided to return to OVH, with a full emergency reconfiguration of the server. Some will feel we were slow to react: our work depends on the availability of each team member, all of them volunteers.

The server was back online at 9:30 PM, after 23 hours and 50 minutes of downtime.

What we’re taking away

We learn from our mistakes. We’ve told our provider how unhappy we are with their response time, and we’ll put precautions in place so this doesn’t happen again.

We know those 24 hours without playing were frustrating. We did everything we could to fix the problem as fast as possible, and we’ll keep you informed.

Thank you for your understanding, and enjoy the adventure.