Further system updates postponed due to end of maintenance window. We still have some secondary systems that needs to be updated, but we will postpone further updates until tonight since we are at the end of the maintenance window. During the upgrade, all systems was down between 02:07 CET and 02:17 CET.
Unfortunately the messaging APIs gave a 500 error until 3:40 CET due to an unnoticed bug with the upgrade.
From 03:40 CET to 04:40 CET messages was accepted by the API, but queued for later delivery due to the previously mentioned bug.
This is the first time we are doing a major upgrade and redeployment of the core services in GatewayAPI. Combined with a major infrastructure change this was a very busy night indeed. Mistaskes was made and corrected, we have learned a lot already.
The remaining updates, postponed for later, should not affect system availability or performance.
Posted Dec 14, 2016 - 07:08 CET
Update
The hardware upgrade is complete. Unfortunately there was some unexpected minor issues so while the API servers came back online quickly they where not accepting SMS'es for a large part of the window, and afterwards a backlog was built up due to another unexpected issue with the OS upgrade. The issues have been resolved and messages are flowing normal again. We still have some additional work ahead but do not expect any more downtime.
Posted Dec 14, 2016 - 04:57 CET
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Dec 14, 2016 - 02:00 CET
Scheduled
We are migrating all our networking hardware to a single vendor (Cisco), which simplifies management and improves performance. This migration will cause an approximate 10 minute downtime for all services.
We do not expect the network migration to cause extended periods of downtime, as we have prepared everything so it's essentially a blade switch module swap, however due to the fact that it's a complete overhaul of the entire network stack, there is a risk that the migration might overrun it's expected timeframe.
During the maintenance window we will also be upgrading the OS of the servers, to lessen the impact on our services we will complete this during the same maintenance window.
We apologize for any inconveniences this might cause for our customers.