Power maintenance December 25 [update]

Friday, December 22, 2017

During 2017 there’s been a complete overhaul of the electrical distribution infrastructure in our California data center, increasing capacity 3x, replacing UPS equipment, updating control and monitoring technology.
Because of the built-in N+1 redundancy all of this work was completed “hot” with no impact to critical production loads, however the final stage will involve a brief service interruption for users.  At this point we have to gracefully shutdown all servers that do not have redundant power supplies,  transfer their rack power supply to the new system, then reboot them all manually.  Our estimate is ten minutes of downtime per rack to complete the process.
Shared hosting servers, rack level switches and firewalls all have redundant power so they can stay running throughout, but there’s some opportunity for latency and packet loss while this transition takes place so even those can expect a few minutes of degraded service.
To minimize the impact of this, we’ve scheduled the work for December 25, 0600-1200 Pacific time, during which time we expect most people to be otherwise occupied with Christmas celebrations.  This is a six hour maintenance window, but to reiterate our estimate is ten minutes per rack.  In case of any problems there’s a standby date of January 1 to complete the work.
Thanks for your understanding as we complete this important work, we will update in the client area and our Twitter feed https://twitter.com/servelink once complete.

[update December28] this work was all completed as scheduled.  We have a routine load transfer test on the morning of December 29, to test transfer from utility power feeds, to generators to UPS and back again.  No client impact is anticipated.

« Back