7th November 2016

Web Hosting wsl01007, wsl01012, wsl01004, wsl01015

We are aware that there are issues affecting this server. We are working to get them resolved.

Update - Mon Nov 7 13:34:30 MST 2016: The hypervisor these servers reside has encountered hardware issues. We are currently working on restoring service and resolving the hardware issue. wsl01007 and wsl01012 are up. wsl01004 and wsl01015 have started automatic filesystem checks.

  • Update - Mon Nov 7 14:12:08 MST 2016: The filesystem check for wsl01004 is at 8.5%. The filesystem check for wsl01015 is at 4.7%

  • Update - Mon Nov 7 15:04:57 MST 2016: The filesystem check for wsl01004 is at 15%. The filesystem check for wsl01015 is at 27%

  • Update - Mon Nov 7 16:12:43 MST 2016: The filesystem check for wsl01004 is at 31%. The filesystem check for wsl01015 is at 55.7%

  • Update - Mon Nov 7 18:30:09 MST 2016: The filesystem check for wsl01004 is at 88%. wsl01015 is online.

  • Update - Mon Nov 7 19:08:56 MST 2016: All systems online.

  • Update - Tue Nov 8 06:13:50 MST 2016: Storage of these servers required RAID re-build due to failed drives in it. The rebuild process is running now which causes some performance degradation on these servers. Currently its progress is at 7.8% and we expect this process to be finished in ~24 hours.

  • Update - Tue Nov 8 23:34:36 MST 2016: The rebuild process is at 18.5% complete.

  • Update - Wed Nov 9 05:26:25 MST 2016: The rebuild process is at 22.5% complete. Since we had to lower rebuild priority in business time to let sites on this server work this will take some extended time.

  • Update - Wed Nov 9 21:59:52 MST 2016: The rebuild process is at 37.5% complete. We are shutting down the following servers through the night to allow the Rebuild to complete: wsl01004 wsl01007 wsl01012 wsl01015

  • Update - Wed Nov 9 22:24:42 MST 2016: The rebuild process is at 50% complete with about 40 minutes remaining.

  • Update - Wed Nov 9 23:07:30 MST 2016: The rebuild process is at 82% complete with 25 minutes remaining.

  • Update - Wed Nov 9 23:31:55 MST 2016: The rebuild process is complete and the servers are booting up.

  • Update - Wed Nov 9 23:52:02 MST 2016: All servers are online.