Server's offline 7/11/23

[=Dallas=]

All offline :(

[=(V)=Mar]

At least some NY endpoints unreachable
Jul 10 2023 02:56:41 PM PT We are currently investigating why our NY location appears to be offline. This is not expected and is extremely unusual; it could represent a router or infrastructure problem (such as power). We will edit this event when we have additional information.

Update @ 6:04pm EDT: Further investigation by our team indicated that this was a problem upstream of us, with our upstream provider (PacketFabric, formerly Unitas Global, formerly INAP). We called them and they confirmed that they experienced some sort of connectivity problem in the metro that they are still looking into; one or more core routers may have failed, or there was a major fiber cut.

The location has become partially reachable and seems to be recovering, likely because traffic is switching over to redundant links and devices. We will continue to monitor.

Update @ 6:36pm EDT: We haven't received any further updates from the upstream, but the location appears to be offline again.

Update @ 6:50pm EDT: The connection is still offline, and when we called them again, our upstream confirmed that it was still a problem on their end, and that the network had come back up for a bit before failing this second time. They are working to bring the location back online as quickly as possible.

Update @ 7:53pm EDT: The location continues to be offline. We contacted the colocation provider (Evocative, formerly INAP), and they are saying that UPSes (Uninterruptible Power Supplies) are offline and are currently investigating.

We received a 3rd-party suggestion that there may have been a fire in the electrical room of the facility, which could have caused significant additional damage, but Evocative said that it is not aware of a fire.

Update @ 9:38pm EDT: The facility provider is now confirming that there was a small fire in a UPS that led to a fire department response. The fire marshal forced offline the datacenter while they investigate the source and evaluate the level of damage. Once the fire marshal gives the go-ahead, the facility can be re-energized. This may be within hours, if no problems are found.

All of our equipment is set to start back up automatically after a reboot, but it is possible that the power failure caused damage, particularly if there was a power spike at the same time. We will be standing by to audit all equipment to make sure that it starts up properly, and to resolve any problem with broken equipment that may occur.

Update @ 11:29pm EDT: The facility is required to thoroughly clean the UPS and all surrounding gear before the marshal will allow the facility to be brought back online. They are working with a vendor on this, but expect it to take at least all night.

Update @ 9:15am EDT on 7/11: The facility has worked through the night cleaning equipment but has not yet reached the next stage of having it inspected. They are hoping for an afternoon re-energizing, but that process will also take a long time (many hours) as the facility has to be cooled as part of it, among other things.

Update @ 1:17pm EDT on 7/11: The cleaning continues, and an inspection is scheduled for 2pm EDT. If the fire marshal clears the site to be re-energized, they can start the process of re-energizing and testing their feeds and UPS equipment; over-cooling the facility; turning on other facility equipment and specific customer infrastructure (such as major backbone routers); then slowly bringing up customers in a staggered manner, being careful not to overload their power feeds. With all of this, they estimate that power may not be restored for another 6 hours.

If you require a temporary service to be spun up in the interim at another location, please let us know.

Update @ 4:41pm EDT on 7/11: We have received the following update from the facility, telling us that it will be at least another full day before our gear can be turned back on.

We have just finished the meeting with the fire marshal, electrical inspectors, and our onsite management. We have made great progress cleaning and after reviewing it with the fire marshal, they have asked us to clean additional spaces and they have also asked us to replace some components of the fire system. They have set a time to come back and review these requests at 9am EDT Wednesday. We are working to comply completely with these new requests with these vendors and are bringing in additional cleaning personnel onsite to make the fire marshal's deadline.

In preparation for being able to allow clients onsite, the fire marshal has stated that we need to perform a full test of the fire/life safety systems which will be done after utility power has been restored and fire system components replaced. We have these vendors standing by for this work tomorrow.

Assuming that all goes as planned, the earliest that clients will be allowed back into the site to power up their servers would be late in the day Wednesday.

Will be offline until late Wednesday :(

[=(V)=RocketJedi]

Thanks was about to post this as well. I put it on FB and Discord if not done already

[=Dallas=]

Thank you for the updates.

[=(V)=~Sean~]

Yes, thank you all.

I am just glad that we all still love this 24 year old game. It's like a friggin' drug...... gotta have it ! :)

Enjoy July.... it only comes around once in a while.

[=Dallas=]

FYI, All appear online now. Thanks
Come get u some! :))