There's real-time drama unfolding at one of the datacentres operating at 111 8th Avenue in NYC, a Google-owned building that occupies a full city block.
There are a number of datacentres housed in the nearly 3 million square foot building that functions as a carrier hotel, a major connection point for hundreds of domestic and international telecommunications providers.
The building, along with all of lower Manhattan, has been on generator power since Con Edison cut service to the area Monday night because of floods caused by Hurricane Sandy.
The loss of power has exposed generator problems at multiple locations around lower Manhattan, including at 111 8th Ave.
Datacentre operator Zayo uses some 26,500 square feet in the building, according to its website. Thursday morning it published an "urgent communication regarding the status of the 111 8th Avenue facility."
The drama has unfolded, so far, in a series of three communications by Zayo to its customers with updates on the company's various efforts to address problems that have come as the data center temperature went up and down.
Zayo is posting all of its emergency network updates here.
In its first missive, Zayo told its customers that the temperature in one of its datacentre suites had reached 93 degrees Fahrenheit. The company blamed "technical difficulties with the fuel pumping system to a subset of generators within the building, affecting the AC power."
In a later notice, published early this afternoon, Zayo said that temperatures in parts of the datacentre "have risen above 100 degrees."
It followed that with a note published after 2 pm Thursday reporting that the generator serving its datacentre suite "was taken off-line to implement a temporary fix." Workers hoped to stabilise the fuel pump and flow to the generator "in an attempt to return portion of cooling online," the note said.
In its latest posting, the data center operator said, "All Zayo network and customer equipment successfully transitioned to Zayo's DC battery plant with no interruption to customers. Zayo's DC plant provides for an estimated 5 hours run-time should the generator have to be taken off-line for maintenance or if there should be a failure.
"At approximately 2:12pm EST, the generator was reengaged and teams are assessing stability to bring some portion of cooling back on-line. With mitigation efforts temperatures have stabilized in a range of 95-100 degrees Fahrenheit and have been generally unchanged over past 2 hours at those levels," Zayo said in the post.
Zayo said it also has other contingencies in motion, including a 2MW generator that is in transit from New Jersey to the facility in New York City.
"If building infrastructure is not restored, Zayo's intention would be to tie its 7th floor services and cooling infrastructure directly to this back-up power source and maintain this through full commercial utility restoration," the company said.
Delivery of the generator is expected to take 2.5 to 3.5 hours depending on traffic.
A Zayo spokesman was not immediately available to comment on the status of the effort.
Telx, which also operates a datacentre at 111 8th Ave., said, in a posted note, that as of 1 p.m. ET, there haven't been unusual temperature increases in its datacentre. Telx is asking customers of its two datacentres in NYC, the other is at 60 Hudson St, to assist in reducing electrical load and curtail their systems where possible and to "power down any nonessential equipment at this time."
In an email to Computerworld, a Telx official said its data center isn't having problems similar to that of Zayo, suggesting that the latter's issue could be specific to it or to its infrastructure.
Telx is not expecting Con Edison to restore power to the building for three or four days.
Another datacentre company operating at 111 8th Ave., Equinix, reported Wednesday that it had a problem a failed generator "that impacted service to several customers."
Equinix said "repairs were made and service was restored within one hour and 15 minutes."