[Faculty] Fwd: Forbes 220 power and cooling issues summary

[Shane Burgess]

Everyone, below is a summary of the challenges our cyber infrastructure 
support people have been working through.

Some of you know that I am a "hawk" when it comes to having a functional 
IT backbone and working email.

I want to thank Ryan Cresawn and Robert MacArthur for their hard work on 
keeping us up and running under very challenging circumstances.

Regards,

Shane


-- 
Shane Burgess
Vice Provost and Dean
College of Agriculture & Life Sciences
Director, Arizona Experiment Station
Registered Lobbyist
The University of Arizona
Forbes Building, Room 306 | 1140 E. South Campus Drive | P.O. Box 210036
Tucson, AZ 85721-0036
PHONE: 520.621.7621
FAX: 520.621.7196
EMAIL: sburgess at cals.arizona.edu


-------- Original Message --------
Subject: 	Forbes 220 power and cooling issues summary
Date: 	Tue, 25 Mar 2014 09:43:24 -0700
From: 	Ryan Cresawn <cresawn at cals.arizona.edu>
To: 	Shane Burgess <sburgess at cals.arizona.edu>
CC: 	Robert MacArthur <robmac at ag.arizona.edu>, "Rahr, Matthew J - 
(rahr)" <rahr at ag.arizona.edu>



Hi Shane,

Tod conveyed to me that you would like a summary of the power and 
cooling issues recently affecting our server room, Forbes 220.

On March 20, the HVAC contractors powered off and began removing the air 
handler located in Forbes 224. HVAC contractors installed portable AC 
units in the room which maintained a warm, but acceptable temperature in 
the room.

In the early morning of March 21, the temperature rose quickly in Forbes 
220. It peaked at 100F, which is dangerous to the equipment in the room. 
Two problems caused this temperature spike: the portable AC units were 
inadequate and the exhaust duct which was taped to the door by the HVAC 
contractors fell off. Tod and I discussed the problems with HVAC 
personnel and proposed the exhaust ducts be bolted to the door. They 
implemented the idea and the ducts have stayed on the door since that 
time. The inadequate AC units maintained a temperature in the room of 
roughly 90F. We proposed to the HVAC contractors that they replace the 
portable AC units with new ones that would cool better.

Around 1 pm yesterday, March 24, HVAC contractors replaced the portable 
AC units with new ones and added a third new one. Over roughly 12 hours, 
the temperature in the room dropped from approximately 90F to the 
mid-70s. When the new portable AC units were installed yesterday it was 
not clear immediately that one of the units had tripped circuit breaker 
19. Circuit breaker 19 provides power to two of our four large UPSes in 
the room. Some of the servers were running on battery after circuit 
breaker 19 tripped. Tod and I alerted HVAC personnel who searched for 
and located the panel with circuit breaker 19. While HVAC personnel were 
resetting the breaker, Tod and I were rerouting the power cord for the 
portable AC unit that was on circuit 19.

Now the portable AC units are all powered by dedicated circuits that are 
not interfering with UPSes that supply power to our servers.

The attached line graph shows the temperature of one of our hard drives 
in one of our servers in Forbes 220. Temperatures are in Celsius. 
Seagate recommends that hard drives temperatures stay between 5-50C. You 
can see that we reached 48C on one our servers.

The attached directed graph shows the power dependencies we have in 
Forbes 220. The columns from left to right are: servers, UPSes, and 
electrical circuits.

Please let me know if you need anything else.

Thanks,
Ryan

-- 
Ryan Cresawn
Systems Administrator, Principal
CALS Communications & Technologies
520-621-2489



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.cals.arizona.edu/pipermail/faculty/attachments/20140327/f37c4dc7/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hddtemp_smartctl-week-20140325.png
Type: image/png
Size: 19803 bytes
Desc: not available
URL: <https://list.cals.arizona.edu/pipermail/faculty/attachments/20140327/f37c4dc7/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PowerDependency.pdf
Type: application/pdf
Size: 15439 bytes
Desc: not available
URL: <https://list.cals.arizona.edu/pipermail/faculty/attachments/20140327/f37c4dc7/attachment.pdf>