Meauring Power Consumption

I manage a computing cluster for the Center for Statistical Genetics. As the cluster has grown and power becomes more and more critical, I had lots of problems trying to balance the load on the circuits and UPS units.

Most of our gear is from Dell and I found that subscribing to the Dell User Mailinglists has been very useful. I asked my questions on that list and was led to these Dell web pages which were very helpful in calculating the power consumption of our nodes. Playing with the configurations, I was able to determine that disk drives require about 10 watts and each GB of memory requires about 12 watts (remember YMMV).

Others told me one should only load a circuit to about 80% (e.g. you only get 24 amps from a 30 amp circuit). Because the various UPS units can only support fixed increments of power (and the bias is to not load them 'too heavily'), most UPS units only have 2-3 machines plugged in (depends on size, of course). This means for every 2 or 3U of machines, one might use 2U of UPS - meaning a third or more of a rack is dedicated to UPS units.

Still, this wasn't enough - more accurately, I didn't trust that I could answer the questions well enough to get a completely accurate estimate. Eventually some kind soul whose name I have lost/forgotten led me to the Kill-A-Watt Electric Usage Monitor (cost $35-40). Just plug this residential-grade device into a three-prong grounded wall outlet, plug in your device and get easy access to the real-time values for your device.

I put the machines under load for a period and got values that I think are accurate - of course the trick is to get the load correct. Still, I can plug the Kill-A-Watt into the UPS and the machine into it and let it sit for some hours and have some confidence I know what the real load is over a period of time. Here are the numbers I've found for the machines in our clusters:

Mfg + Model Max Amp Max Watts Notes
Dell PE2600 1.60 200 2xIntel@2.3 GHz processors, 6GB memory, 6x30GB internal SCSI drives, dual power supplies
Dell PE6600 6.60 800 4xIntel@2.8 GHz processors, 16GB memory, 6x100GB internal SCSI drives, SCSI card, dual power supplies, remote management, dual power supplies
Dell PE1850 2.55 300 2xIntel@3.0GHz processors, 4GB memory, one hardly used SCSI drive, single power supply
Dell PE1425 2.34 276 2x3.0GHz Xeon processors, 2GB memory, one hardly used SATA drive, single power supply
Dell PE1950 3.90 460 2xIntel dual core Xeon 3.2GHz processors, 8GB memory, 2xSATA 750GB configured as RAID1 (Perc/5i), SCSI card, dual power supplies, remote management
Dell PE1950 2.00 240 2xIntel dual core Xeon 3.2GHz processors, 4GB memory, one hardly used SATA drive, single power supply
Dell PE1950 2.24 264 2xIntel quad core 5130@2.0GHz processors, 8GB memory, one hardly used SATA drive, single power supply
Dell PE1950 2.88 345 2xIntel quad core 5345@2.33GHz processors, 8GB memory, one hardly used SATA drive, single power supply
Dell PE1950 2.40 280 2xIntel quad core 5420@2.33GHz processors, 8GB memory, one hardly used SATA drive, single power supply
Dell PE1950 2.71 325 2xIntel quad core 5420@2.33GHz processors, 16GB memory, one hardly used SATA drive, single power supply
Dell PE2950 3.30 396 2xIntel quad core 5345@2.33GHz processors, 16GB memory, 6 internal SATA drives, PERC/5i controller,dual power supplies, remote management
Dell PE2950 3.54 425 2xIntel quad core 5440@2.83GHz processors, 32GB memory, 6 internal SATA drives, PERC/6i controller, single power supply
Dell R610 2.65 318 2xIntel quad core 5570@2.93GHz processors, 32GB memory, one hardly used 2.5 inch drive, single power supply
HP DL360 2.27 272 2xIntel quad core L5420@2.5GHz processors, 4GB memory, 2 internal SAS drives
HP DL360 3.46 415 2xIntel quad core X5450@3.0GHz processors, 16GB memory, 2 internal SAS drives
SUN x4500 9.0 396 2xAMD dual core Opteron 290@2.8GHz processors, 16GB memory, 48 internal SATA drives, dual power supplies
IBM x3650 4.1 490 2xIntel quad core 5440@2.83GHz processors, 32GB memory, 6 internal SATA drives, RAID controller, single power supply

In addition I've measured the power consumption of our RAID units. As you might expect the more drives, the more power, but compared to our servers, the RAIDs draw little power for the space they require.

Manufacturer Max Amp Max Watts Notes
DataMax 8800 0.70 87 SCSI attach, 8x250GB IDE drives (http://www.dnfstorage.com)
SIVY SA-7230 1.65 200 SCSI attach, 12x400GB SATA drives (http://www.maxtronic.com/)
JetStor 416S 2.10 252 SCSI attach, 16x750GB SATA drives (http://www.acnc.com/)
Infortrend A16E iSCSI 2.25 270 iSCSI attach, 16x750GB SATA drives (http://infortrend.com/)
Dell MD1000 1.30 156 SCSI attach, 15x750GB SATA drives (http://www.dell.com/)
SUN j4400 3.3 396 SAS SFF-8470 attach. Non-RAID, array of 24 SATA drives, dual power supplies