Ensuring uptime in health care data centers

Maintaining continuity of electrical power is critical to health care data center operation.

The components that ensure continuity include generators, fuel systems and uninterruptible power supplies. This equipment is costly, requires considerable space and often structural accommodations to support it.

Consequently, proper sizing of the equipment is essential for determining site requirements and project budget when planning backup power systems for health care data centers.

Establish reliability requirements

The data center’s required level of reliability will have a strong impact on design, space requirements and cost of the infrastructure supporting the data center. It’s customary to describe levels of reliability in terms of one of the Seattle-based Uptime Institute’s Tier standards for data center design and construction, numbered I though IV. The reliability target for a hospital data center is often at Tier III.

From the standpoint of its electrical system, a Tier III data center is required to be “concurrently maintainable” in that any single capacity component or element of a distribution path can be removed from service for maintenance, without disrupting information technology (IT) functions. On-site power production is viewed as the primary service, and electric utility power as an economic alternate of no meaningful reliability. Capacity components must be individually maintainable while the data center is powered by its generators. Twelve hours of on-site fuel storage are required for compliance. Generators must be rated for unlimited runtime at the full load of the essential systems of the data center. A Tier III data center also must have more than one electrical distribution path serving the IT load. IT equipment must be dual-powered to take advantage of redundant distribution.

However, Tier III requirements are onerous, expensive, space-intensive and they often are unsuitable for hospital requirements, existing infrastructure, budgets and available space. Tier III requirements were developed in view of the requirements of general-purpose data centers, rather than data centers as components of active hospitals or hospital systems. In practice, hospital data centers often don’t strictly conform to the requirement of Tier III. In particular, a data center located on a hospital campus and sharing its essential power system will deviate from the standard in a number of ways.

Determine system requirements

To avoid potential service disruptions, the essential power system’s initial capacity must be adequate to serve its initial demand. Its ultimate capacity must satisfy its highest anticipated demand, and the design must allow for capacity additions as required to meet intermediate load levels, with a reasonable capacity margin at each stage.

Estimates of the total demand of IT equipment can be developed in a number of ways. For some projects, the health care facility has an existing IT installation to be migrated to a new data center. Under those circumstances, it’s possible to measure demand at an existing facility, and predict the initial IT demand in a new data center with some accuracy. More commonly, the facility will migrate some of its systems and add other equipment whose characteristics aren’t well-known.

A substantial amount of existing equipment is certain to go obsolete during design and construction, making the initial demand more uncertain. And unforeseen corporate initiatives or governmental mandates can substantially impact the data-processing load and the electrical demand. Accurate forecasts of future demand are nearly impossible to produce. Sizing of the essential system is a trade-off, in which the risk of stranded investment from too large an infrastructure is balanced against the risk of substantial future costs of remediation or premature obsolescence of the data center.

For Tier III compliance, generators must be rated for unlimited runtime at full load of a data center’s essential systems.

Photo courtesy of Caterpillar Inc.

Estimates of initial demand may be extrapolated from that of similar existing facilities, as described, or by direct calculation for specific equipment. A variety of estimating tools are available from IT and power equipment vendors. Future load growth typically is estimated by applying historical load growth data developed directly from the health care facility’s
experience, or from industry sources, over the life
of the facility. 

The essential power system must support HVAC, lighting, battery charging, equipment inefficiencies, necessary administrative functions and other support loads in addition to direct IT demand. HVAC demand is largely based on IT demand, in that practically all of the energy consumed by IT equipment is delivered to the space, and must be rejected by the HVAC system. The actual HVAC demand will depend on the technology used to cool the data center.

In a major hospital building, chilled water often is available from a central plant, or from a high-reliability, third-party supplier, substantially reducing the demand of the HVAC system. A water-cooled chiller dedicated to the data center will require more essential power, an air-cooled chiller will require still more, and direct-expansion units will have even a higher demand. A redundant chiller, if provided, must also be included in supporting equipment-demand calculations.

Equipment selection

Emergency power equipment sizing and specification requirements are based on a number of variables, depending on size, criticality and regulations. Some of the major considerations include:

Generators. A data center located on a hospital campus often will employ the hospital’s generator system as the source of essential power. For this strategy to be viable, the hospital system must have adequate capacity to serve both the hospital’s operational requirements and the data center essential load simultaneously. To provide N + 1 redundancy, the system must be able to serve the loads with its largest generator out of service. For a new hospital facility, with the hospital and the data center designed concurrently, achieving this level of service is largely a matter of coordinating the data center requirements with the design of the essential system.

Paralleling switchgear for four generators producing 2 megawatts of power each.

Photo by Tom Devine

For existing facilities, using the hospital system is more challenging. A data center capable of serving the needs of an urban hospital system can require 1 to 2 megawatts or more, an appreciable fraction of the total generating capacity of all but the largest hospital installations. A careful review of the capacity and demand of the existing essential system will determine whether it can support the addition of the data center.

A review of the existing system’s priority load-shed sequence is also necessary. Most hospital systems are designed to limit the total system load to something less than the total connected generating capacity, to avoid overloading the system in the event that one or more generators fail. High-priority loads, such as egress lighting and critical patient care systems, are served first, followed by lower-priority loads as capacity is available. Loads are generally added in large blocks, with the number of blocks equal to the number of generators. The data center typically will be prioritized below patient care and essential operational loads, and above optional loads. To ensure that the system behaves as an N + 1 system from the data center’s viewpoint, the center must be prioritized in something other than the lowest-priority block. Inserting a data center into an existing priority load-shed scheme often will involve delicate negotiations.

A partial diagram of a Tier III data center from the viewpoint of the IT load, with distribution not shown. To remove the UPS from service, Generator #2 is started and operated for the duration of the UPS outage, during which the system is vulnerable to a failure of the generator.

Diagram by Tom Divine

The solution to inadequate capacity on the hospital system typically will be additional generation. Many hospital systems are designed to accommodate additional generators, limited by the number of generator positions available in the existing paralleling gear. Failing that, the data center project must supply its own power generation.

Operating a data center from a hospital essential system will necessitate certain deviations from Tier III requirements:

• Hospital generators typically are rated for standby service. Standby generators are rated for a limited number of operating hours, with a varying load. Tier III requires that generators be capable of delivering rated load indefinitely. In practice, hospital systems generally run for a limited number of hours per year except under the most unusual circumstances. This deviation from the requirements of Tier III often is considered acceptable by the health care facility. Otherwise, dedicated generation must be provided for the data center.
• Many hospital systems are arranged with a number of generators serving a single paralleling gear. Consequently, the paralleling gear cannot be taken out of service while the essential system is delivering power, without interrupting service. Hospitals rely on these systems for continuity of patient care, and generally find their reliability acceptable.
• Generator fuel systems may not be entirely independent in that a single storage tank or fuel-transfer system may serve all generators.

Fuel system. For Tier III compliance, the system must have on-site fuel storage to support 12 hours of operation, with any of the system components — including fuel tanks and delivery systems — out of service. A hospital essential power system typically will have adequate fuel to meet the storage requirement, but may not have it in multiple tanks with independent fuel management. 

Uninterruptible power supplies. A data center uninterruptible power supply (UPS) provides continuous power to the load during the brief delay between the failure of utility and the delivery of power from the generators. An online UPS, which converts incoming energy to DC internally and back to AC for delivery to the load, also provides a high degree of power conditioning, ensuring that its output power is close to ideal.

A Tier III data center requires at least one UPS. With dual-powered IT equipment, a system with two distribution paths can support its load by delivering the load’s full power requirement in one path from the time utility power fails until the generator starts and accepts the system load. UPS capacity must be at least equal to the full IT equipment demand in its distribution path, plus whatever allowance for growth can be allocated to its load. In practice, data center managers are loathe to expose IT equipment to unconditioned power, and all distribution paths will be supported by a UPS.

The primary characteristics of UPS capacity are total output power and runtime — the length of time that the UPS can support the IT load before its stored energy is depleted. In a Tier III data center, a UPS must be able to support the full requirements of the load it serves when any other distribution path is delivering no power to the load. Though many topologies are possible, a data center UPS typically will share loads with one other similar unit, so this requirement means that the UPS must be able to directly support all of its loads unassisted. The UPS must also be sized for expected load growth. Commonly, the UPS will be constructed of a maximally sized frame with modular power components. The ultimate size of the unit is selected at design time, and the unit is equipped with adequate power modules to support the initial load. More modules are added as load increases, to increase the UPS output capacity.

Runtime typically is specified to support the load during the time it takes the generator to start and accept the UPS load and, should the essential system be unavailable, long enough for the IT equipment to effect an orderly shutdown with a reasonable margin. Most battery-based systems will be specified with runtimes of five to 15 minutes, while rotary systems typically will have full-load runtimes of 15 to a few tens of seconds. Extended runtimes for battery-based systems can be expensive, as they require space and structural accommodations for a large, heavy battery installation.

For data centers with dedicated generation, the period from an outage to availability of essential system power can be quite short, on the order of 10 seconds. For centers sharing a hospital’s essential system, the delay can be considerably longer. Large hospital systems consist of several paralleled generators, and they operate with a prioritization system to ensure that the most essential loads are supported quickly, while less urgent loads wait for adequate generation to come online. Depending on the paralleling technology, the number of generators and the load blocks associated with the data center, the delay between loss of utility and availability of generator power can be considerably longer [see sidebar, Page 28]. These delays must be considered in establishing the required runtime for a UPS system.

Reasoned estimates

Data center components represent considerable cost and require considerable space and support from a health care facility’s physical infrastructure.

Well-reasoned capacity and size estimates made during the project planning stage will help to avert many unpleasant surprises later.

Tom Divine, P.E., is senior electrical engineer at Smith Seckman Reid Inc.'s Houston office.