Best practices for hospital power system reliability

Photo courtesy of Caterpillar Health facilities professionals should consider installing leak-detection systems in generator rooms.

Health care facility power systems are required to operate dependably, with high reliability and availability. They must operate and give the same results on successive trials as well as function at any instant required and from that point forward.

Best practices can be undertaken throughout the life cycle of power systems. The power system life cycle starts with planning, design, construction, installation and commissioning. It then continues with ongoing inspection, testing and maintenance. Existing power systems may have unknown vulnerabilities that can be uncovered and removed or mitigated. And, finally, management of the power systems should include identifying preferred failure responses and planning appropriately for them.

This, of course, is in addition to codes, standards or accreditation-related compliance requirements.

System installation

Many of the best practices for high-reliability designs are well-known but not always attainable because of cost — minimizing physical risks due to environmental causes such as flooding, windborne hazards and other potential hazards. Protection from flooding should consider both external (such as inclement weather) and internal (such as a broken pipe or tank in a nearby mechanical room) causes. Health facilities professionals also should consider that water is almost always likely to find the lowest elevation and a sump pump is only as good as its power source and maintenance.

Despite best efforts, power equipment sometimes fails. Installing power systems that mitigate the impact of discrete equipment failures can be a best practice that allows rapid response for isolation and repair of failed equipment. Examples are power systems with multiple electrical utility feeds as well as main and tie device arrangements that can be switched automatically, manually or both. The adverse impact of incoming utility service failures also might be mitigated if the facility power system contains means to connect portable equipment such as generator sets. All portable equipment connections also should be designed considering electrical safety.

Other best practices include both physical separation and fire separation between normal power equipment and emergency power equipment, and limitations on possible equipment locations to those that are at less risk to adverse environmental influences. Sometimes further separation between similar systems (normal power “A” and normal power “B” for example) also can be considered a best practice to reduce the potential for putting all eggs in one basket.

Power equipment design choices also can encompass best practices by specifying equipment details that facilitate ongoing inspection, testing and maintenance. Modern power system equipment contains many features that can be used to maintain or improve dependability.

For example, infrared viewports can be installed within the outer enclosures of power equipment. These viewports allow regular infrared thermographic scanning of interior components to be performed without the risk of opening the covers of energized equipment. Likewise, manufacturer-recommended maintenance of emergency power automatic transfer switches (a National Fire Protection Association NFPA 110 requirement) is facilitated when the automatic transfer switches are bypass-isolation design. The bypass-isolation design is intended to permit the electromechanical transfer mechanism to be maintained without causing a power outage of its high-priority loads. Additionally, electrical power switchgear can be provided with special features to improve electrical safety during equipment testing and maintenance.

Commissioning is recommended whenever power system components are installed or modified. The purpose of commissioning, which goes beyond the typical design team construction observation and punch list process, is to ensure that the power system installation is in accordance with the contract documents and the owner’s project requirements. Some power equipment failures were traced to deficiencies that could have been found and corrected by a comprehensive, commissioning and installation acceptance testing process.

Finally, design projects commonly result in good documentation of system configuration; however, hospitals usually have many changes over time. The overlapping quilt of infrastructure changes over years will make even the best construction drawings outdated and, thus, not very usable for operational needs. A best practice is to maintain current documentation of the information necessary for operational needs such as internal system failure response.

Proactively determining the changing load on major power equipment as the facility evolves is a best practice. There are many ways to do this, including power monitoring systems, portable metering and load profile analyses.

Maintaining performance

Regular inspection of high-value electrical equipment is important. It allows the operational staff to determine whether adverse environmental changes or equipment changes have occurred that should be corrected to maintain desired reliability. Facilities personnel are the eyes and ears of the organization; through regular rigorous inspection protocols, they are more likely to find potential environmental and equipment problems before equipment failures occur.

As the demands upon our facilities change, the power systems may not always change with them. Many hospitals operate with power equipment that is decades past its useful life. In those cases, enhanced inspection and maintenance are critical to continued dependable operation.

Management of portions of the power system, such as the emergency power supply systems, must meet the minimum inspection, testing and maintenance requirements stipulated in NFPA 110, Standard for Emergency and Standby Power Systems. For the normal power systems along with other utilities, however, the Centers for Medicare & Medicaid Services (CMS) now also requires that organizations know the manufacturer’s recommended maintenance activities and frequencies. The organizations are required to conduct a risk-based analysis to justify deviating from those recommendations.

Annual infrared thermographic scanning of electrical power equipment is a best practice. It helps organizations to discover potential problem areas and correct them before they develop into dangerous and disruptive failures. It also can be used as a predictive maintenance tool to help focus limited funding.

Electrical power equipment maintenance is a best practice, but is hard to accomplish within health care facilities. The 2012 American Society for Healthcare Engineering (ASHE) management monograph titled “Managing Hospital Electrical Shutdowns” contains a robust discussion of issues and recommendations related to this topic. Other recommended actions include:

Planning for contingencies. Although most hospitals have some form of utility equipment failure procedures, a best practice can be validating that the failure procedures will pass the responding staff member’s reality test the day the failure occurs.

Many power failure procedures simply assume the loss of incoming utility service. Although this assumption certainly has a major impact on the facility, it is not the most common example of a power failure experienced within hospitals. Power system-related failures are more commonly a malfunctioning component such as a transfer switch, a short circuit within a feeder due to construction or other damage, a lighting ballast failure that trips out a major riser to multiple floors, or any other failure that results in the loss of one or more panelboards, motor control centers or switchboards.

A best practice is to recognize that these internal failures happen and plan appropriate responses ahead of time. Consider the different failure points, not just at the electrical service mains. The failure responses should be different for each type of failure, and it is too late to formulate a response after the failure.

Finding and mitigating vulnerabilities. There are many available activities that can have a positive impact on the dependability of existing power system installations. Facility personnel can find power system vulnerabilities by assessing their physical installations, operations, training, communications, inspections, testing, maintenance, electrical safety, contingency planning and hidden common-mode failure potential for their effect on reliability, availability and dependability.

Options for reviewing and improving the dependability of existing installations can include conducting risk assessments, identifying existing or potential hazards and undertaking hazard mitigation strategies.

An example of proactive inspection resulting in identification of vulnerabilities includes:

• Visiting the facility’s high-value electrical equipment rooms, such as those containing emergency power equipment or main normal power electrical service equipment.

• Determining whether any of those rooms are adjacent to mechanical rooms and whether it is possible for water to enter the electrical rooms if any of the mechanical pipes or tanks leak or break.

• Determining whether electrical rooms on lower floors are subject to external flooding in certain situations.

Facilities professionals should consider using leak detection in the high-value electrical rooms, which warn of water-based vulnerabilities. The leak detection equipment should be monitored and subject to its own inspection, testing and maintenance processes.

Another simple example is one in which the organization relies on sump pumps to protect critical equipment or processes. Facilities professionals should check whether the sump pumps themselves have been looked at as a potential vulnerability upon failure. Additionally, they should check whether the sump pumps are monitored and whether they are included in any inspection, testing and maintenance processes.

Assessing failure risks. Risk assessments are used regularly where required in many aspects of the health care physical environment. They are not often used, however, to assess existing power systems, where they may be considered to be a best practice. Examples of power system risk-assessment topics can include:

• Locations of normal power equipment;

• Locations of emergency power equipment;

• Whether to upgrade or replace older equipment;

• Assessing potential vulnerabilities that have been discovered;

• Determining the need for more robust utility failure procedures;

• Whether to provide emergency power (beyond regulatory requirements) to certain mechanical equipment;

• Inspection, testing and maintenance options beyond regulatory requirements.

• Analyzing vulnerabilities. Power system vulnerability analyses are a best practice and can be used in many ways. For example, vulnerability analyses can be used to determine what types of loads that are presently powered only by normal power should be powered from emergency power.

A vulnerability analysis can be used to identify the importance of findings and observations from ongoing inspection, testing and maintenance activities.

Vulnerability analyses also can assist in identifying the potential for common-mode failures, which are failures of two or more systems or components due to a single event or cause. A safety engineering concept states that once a failure mode is identified, it usually can be mitigated by adding extra or redundant equipment to the system. However, the existence of an uncorrected common-mode failure potential removes the advantage of other redundancies. Examples of potential common-mode failure points in an emergency power system are a single fuel storage tank, a common fuel oil riser pipe, or even a single fuel oil pump control panel.

When looking for potential vulnerabilities, consider that things break. Just because equipment worked well yesterday does not mean that it will work well tomorrow. Pay attention to the details — often those details can result in unexpected occurrences. When failures occur, look for any commonalities that suggest system issues or process issues rather than just single failures. Remember that you can’t control what you can’t control, so you should plan for the unwanted. Have rigorous failure procedures that can be understood and followed by those who may need to respond.

Analyzing performance gaps. Many hospitals employed gap analyses to assess what equipment was powered by emergency power and whether there were gaps in the selection of power sources that needed to be corrected. This approach was somewhat common among Joint Commission-accredited hospitals after it issued its 2006 Sentinel Event Alert, Issue 37 titled “Preventing adverse events caused by emergency electrical power system failures.”

Gap analyses can be a structured approach to assessing any problematic situation and developing solutions to rectify the problem areas. As a best practice, a gap analysis also can be used to address the results of a vulnerability analysis. This can assist organizations in formalizing an action plan to assess specific vulnerabilities, the types of failures that might be caused by those vulnerabilities, and how the vulnerabilities can be eliminated or reduced.

A gap analysis can be structured as the following action-oriented process:

• Define the concerns, policies, urgency, needed data and metrics;

• Assess the current situation;

• Analyze data and summarize gaps;

• Develop recommended actions to close the gaps;

• Brainstorm strategies to bridge gaps and recommendations;

• Determine the best short-term and long-term options;

• Develop action plans;

• Implement the action plans.

Finally, it can be helpful to understand the vulnerability management life cycle. This includes finding potential vulnerabilities, prioritizing their potential likelihood and effect, assessing their impact on operations and patient care, reporting the results, taking short-term and long-term actions to mitigate the impact, and then verifying that the actions taken really mitigated the vulnerability.

Important function

Many of these best practices are covered in the 2014 ASHE management monograph titled “Managing Hospital Emergency Power Systems: Testing, Operation, Maintenance, Vulnerability Mitigation, and Power Failure Planning.” But, regardless of the source, ensuring emergency power reliability is among a facilities professional’s most important functions.

David Stymiest, PE, CHFM, CHSP, FASHE, is a senior consultant at Smith Seckman Reid, Nashville, Tenn., specializing in independent review of power systems, facilities engineering and regulatory compliance. Although he is chair of the NFPA Technical Committee on Emergency Power Supplies, which is responsible for NFPA 110 and 111, the views and opinions expressed shall not be considered the official position of NFPA or any of its technical committees and shall not be considered to be, nor be relied upon as, a formal interpretation. He can be reached at DStymiest@SSR-inc.com.

Educating clinicians about emergency backup

Lessons learned during recent major disasters indicate that many clinicians have unrealistic expectations about normal and emergency power failures. A best practice is for facilities professionals to educate clinicians about different power failure modes and the necessary responses to each.

The education should be comprehensive enough so that clinicians understand the different potential types of electrical failure in any critical care unit, each with its own response. These include:

• Failure of some portions of the normal power system with the remainder of the normal power still online and the emergency power system, also called the essential electrical system (EES), also available;

• Failure of the entire normal power system with the EES still online;

• Failure of one or more of the EES branches (e.g., life safety branch, critical branch or equipment branch) with the normal power system still online;

• In critical care spaces that have two separate critical branch sources, failure of one of the two critical branches with the other critical branch source still online;

• Total electrical failure to the space, either simultaneously or as the result of cascading failures over a period of time.

With each of these failure scenarios, the resulting impact on activities and required actions by caregivers may be different. Whereas, the most common anticipated response to a utility service outage is ensuring that critical equipment is plugged into emergency power (red) outlets, the actual requirement in the event of a critical branch failure would be to unplug equipment from the red outlets and plug it into the alternative outlets. Quickly differentiating between different scenarios and their necessary responses can improve patient care and patient safety when adverse power conditions occur.