From a datacom power and cooling infrastructure planning perspective, the two common means of maximum load characterization are watts per square foot and kilowatts per datacom equipment rack or cabinet. The datacom industry sometimes uses granularity as a means of describing the unit size.

Many in the datacom industry think that kilowatts per rack is superior to watts per square foot. However, at the start of a project, there may be insufficient information about the quantity of racks or their expected contents, making that metric too granular. Professional judgment is critical to deciding which maximum load characterization to use.

Of equal importance is characterizing the minimum load as well as the load variation. The time increment for load variation can be very short (e.g., seconds, minutes) or very long. It is important to obtain or develop a detailed load profile including future possibilities.

Servers tend to be the most common equipment within a datacom space; there are many different types of servers and any type of server can go into any type of datacom facility. Although there are no set rules regarding what constitutes any specific server type, the following classifications help provide some guidance:

Standardized nomenclature defining the cooling airflow paths for datacom equipment have remained unaltered since 2004 (Figure 4). Most datacom equipment now uses the front-to-rear protocol. The exceptions are some legacy telecommunications equipment and some network switches. These may use a side-to-side protocol, or a mix of side-to-side-to-top and/or to-rear air flows, that are not shown. When airflow does not follow standardized protocol, special rack mountings and/or air deflectors may be necessary to achieve proper cooling in facilities designed predominantly for front-to-rear cooled equipment.

The increasing heat densities of modern electronics are stretching the ability to adequately cool the electronic components within servers with air. The trend to higher recommended inlet air temperatures, done with the goal of saving energy, exacerbates the problem. Liquid cooling is therefore becoming more prevalent.

Figure 4. Equipment Airflow (ASHRAE 2015a)

Liquid cooling is defined as the process where a liquid (rather than “fluid” air) is used to provide the heat removal (i.e., cooling) function. There are many different liquid-cooling solutions for datacom rooms. The most common implementations are

These definitions do not limit the cooling fluid to water. Various liquids could be considered for application, including some that could be in a vapor phase in part of the cooling loop.

Figure 5 depicts one example of liquid-cooled datacom equipment where a liquid loop internal to the rack is used to cool the components in the rack. In this case, the heat exchange is with a liquid-to-facility-water heat exchanger. Typically, liquid circulating in the rack is kept above dew point to eliminate any condensation concerns.

Most datacom facilities are well designed and are geographically located in areas with relatively clean environments. Therefore, they do not have significant contamination concerns. However, the overall cleanliness of the environment is only one consideration for the location of a datacom facility, so potential contamination is often not considered to be a major driver.

Figure 5. Internal Liquid-Cooling Loop Exchanging Heat with Liquid-Cooling Loop External to Racks

There are two types of contaminants: particulate and gaseous. Some datacom facilities may have harmful environments arising from the ingress of outdoor contamination. In some rare instances, contamination has been generated within the datacom facility itself.

If a datacom facility serving a critical application happens to be susceptible to gaseous or particulate contamination, the consequences could be severe. As a result, it is important to address the potential for contamination and mitigate the risk as much as practical.

A number of factors can result in an increased failure rate. Changes in solder type (lead based to lead free), and an ongoing miniaturizing of datacom equipment components increase the risk. Changes in datacom room temperature and humidity operating conditions combined with a lower priority consideration for the surrounding air quality are other factors of concern.

Contaminants can cause either electrical or cooling failures within datacom equipment. Electrical circuits typically fail in either an open or a shorted condition. Datacom equipment circuits are much smaller than normal power circuits, often with conductors smaller than a human hair. They are, therefore, more susceptible to damage, but they fail in a similar manner. Printed circuit boards use tiny copper wires (lands) and components are attached with silver solder, and the two common datacom equipment circuit failures are copper creep corrosion and silver creep corrosion.

Airborne dust contaminants can be detected from detailed visual inspections of filters in the air-handling systems. Gaseous contaminant presence may require seasonal or periodical monitoring and measurement through the use of copper and silver coupon testing in the datacom rooms. The coupons react when exposed to various gases, with the typical exposure period being around one month. A subsequent lab analysis of the coupons can quantify the level of contaminants present.

Filtration systems (particulate filtration or gas filtration units) can be used to mitigate the risk of contaminants in the datacom facility. More information on this topic can be found in ASHRAE (2015a).

The first edition of ASHRAE’s Thermal Guidelines for Datacom Processing Environments in 2004 created a common design point: the inlet temperature for datacom equipment. The 2008 edition expanded the recommended thermal envelope, and the 2011 edition increased the datacom class definitions from two to four, with wider thermal ranges. The fourth edition (ASHRAE 2015a) makes significant changes to the humidity ranges as well. All of these changes were made after a great deal of industry study and, in the case of the humidity changes, a major ASHRAE research study (Pommerenke and Swenson 2014). Important considerations include the following.

Recommended Environmental Range. To achieve both energy efficiency and equipment operating reliability and longevity, facilities must be designed to achieve, under normal circumstances, ambient equipment inlet conditions that fall within ASHRAE recommended temperature and humidity ranges. See Table 1 for this range, or use the process defined by ASHRAE (2015a).

Allowable Environmental Range. The allowable envelope is where datacom equipment manufacturers test their equipment to verify that it will function within those environmental boundaries. Typically, datacom equipment manufacturers perform tests before product announcement, to verify that products meet all functional requirements within this environmental envelope. This is not a statement of reliability, but rather one of functionality of the datacom equipment. In addition to the allowable dry-bulb temperature and relative humidity ranges, the maximum dew point and maximum elevation values are part of the allowable operating environment definitions.

Practical Application. Prolonged exposure of operating equipment to conditions outside its recommended range, especially approaching the extremes of the allowable operating environment, can result in decreased equipment reliability and longevity (server reliability values versus inlet air temperatures are provided in ASHRAE [2015a] to provide some guidance on operating outside the recommended range). Exposure of operating equipment to conditions outside the allowable operating environment risks catastrophic equipment failure. With equipment at high power density, it may be difficult to maintain the air entering the equipment within the recommended range, particularly over the entire face of the equipment. Reasonable efforts should always be made to achieve conditions within the recommended range. However, if these efforts prove unsuccessful, operation outside the recommended range, but within the allowable environmental range, is likely to be adequate, but facility operators may wish to consult with the equipment manufacturers regarding the risks involved.

Environmental Class Definitions for Air-Cooled Equipment. For any piece of datacom equipment to comply with a particular environmental class (ASHRAE 2015a), it must be able to reliably provide its full operational capabilities over the entire allowable environmental range, based on nonfailure conditions. The recommended and allowable ranges for each datacom equipment class are given in Table 1. The allowable environmental ranges for the four datacom equipment classes are illustrated in psychrometric format in Figure 6:

Dry-bulb temperature must be derated based on altitude for all classes. See ASHRAE (2015a) for more information on derating methodology.

Figure 6. Environmental Classes for Datacom Equipment Classes (ASHRAE 2015a)

The latest guidelines were developed with a focus on providing as much information as possible, so that datacom facility operators can maximize energy efficiency without sacrificing the reliability required by their businesses. This assumes that the designs enable them to take advantage of reduced energy operation.

For any piece of datacom equipment to comply with a particular environmental class, it must be able to reliably provide its full operational capabilities over the entire classification temperature range based on non-failure conditions.

For datacom equipment that meets the higher supply temperatures as referenced by the ASHRAE classes in Table 2, enhanced thermal designs are required to keep liquid-cooled components within the desired temperature limits. Generally, the higher the supply water temperature, the lower the cost of the datacom facility cooling solution.

For classes W1 and W2, the datacom equipment should accommodate facility water supply temperatures that may be set by a campus-wide operational requirement. In these cases, condensation prevention is a must.

Availability of datacom equipment rated for classes W3 to W5 is limited. It is anticipated that future designs in these classes may involve trade-offs between IT cost and performance. However, these classes allow lower-cost data center infrastructure in some locations, as well as significantly reduced operating expense.

Facility water flow rate requirements and pressure drop values of the datacom equipment vary. Manufacturers typically provide configuration-specific flow rate and pressure differential requirements that are based on a given facility water supply temperature and rack heat dissipation to the water. Conformance with the water quality requirements for each cooling solution is important to long-term reliability.

A power supply nameplate rating indicates the maximum power draw for the datacom equipment’s safety and regulatory approval. A nameplate rating does not represent actual power draw during usage and should not be used as a measurement of datacom equipment heat release.

Manufacturers that follow ASHRAE guidelines utilize a template for each product that tabulates heat release based on configuration and use. In addition, most major datacom equipment manufacturers have online tools that can provide even more specific and detailed heat release and airflow information. Obtaining realistic heat release and airflow information is critical to the datacom facility and datacom equipment communities for use in datacom facility planning and designs that are both ample and energy efficient.

Datacom equipment manufacturers compete to create equipment that balances power and performance based on the markets and workloads they are targeting. Datacom equipment is no longer one size and one configuration fits all. More IT departments are shifting to purpose-built servers in order to meet customers’ specific business needs. These purpose-built servers include specific features and components sized to meet a customer’s workload requirements. This requires facility power projections to comprehend the software workload being deployed. ASHRAE (2018) captures these power trends by select workload (Table 3) for typical and maximum equipment configurations.

A workload-based methodology provides a much more accurate estimate of actual power consumption in a modern data center, compared to using the maximum power for a given server family from a datacom hardware provider. An example of this trend methodology is shown in Figure 7 (for a 2U 2-socket [i.e., two central processing units] server configuration).

An important addition to the 2018 edition of ASHRAE’s IT Equipment Power Trends was the introduction of the power compound annual growth rate (CAGR) for the years 2016 to 2025. The CAGR allows the measured power of a server running a business’s specific application(s) to be used to project the future power for a similar workload demand. The most striking growth rates (Figure 8) occur in the scientific (4.6%) and analytics (5.9%) workloads at maximum expected configurations. These higher growth rates can be attributed to higher-power CPUs, maximizing the number of components, and the potential use of graphical processing units (GPUs) or another application-specific processor technology.

When appropriately applied, knowledge of datacom equipment power trends can be a powerful tool in considering what future loads might be in a facility or space. Future load is a critical component in the planning, design, construction, and operation of facilities to avoid ineffective expenditures, premature obsolescence, stranded cost or assets, energy waste, etc.

Refer to ASHRAE (2018) for details on how the trends were created, along with how to apply them.

Figure 7. ASHRAE Projected Power Trends for 2U 2-Datacom Hardware by Workload Type (ASHRAE 2018)

The goal of a good datacom facility cooling design is to match cooling capacity to actual heat load. This requires a correct and realistic assessment of the heat release of the projected datacom equipment. Even when actual datacom equipment is known, this can be challenging, and is often done incorrectly. A basic understanding of datacom equipment thermal design is therefore valuable, to comprehend how the datacom equipment interacts with the data center and vice versa.

The thermal design must ensure that the temperatures of all datacom equipment components (e.g., processors, memory, storage, I/O, power supplies) are maintained between the high and low limits of their specifications. Datacom equipment components have functional, reliability, and damage temperature specifications. Maximum functional temperature limits for silicon components are generally in the 185 to 221°F range.

The thermal management system (Figure 9) in the datacom equipment must take the appropriate actions to ensure compliance with these specifications. This ensures data integrity and maximizes equipment service life.

Figure 8. ASHRAE Power Compound Annual Growth Rate for Datacom Hardware by Workload Type and Size (ASHRAE 2018)

A well-designed thermal management implementation balances component temperatures, datacom equipment performance, humidity, and acoustics, to achieve reliable equipment performance with minimal power consumption.

Figure 9. System Thermal Management (ASHRAE 2016a)

Air-cooled solutions are currently the most common approach for datacom equipment. The information described here is applicable to most mainstream, air-cooled volume servers; however, the principles apply to most types of datacom equipment.

Typical datacom equipment relies on variable-speed, forced-convection cooling to maintain the required temperatures. Component temperature is driven by one of three factors in an air-cooled system (Figure 10):

Datacom equipment manufacturers develop component- and equipment-level cooling solutions that balance cost, performance, and energy consumption, including trade-offs between air movers and heat sink designs. Cooling performance, power consumption, acoustic signature, fan reliability, and redundancy features are also important characteristics that must factor into the overall solution.

Figure 10. Example Component in System and Rack (ASHRAE 2016a)

Fan or cooling zones are often used to precisely adjust specific fans to the needs of the components most coupled with those fans. Cooling zones can be proximity based or physically separated. By using a fan zone approach, total fan power and acoustic output can be minimized. Fans in a nonstressed zone can run at lower speeds than those in a more highly stressed zone.

Thermal control enables optimization of datacom equipment system performance as a function of usage or workload, configuration, cooling capability, and environment. Underlying this optimization is the use of fan speed control and power management operating in parallel. Optimization for differential air temperature ΔT through the datacom equipment is generally not a significant design consideration because of the more critical requirement of ensuring that functional limits are maintained.

Components and their specifications are the primary drivers in a server’s thermal design (e.g., heat sink, fan selection, airflow management).

Power management features enable all components to stay within temperature limits while minimizing overall power consumption during periods of low activity.

Highly advanced control algorithms vary the datacom equipment fan speeds and airflows, and tune the fan speeds based on the datacom equipment’s usage model. Multiple algorithms can be used simultaneously, with the final fan speeds determined by comparing the results of these algorithms.

Sensors (and proxy sensors) create the data necessary to trigger power management, and are the basis of a cohesive thermal management implementation.

As important as it is to control temperature within a data center to maintain high reliability, it is equally important to control moisture content. If both moisture and temperature are properly controlled, the result will be more reliable long-term operation, plus significant energy savings in the operation of the data center.

The effects of improper moisture control on a data center operation are twofold:

Based on Pommerenke and Swenson’s (2014) results, the recommended moisture limits have been greatly expanded in the fourth edition of the Thermal Guidelines for Data Processing Environments (ASHRAE 2015a). The recommended lower moisture limit has been significantly reduced, as shown in Table 1 and Figure 6 for the ASHRAE environmental classes. Note that the recommended upper and lower moisture limits are represented by dew point (dp) limits rather than relative humidity. Although static concerns are actually related to relative humidity, dew point is used because it is fairly constant throughout the data center, whereas relative humidity varies widely. Because dew point can be easily monitored and consistently controlled, best practice is to monitor moisture content in a data center using dew point rather than relative humidity.

With increasingly dense datacom equipment packaging, some components may require liquid cooling to maintain the environmental specifications dictated by the manufacturer.

Liquids considered for cooling electronic equipment are dielectric, engineered fluids, water, oils, or refrigerants. Heat transfer from the liquid-cooled datacom equipment or components to the datacom facility generally takes place through a liquid-to-liquid heat exchanger.

Some liquid-cooling solutions include immersion of the datacom equipment components directly in a dielectric fluid, in either single- or two-phase applications. Dielectric fluids include mineral oil and fluoroketones.

A datacom facility can be a dedicated building, or be part of a general purpose building that houses other business functions or tenants. Regardless of the type of building in which it is housed, or its location within a building structure, a datacom facility is comprised of a number of spaces having different but interrelated functions (see Figure 1).

The main computing area is identified by different names: computer room, machine room, raised floor area, or white space. This chapter uses the term datacom room to differentiate it from the other areas that support it, and which comprise the complete datacom facility.

Determining the appropriate size of a datacom room is more challenging than it has ever been. There are three major reasons: the ever-increasing demand for computing services; the consolidation, virtualization, and increasing density of datacom equipment; and the transition of many computing services to the cloud or to leased colocation facilities.

It is all too common to underestimate the amount of electrical/mechanical space required to support a datacom facility. The need for reliability in these critical facilities dictates a requirement for adequate maintenance space. Overcrowding, even if minimum legal or manufacturer-dictated service clearances are maintained, can lead to inadvertent interruption of one system while servicing another. A rule of thumb, to be used as a starting point only, is to base minimum electrical/mechanical support space requirements on a percentage of the datacom room area:

Every increase in reliability requirements also increases the need for more redundant pieces of equipment, which in turn requires yet more support space. Further, highly redundant facilities require physical compartmentalization of duplicate or parallel systems by fire-rated walls, further increasing support space requirements.

The structure enclosing a datacom room should provide good thermal separation from the surrounding areas, whether those are exterior or interior spaces. The primary concern with the overhead structure, regardless of its construction or intended use, is that it not be a source of particulate contamination or water leakage.

The overhead structure must be cleanly finished and sealed to avoid concrete or insulation flake-off. If it is a roof structure, take extra precautions to preclude leakage. In highly critical spaces, a double roof structure is often used for insurance. Gaps and joints should be caulked.

Suspended ceiling tiles must be either metal pan or plastic encapsulated on both sides to prevent flake-off. This is particularly important when the above-ceiling plenum is used to convey return air. Cut edges must be sealed with spray paint or similar. Any suspension rods that penetrate the tiles should also be sealed at the penetrations. Metals used above a return air plenum ceiling should be either hot-dip galvanized or of a type that will not grow zinc whiskers.

Walls surrounding a datacom room should be well insulated to avoid both cooling loss and heat infiltration. All cracks should be sealed, which is mandatory if the room is also protected by a gas-based fire protection system.

Although most datacom equipment can accept a broad range of allowable humidity levels, consider installing vapor barriers for datacom spaces. Avoiding condensation anywhere in the room is very important.

Windows should generally be avoided in a datacom room, but if they exist or are somehow necessary, they should be double-glazed and sealed. If covering the windows is allowed, but replacement is possible from the inside only, it will be necessary to make the coverings removable and to avoid blocking access with large pieces of mechanical/electrical equipment.

Although raised access floors are still used in many datacom rooms, they are no longer a standard requirement. It is not only possible, but now relatively common, to put the entire power, cooling, and network infrastructure overhead, particularly when close-coupled cooling is used. In these designs, the raised floor is not necessary to convey air.

However, with the amount of piping often used to service in-row, rear-door, and direct water cooling, raised floors are often used anyway to avoid concerns about overhead water, as well as to minimize congestion above cabinets. When power, cable tray, and lighting are all run overhead, the vertical space can become congested and difficult to coordinate. Three-dimensional modeling of the space is highly recommended when overhead infrastructure is used, to avoid both installation conflicts and long-term operational difficulties.

There are several advantages and disadvantages to using raised access floors, regardless of their purpose. Once it is determined that a raised access floor will be used, several factors should be considered in its selection and design.

The most obvious advantages of raised access floors are to provide a space for permanent infrastructure such as power, piping, and cabling, but raised floors have also been historically used to convey cooling air through the plenum space. For slab variations too large to be leveled with patching, and unrealistic for self-leveling cement, they can also provide a level floor.

However, a raised access floor adds total weight to the structure. It must also be maintained, which includes releveling every few years, particularly if technicians do not take care in replacing tiles where they were removed, or open too many tiles in a row and destabilize the floor. The plenum space can also become a tangle of wire and cable if care is not used in installing new cable and removing old. If the floor is used to convey air, masses of unmanaged cable can reduce or totally block airflow. If cables are located in a raised floor plenum that is also used to convey cooling air, best practice is to run cables parallel to airflow, and to provide overhead cable pathways for ad hoc cable installation. It is even better not to locate cables in an air-plenum floor space at all.

The height of a raised access floor is determined by its purpose. If it is used to convey cooling air, it must be high enough to deliver the required air quantity while maintaining the necessary static pressure as evenly as possible across the floor area. Computational fluid dynamics (CFD) modeling is generally recommended to confirm air flow patterns and adjust cooling designs to maximize cooling effectiveness, particularly under failure-mode scenarios where redundant cooling systems are used. Further information on CFD modeling is provided in the section on Computational Fluid Dynamic (CFD) Analysis.

Piping, power systems, or cable tray that will also occupy the space must be taken into consideration in determining the floor plenum height and its effect on air flow. It is generally accepted today that a raised access floor used to convey cooling air needs to be at least 24 to 30 in. high to be effective, and that even higher is better.

After height, the biggest consideration is floor structural strength. Raised floors for datacom rooms should use bolted stringer substructures to increase load capacity and to make it easy to remove and replace tiles without destabilizing the floor.

Newer computing equipment cabinets are usually rated for 2500 to 3000 lb., which is significantly higher than most legacy cabinets. Even if they are not full of the heaviest available equipment, floor loading must be planned as if they will be to address potential maximum loading in the future.

However, cabinets with these load ratings also tend to be larger than legacy 24 by 24 in. cabinets, so the load is spread over more than one floor tile. It is not unrealistic to specify raised-floor systems designed for 100 psi or 3000 lb rated tile capacity. Any abnormally heavy equipment can be supported with supplemental floor pedestals under the tiles, so long as the slab structural strength is sufficient for both the total and point loads.

In selecting floor strengths, it is particularly important to examine the rolling load characteristics along with the static structural ratings, because equipment must often be moved into position on small integral wheels. The rolling load tests are generally performed for 10 passes and 10,000 passes with weight on test wheels of particular sizes in accordance with testing methods established by the Ceilings & Interior Systems Construction Association (CISCA) (CISCA 2007). However, not all tests are performed with the same wheel sizes, and the results can be misleading.

It is always safest to put thick hardboard or plywood over the floor when moving particularly heavy loads. This is especially important when rolling equipment through cold aisles with perforated airflow tiles, many of which do not have a rolling load specification at all and are easily deformed.

The most common surface material for raised access floor panels is high-pressure laminate (HPL). This material holds up well to heavy rolling loads without deforming or cracking, has good static dissipative characteristics when properly bonded to a grounded surface, is available in light colors to maximize lighting effectiveness, and is easy to maintain with damp mopping.

Heavy scrubbing, buffing, or waxing should never occur in a datacom room. This precludes the use of vinyl composite tile (VCT), pure vinyl tile (which can also be easily deformed under rolling loads), or linoleum (which is also too easily damaged). Carpeting, of course, should never be used in a datacom room, even if it is antistatic, because it both accumulates and generates particulate contaminants. (Note: it is generally accepted that the ground resistance of an installed raised-floor panel, when properly connected to a robust grounding system, should be in the range of 10⁴ to 10⁶ Ω to minimize any potential for static generation.)

One of the most challenging decisions in selecting materials for air plenum raised access floors is the airflow panels. A range of types is now available, including legacy perforated tiles (25% open), grate-style cast aluminum tiles (56 to 63% open), and tiles incorporating directional vanes, air boost fans, and automatic air flow control. (See further discussion of airflow tiles in the section on Underfloor Air Delivery).

All air plenum raised floors leak air, and because cool air is expensive to produce and requires considerable fan energy to distribute through the plenum, this wastes energy. A good-quality raised-floor installation should leak no more than 2% air.

For datacom rooms designed without a raised access floor, the primary concern is that all power, cooling, and network infrastructure must be routed overhead. Depending mainly on the cooling method used, this can create a congested overhead space that requires careful design coordination and exacting installation. As noted previously, 3D modeling techniques are highly recommended when designing complex overhead systems. It should also be recognized that the cost of overhead infrastructure, particularly if extensive ductwork is necessary, can be very similar to the cost of a raised access floor.

Space must be allocated within a datacom facility for storing components and material, support equipment, and operating and servicing the datacom equipment. Some ancillary spaces may require environmental conditions comparable to those of the datacom equipment, whereas others may have less stringent requirements. Continuous operation of some support spaces is often vital to the facility’s proper functioning.

Electrical power distribution equipment can typically tolerate more variation and a wider range of temperature and humidity conditions than datacom equipment. Equipment in this category includes incoming service/distribution switchgear, switchboards, automatic transfer switches, panel boards, transformers, and standby generators. Manufacturers’ data should be checked to determine the amount of heat release and design conditions for satisfactory operation. Further information and guidance can be found in IEEE Standards 446 and 1100.

Uninterruptible power supplies (UPSs) come in various configurations, but most use batteries as the energy storage medium. They are usually configured to provide redundancy for the central power buses, and typically operate continuously at less than full-load capacity. They must be air conditioned with sufficient redundancy and diversity to provide an operable system throughout an emergency or accident.

UPS power monitoring and conditioning (rectifier and inverter) equipment is usually the primary source of heat release. This equipment usually has self-contained cooling fans that draw intake air from floor level or the equipment face, and discharge heated air at the top of the equipment. Air-distribution system design should take into account the position of the UPS air intakes and discharges.

Installation of secondary battery plants as a temporary back-up power source should be in accordance with IEEE Standard 1187 and NFPA Standard 70. Refer to other applicable standards, in addition to a design review with the local code official. Other relevant sources of guidance are NFPA Standards 70E and 76.

Several types of batteries are used with UPS systems. Flooded lead-acid (wet cells) are generally considered the longest lasting, but also present the highest initial cost. They require special rooms with, among other things, containment for possible acid spills, deluge shower and eye wash stations, as well as hydrogen gas detection and exhaust fans (IEEE Standard 484). It is important to locate the battery room close to the UPS room to minimize loop current losses in the large DC conductors. More commonly used today are valve-regulated lead-acid (VRLA) batteries, also known as sealed cells or maintenance free. These can be colocated with the UPS system, which can be in the datacom room (IEEE Standard 1187). However, if the ambient temperature in the datacom room is not appropriate for the batteries, battery life may be adversely affected. VRLA batteries are available in different qualities, but those usually supplied with a UPS system may have a service life of only three to five years before replacement is required, depending on usage conditions.

The newest battery used with UPS systems is the lithium-ion (Li-ion). There is still very limited history with these batteries in this type of service, but they are promoted as having significantly longer service lives than VRLA, lower weight, and less stringent operating conditions. However, probably due to the negative publicity associated with cellphone and other small-device battery explosions, local codes may preclude their use, even though the chemistries, case constructions, and reliability testing of Li-ion batteries intended for commercial use are all very different than for the ultra-compact batteries that have exhibited problems.

Temperature in a battery area is crucial to the life expectancy and operation of the batteries. The optimum space temperature for lead-calcium batteries is 77°F. Per IEEE Standard 484, if higher temperatures are maintained, it will reduce battery life; if lower temperatures are maintained, it may reduce the batteries’ ability to hold a charge. Recommended ambient temperatures for other battery types should be verified with the battery manufacturers.

Engine-driven generators used for primary or standby power typically have air-cooled radiators and require large volumes of outdoor air when running. Designs should ensure that engine exhaust air does not recirculate back to any building ventilation air intakes. Commonly, up to 72 h of fuel oil storage is required, so fuel oil storage tanks and distribution systems need to be integrated into the overall facility design and planning. The governing codes often mandate specific requirements for containment, location of fuel oil storage, fire resistance ratings, etc. However, as has been unfortunately demonstrated in several weather disasters, local codes may impose requirements that can negate the benefits of generators unless all potential conditions are taken into account in the designs. Fuel tanks are heavy when full, but will start to float in a flooded basement as fuel is used, breaking the pipes. Both fill and pressure relief pipes must be located high enough above ground to remain both accessible and impervious to flood waters. Generators themselves must also be carefully located and protected from potential problems.

Fire Protection. Datacom fire protection involves a combination of strategies starting with prevention and continuing through detection, suppression, and response to a fire event. The National Fire Protection Association (NFPA) has several standards addressing design, installation, maintenance, and operation of fire protection systems in datacom facilities. Worldwide, additional fire protection standards may apply as well; consult local governments. Major NFPA standards include the following:

NFPA Standards 75 and 76 offer both prescriptive and performance-based approaches. Most designers defer to the prescriptive path, but a growing number of firms provide performance-based designs. These offer more flexibility, and can be tailored to a company’s specific risk and business models. As another alternative, some companies apply provisions from both standards, and often exceed one or more portions of either standard based on their own risk assessments or experiences.

There are several options for providing fire suppression in datacom rooms. Many older (and even some newer) datacom facilities use a code exemption to suppression. More commonly, however, datacom facilities are equipped with either a sprinkler or gaseous suppression system for a combination of life, structure, asset, and service protection. The conventional wisdom, invoked by many code authorities, is that gas protects equipment but sprinklers protect people and structures. One thing to be aware of with inert gas fire suppression systems is that, in some cases, the discharges can cause temporary or permanent failures to hard disk drives with rotating storage media due to extreme acoustic levels and the resulting acoustically driven vibration. This can usually be mitigated with the proper selection of gas discharge heads, although this may also increase the time for a discharge to extinguish a fire.

Air containment, either hot aisle or cool aisle, has become a common method of improving cooling performance and reducing energy usage. However, when containment systems are retrofit or designed into a new facility, effects on the required detection, suppression, release system, materials of construction, and prevention of fire must be considered. These important considerations are addressed in detail in the NFPA standards. The added obstructions often necessitate modifications to the suppression systems (sprinklers or gaseous agent nozzles) to ensure proper suppression release and dispersion. An alternative to suppression system changes can sometimes be partial containment, which has been shown to be as much as 80% as effective in improving air control, but does not block existing sprinkler or gas discharge heads. (See the section on Containment.)

Water Concerns. Water damage is always a concern in a datacom facility. It is best to locate the room above grade if possible, but this is not always practical.

There are other sources of water leakage as well: designs must consider the possibility of leaks from overhead. Datacom rooms and supporting electrical equipment should not be located below bathrooms, pantries, laboratories, or the like. If unavoidable, the space above should have waterproof membrane floors. Liquid piping should also be routed around the datacom room, but if this is not possible, should be provided with drip pans and leak detectors. Leak detectors should also be provided anywhere water can infiltrate, particularly if it could affect electrical infrastructure. A common problem in buildings not specifically designed for high-availability datacom facilities is primary power switchgear and bus duct terminations in the lowest level of the building, where they are subject to flooding. These conditions cannot likely be changed in existing buildings, but should be avoided in purpose-built facilities.

Acoustics. The rapid increase in density and power draw of datacom equipment has brought with it commensurate increases in required cooling. Air cooling requires substantial volumes of air movement, which generates sound levels that can be problematic for worker health and might require a hearing protection plan. Increased sound pressure levels required by increasing datacom fan speeds may also lead to reduced hard disk drive (HDD) performance, due to acoustically driven vibrations.

Sound level exposure limits in datacom rooms and their associated mechanical/electrical plant facilities are governed in the United States by the Occupational Safety and Health Administration (OSHA [Annual]) in Code of Federal Regulations (CFR) 1910.95. Similar regulations exist in other countries.

Sound emissions from heat rejection equipment (cooling towers and/or air-cooled chillers) as well as emergency and/or prime power-generating equipment for the datacom facility’s mechanical/electrical plant must also be considered. Noise generated outside the building, typically from rooftop chillers and cooling towers, must also be mitigated so that sound levels in the building are conducive to conducting normal business activities.

Community sound levels, mostly from exterior heat rejection and power-generating equipment, must typically comply with state, regional, or local noise codes, ordinances, guidelines, and/or regulations. Community sound level limits are typically cited at property lines and/or anywhere on the property of a potential complainant.

Sound levels of exterior equipment during normal, emergency, and test operation should allow for relatively easy communication among service personnel, as well as auditory awareness of vehicle and general service activities in the area. A sound level at or below 70 dBA in service areas and equipment yards, with all equipment operating, is an ideal goal.

Vibration. Vibration levels in datacom facilities must be considered as well. The greatest vibration concern in datacom installations is usually roof-mounted support equipment, such as air handlers, cooling towers, chillers, and generators, although similar equipment mounted inside the building can also create vibration issues. See the ASHRAE TC 9.9 datacom book series, especially ASHRAE (2008a), for additional information.

Some datacom equipment, such as very high-density disk drives, can be sensitive to vibration. Even vibration induced within a server (e.g., by fans) can be an issue for disk drives at the higher airflow rates that are required to cool some components. Wherever there is concern, vibration specifications should be obtained from manufacturers, and the datacom facility floor’s vibration dynamics studied to determine compliance with vibration limits.

Many locations are considered seismic zones, requiring special bracing and safety restraints for much of the infrastructure. However, the critical nature of many datacom facility operations mandates consideration of special structural supports and restraints, even where seismic regulations are minimal or do not exist.

As described in ASHRAE (2008a), it is important for both the owner and the designer to understand the potential hazards, including seismic, wind, etc., of the region where the facility is located.

Clear operational criteria should be established and used in system designs. These may include recommendations for structural restraints and bracing beyond what is required by law. Local code requirements must be identified and understood, as well as the requirements set forth in ASCE Standard 7, which provides further information and direction.

Lighting. In datacom rooms, lighting should usually be centered in the aisles, not over equipment cabinets or cable trays where much of the light energy would be wasted. Fixtures should also be suspended 8 to 9 ft above the floor so as to deliver maximum illumination over the heads of technicians and into cabinets. Higher mountings may be necessary to clear other overhead infrastructure, but this disperses more light energy over the tops of cabinets and other obstructions. Although lighting is a small part of datacom room energy consumption, and LED has become the light of choice, proximity sensors should be still used to ensure they are not left on when there is no activity.

The photometric curves of many architectural luminaires are inappropriate for datacom facility lighting. Fixtures with wide horizontal dispersion patterns are needed. This requirement is very similar to the lighting of library book stacks, where the purpose is more to support the reading of titles on the books than to provide for reading books in the aisles. An illumination level of 30 footcandles on the vertical surfaces of cabinets is generally sufficient.

Lighting systems and lighting control in datacom rooms can be provided by several different methods. All systems require sensors, and automatic control devices are recommended. Low-voltage (0 to 10 V) or power over Ethernet (PoE) controls are a good application for LED lighting control in data centers. PoE lighting controls work well in data centers because the data racks are typically already in place and the power and control wiring is run through Ethernet cabling. However, if a PoE lighting solution is chosen, larger racks, additional power supplies, and controls may be needed.

Emergency lighting for PoE systems also should be considered. UL 924-listed battery packs are now available that can be added to any PoE lighting fixture to make it into an emergency fixture. This is an advantage over conventional lighting, which requires selected fixtures to be designated and provided as emergency luminaires. Whichever emergency system is decided on, it must meet local requirements for illumination and run time in the event of loss of normal power.

PoE lighting and controls can offer flexibility, low initial installation and operating costs, and allow customers to monitor lighting and energy usage with a centrally hosted system control. PoE systems provide reliability, scalability, and flexibility to support easy modification or expansion within a datacom room.

It is axiomatic that redundant systems should improve the reliability of a facility, but how much redundancy is justified is always a question. Redundancy decisions must consider business and operational needs as well as economic justification. Unfortunately, redundancy alone does not guarantee increased reliability.

It is not uncommon for large investments to be made in duplicate power and cooling systems that have been configured or installed in ways that defeat or greatly compromise their purposes. Consequently, a careful analysis of all possible failure modes should be an integral part of the design phase of any datacom facility.

The primary goal of redundancy should be to provide for concurrent maintainability. This requires a design that allows any element in the power and cooling infrastructure to be shut down and removed from service for maintenance without compromising the computing systems that depend on that infrastructure. This level of redundancy is commonly known as N+1, meaning that every system has at least one extra component and pathway.

Higher levels of redundancy require some degree of duplicate systems, such as two identical and fully load-sharing chiller plants with duplicate piping systems. This is known as 2N redundancy. An even more stringent design would have duplicate systems, but with additional redundant components in each. Depending on how the additional redundancy is configured, the systems may be known as 2N+1, in which an additional unit (e.g., a chiller module) is made available to either of the duplicate systems, or 2(N+1) in which both redundant systems each have their own redundant modules. Several methods have been developed to classify levels of redundancy and their resulting reliabilities and uptimes (e.g., ANSI/TIA Standard 942-B).

It is standard practice to power datacom equipment from an uninterruptable power supply (UPS). UPSs have two main purposes: to isolate the datacom equipment from power line disturbances; and to maintain ride-through power to the datacom equipment until back-up generators start, or long enough to accomplish an orderly shutdown.

With today’s heat densities, datacom systems cannot be maintained for very long on UPS alone. The usual maximum back-up time is 15 to 20 min, before thermal rise causes a shutdown of the datacom equipment and/or the UPS. High-performance computers may shut down in minutes or even seconds if cooling is interrupted. It is therefore necessary to have a means of maintaining cooling for the most critical systems until either generators start, or systems can be shut down properly.

It is generally impractical to run large cooling systems on UPSs. If this must be done, the cooling equipment’s electrical characteristics make it prudent to use a separate UPS. Further, the substantial power draws and high in-rush currents on compressor start-up and cycling require large and expensive UPS systems.

In most datacom rooms it is not necessary to maintain full cooling for an extended period. If cooling can be continued to the most critical computing systems, this should suffice until both generators and full cooling restart. If a chilled-water system and close-coupled liquid-based cooling have been selected, this can be relatively easy to accomplish. There may be sufficient residual water in header pipes to cool critical systems for several minutes. If not, additional water can be stored in tanks.

Long battery life is of no value if the UPS it supports is without cooling. A UPS generates substantial heat under load, as do the batteries when they take full load after a power failure. Batteries also emit heat as they recharge once the generators start. This heat generation should be considered when choosing the location for the UPS, which is often relegated to a location that is less desirable for use as personnel spaces. This is sometimes in an electrical or mechanical room that generates additional heat, in a corner of a parking garage, or even in a roof penthouse that is exposed to high sun loads. These kinds of locations can dramatically shorten the actual back-up duration of the UPS, particularly if the batteries are also exposed to continuous heat. The cooling system in the UPS room should have the same level of redundancy as the cooling for the datacom room.

Datacom equipment rooms can be conditioned with a wide variety of systems, including packaged computer room air-conditioning units and central-station air-handling systems. Air-handling and refrigeration equipment may be located either inside or outside the datacom equipment rooms.

The following system configurations are some of the most commonly used solutions to providing sufficient cooling to air-cooled datacom equipment.

Computer Room Air-Conditioning (CRAC) and Computer Room Air-Handling (CRAH) Units. Despite the development of a variety of newer cooling technologies, CRAC and CRAH units remain the most common datacom cooling solutions. They are specifically designed for datacom equipment room applications and should be built and tested in accordance with the requirements of ASHRAE Standard 127.

CRAHs are special-purpose chilled-water air handlers designed for datacom applications. CRACs are compressorized cooling systems and are available in several configurations, including direct expansion (DX) air-cooled, DX water-cooled, and versions that include a water- or refrigerant-cooled economizer coil. Both CRAH and CRAC units are available in either downflow or upflow designs. Downflow units are used primarily for underfloor air delivery and have top air returns. Upflow units discharge air overhead, often into ducts, and can have either front or rear air returns. Whereas older CRAH/CRAC units use belt-driven forward-curved centrifugal supply fans (and are often constant volume), newer models tend to use plenum-style plug fans, which are direct drive and paired with electronically commutated motors and variable frequency drives (VFDs) for speed control. The limited static pressure available from these newer computer room units means they are not typically as suitable for ducted applications, so are generally located in or immediately adjacent to the datacom space. As a result, they often have limited flexibility for incorporating air-side economizer solutions.

CRAC and CRAH units are usually located in the datacom equipment room, but may also be in mechanical galleries adjacent to the datacom room, or installed remotely and ducted to the conditioned space. Ducted designs require consideration of the relatively low-static-pressure designs of CRAC and CRAH units with plug-type fans. Ducted designs may require conventional forward-curved fans that can work against higher static pressures, but variable-speed motors can still be used to improve energy efficiency.

If CRACs or CRAHs are used in datacom rooms without a raised floor, or in rooms with a raised floor that is not used for cooling, overhead air delivery is required, which means using upflow cooling units. The return air grills on these units are at the bottom, so cannot efficiently capture hot return air in an open space. When upflow cooling units are used, it is necessary to also use cold aisle containment to locate the cooling units in mechanical galleries separated from the datacom equipment by a demising wall, or to use ducted rear returns to efficiently get hot return air back to the units and prevent them from re-entraining their own cool discharge air.

With either placement, temperature and humidity sensors must be located to properly control air delivery in order to keep inlet air conditions to the datacom equipment within specified tolerances.

Centralized Air-Handling Systems. Traditionally, many telecommunications central office facilities and datacom facilities with overhead air delivery used central-station air handlers. Larger, centralized air handlers, typically either roof mounted or adjacent to the datacom space, have been gaining popularity as air-side economizer-based solutions have become more common. These air handlers may include DX cooling coils, chilled-water coils, adiabatic cooling sections, and indirect economizer solutions (such as air/air heat exchangers). Larger air handlers may use a fan array consisting of multiple direct-drive plug fans.

Control of Variable-Speed Fans. There are several ways to control fan speed. The most common are underfloor pressure, cool-aisle containment pressure, differential pressure, supply air temperature, and return air temperature.

Traditionally, telecommunication spaces had no raised floor and used overhead ducted air delivery, whereas datacom facilities used raised-flooring systems as supply air plenums.

Underfloor Air Delivery. The interstitial space under the raised floor creates a large-volume air plenum that, if properly configured, can deliver relatively uniform air pressures across the entire room area. However, because the floor plenum is also often used for piping, power and cable, there are many potential obstacles to airflow that can be challenging to mitigate.

Underfloor air delivery to cold aisles is provided and balanced using a range of airflow tiles, which are available in 25% open, 56 to 63% open, and fan boosted, and both dampered and undampered. Even distribution of air through the airflow panels is a function of the evenness of the static pressure below the floor. However, the pressures are altered by the existence of the airflow tiles. Therefore, though it may be tempting to use high-airflow tiles to ensure sufficient air delivery to all cabinets, quantity and location must be balanced with the available air volume and static pressure. As with any fluid, air will take the path of least resistance, so too many of these tiles in one area can result in air starvation for equipment in other areas. Likewise, tiles with integrated air-booster fans may solve a spot cooling problem, but because fans will take the air they want, adjacent cabinets may receive less than they require, resulting in unintended additional cooling problems.

Tiles with variable dampers are available to enable adjustment of airflow to match the requirements in each part of the floor. However, the addition of a damper to any airflow tile results in reduced airflow and cooling capacity, even when the damper is 100% open. In short, air balance with underfloor systems can be challenging, and specifications should be carefully examined when selecting from the wide range of air flow tiles now available.

Leakage between the floor tile joints will reduce the expected airflow. Likewise, air will leak and be wasted through unsealed gaps at the raised-floor-to-wall or raised-floor-to-column junctions, and through any floor cutouts for cable or chilled-water lines from the raised-floor plenum that are not correctly sealed. A properly installed and maintained floor should leak no more than 2% of the delivered air.

Overhead Air Delivery. Delivering air overhead requires ducts large enough to convey the air volume needed to cool the equipment in each aisle, at velocities and pressures that enable air flow to be easily adjusted and balanced in each aisle. To properly control airflow to match loads in an aisle, it is important that one duct runout serve only one cold aisle, even though planning where runouts should be to accommodate future cold aisles and rows of IT cabinets may be challenging. Irrespective of planning difficulties, overhead air delivery may still provide more effective airflow management than underfloor air delivery because overhead air volume can be tied to the measured temperatures in each aisle. This means the system can dynamically increase or decrease airflow to each cold aisle in response to measured conditions.

As noted above, CRAH and CRAC units are not easily configured for higher-static-pressure systems or for systems with outdoor air economizers. It is, therefore, more common to use central station air-handling systems when designing for overhead air delivery.

Effects of Air Mixing. Air mixing occurs in two ways: (1) when hot air discharged from computing equipment recirculates back to the air intakes, thereby increasing the inlet air temperature at the computing equipment; and (2) when cool supply air bypasses the computing equipment and mixes with hot discharge air, thereby lowering return air temperature. Reduced return air temperature decreases the cooling capacity of the air-conditioner coils by decreasing the system ΔT, which requires an increase in airflow to meet the load. If air paths through or between the datacom equipment racks exist, then some of the cool supply air will bypass the datacom equipment, and some of the discharge air will recirculate to the front equipment intakes. Use blanking or filler panels to minimize air mixing.

If the supply air temperature has been set toward the upper limits of the ASHRAE recommended envelope, hot-air recirculation may result in equipment seeing inlet air that is warmer than the design temperature. Avoiding or minimizing air mixing requires separating the supply air from the return air and the datacom equipment intake air from the datacom equipment discharge air. The more complete the separation, the more effective and energy efficient the cooling system will be.

Hot Aisle/Cold Aisle. The first step in avoiding air mixing is to arrange cabinets in hot aisle/cold aisle configuration. This means that racks and cabinets are installed facing back-to-back and front-to-front. This arrangement keeps the hot-air discharge from one row of cabinets from directly entering the intakes of cabinets in the next row. This, of course, assumes that all datacom hardware has been designed with industry-standard front-to-rear airflow. Equipment that uses a nonconventional airflow pattern must be dealt with using special racks, cabinets, and air deflectors, as discussed in the section on Datacom Equipment Racks.

Containment. Containment further segregates the supply and return airflow paths by preventing mixing at the top of the equipment racks and at the ends of equipment rows. There are several types of containment, including hot-aisle containment (HAC) and cold-aisle containment (CAC), either of which can be full or partial; as well as rack-based containment, commonly associated with active or passive chimneys. These main types of containment are illustrated in Figure 11.

One of the main challenges to maintaining the high availability required for datacom rooms is delivering cooling effectively and efficiently to all the equipment, wherever it is in the room. Complexities created by widely variable heat densities, plus the disruptions to airflow patterns created mainly by underfloor obstacles, make it difficult to envision air movement in the space.

Figure 11. Examples of Main Types of Containment

CFD simulations are a useful tool for predicting actual cooling performance. This requires building a 3D computer-generated model of the datacom room. Of most practical importance is the way in which the user defines the space (and the equipment in it): a model is only as good as the input data, regardless of the program’s sophistication. A CFD model needs to represent the physical room geometry, and anything that might add or stimulate airflow or heat transfer, such as fans and vents. It must also include items that impede airflow, such as underfloor pipes and cables, and interactions with the surrounding environment, such as columns and oddly shaped walls and cabinets (the boundary conditions).

Several simulations are commonly completed for datacom rooms. These may be based either on assumed datacom equipment layouts and projected heat densities, or on actual datacom equipment installations, and often include

Although CFD is a powerful tool, it is also easy for it to be misapplied and misinterpreted. Data centers are complex, and infrastructure and equipment must be simplified for models to be practical. It is critical, therefore, that the modeler understands the key elements of the data center and the fundamentals of CFD modeling for the application of CFD to be successful. Proper use of CFD can identify and help the designer avoid most major cooling problems before construction occurs, but expecting the model to be a 100% accurate representation of the finished installation is not realistic.

In conceptual design of most enterprise facilities, the modeler will probably not know detailed information about the datacom equipment type or detailed configuration. Similarly, the precise locations and routings of cables and other physical infrastructure may not be known, and even the cooling system manufacturer or model may not yet have been selected. In such instances there is little point in putting excessive detail into the model, but at the same time the modeler must interpret the results accordingly: that is, understand that the predictions are limited to high-level system design decisions and recognize that performance will likely be a best-case solution because best practice has been assumed.

Where real facilities are being modeled, the models need to be more representative of the actual installation. This normally means basing the model on a physical survey of the facility, infrastructure, and datacom equipment configuration. Even so, the real infrastructure and equipment cannot be represented in ultimate detail. For example, a bundle of cables will be represented by an approximate obstruction or resistance to airflow rather than explicitly modeling each and every cable.

To ensure that judgments are made appropriately and that the model is accurate, compare simulation results with measurements of airflow and temperature. This is generally regarded as a calibrated model, because actual conditions can be measured and compared. Even this will have discrepancies from the reality, which will be difficult, if not impossible, to resolve, but at least they will be known. Then, and only then, should the model be used for sensitivity studies to upgrade the facility, troubleshoot problems, or make deployment decisions.

Although CFD’s primary focus for datacom facilities is determining the effectiveness and efficiency of cooling delivery to the computing equipment, it can also be used to analyze such things as airflow around air-cooled chillers, generators, and other critical equipment.

CFD is also a recommended component of The Green Grid’s (www.thegreengrid.org) most recent performance assessment tool: the Performance Indicator (PI). The PI is an extension of the PUE metric (see the section on Energy Efficiency) and examines a composite of energy efficiency (PUE), thermal conformance, and thermal resistance. Each of these parameters can be optimized in different ways. The PI illustrates them in a spider diagram format as an aid to achieving an efficient and cost-effective balance among the variables. Linking the PI parameters to a calibrated CFD simulation gives a clearer picture of how cooling is performing in the room, and allows scientific analysis of which physical and operational changes will deliver the most effective improvements.

Liquid-cooling equipment may be integrated with a facility-level cooling system in various ways, including the following.

Modular Room-Based Systems. The most common liquid cooling requires that facility chilled water be delivered to a heat exchanger (often called a cooling distribution unit [CDU]) located in or adjacent to the datacom room. The CDU has piping that connects to the datacom equipment; this is called the technology cooling system (TCS). The TCS connections may be to a centralized heat exchanger at the datacom equipment rack or may connect with the datacom equipment itself (e.g., multiple connections per rack). An example of this configuration is shown in Figure 12.

The fluid in the technology cooling system may be chilled water, deionized water, refrigerant, or other liquids. The cooling distribution unit typically also contains pumps, valves, temperature monitoring and control, and operating software. Refrigerant-based systems have many of the same components as well as compressors and/or pumps and related control components. One of the most important functions of the CDU, or whatever alternative distribution and control mechanism may be utilized, is to maintain coolant temperatures above the dew point. It can be easy to create condensation with liquid-cooled systems.

Figure 12. Typical Liquid Cooling Systems/Loops Within Datacom Facility (ASHRAE 2015a)

It is important to understand that many liquid-cooled datacom equipment solutions are not entirely cooled by liquid. Often, the datacom room needs to support a hybrid of air cooling and liquid cooling. A potential advantage to these systems is cooling ride-through in the event of primary cooling system failure. The residual liquid in header pipes can often be sufficient to maintain critical system cooling until generators start, and full cooling can be restored with only the addition of small supplemental pumps on UPS back-up. Chilled water or ice storage can also be used to supplement the residual capacity.

Direct Component Liquid Cooling. This type of system delivers the cooling medium directly to the individual datacom equipment, and often straight to the components. These systems are typically used in high-performance computing (HPC) or supercomputing platforms and have limited applications for typical commercial installations. They require completely dedicated piping distribution installations, as well as specialized heat exchangers, and related components between the liquid cooling equipment and the facility climate control systems.

Immersion Cooling. In this type of system, the datacom chassis are fully immersed in a liquid bath. The cooling medium completely surrounds the devices, and circulates through the datacom enclosures or individual chassis subsystems. The pumped fluid transfers the heat to a dedicated coolant-to-water heat exchanger, which is connected to the facility chilled-water loop. Because of the thermal mass of the liquid vat, these systems can often “ride-through” a cooling failure with little or no supplemental circulation.

Facility water distribution systems that serve datacom equipment should be designed to the same standards of quality, reliability, and flexibility as other datacom room support systems. This means that it is important to configure systems so they can be expanded or modified as needed to accommodate changes in datacom equipment without needing extensive system shutdowns. Further, the effects on the distribution system when valves must be serviced should be considered.

Figure 13 illustrates a looped chilled-water distribution system with sectional valves and multiple valved branch connections. The branches could serve air handlers or liquid-cooled datacom equipment. The valves allow modifications or repairs without a complete system shutdown.

Additional piping concepts are detailed in ASHRAE (2015a).

Figure 13. Example of Chilled-Water Distribution Piping System

WUE is a site-based metric, developed and popularized by The Green Grid, to assess the water used on site for operation of the data center. It presents a comprehensive evaluation of water usage in a datacom facility, where it is affected by a range of factors such as location, IT load, quality of available water source, type and efficiency of cooling equipment, and humidification loads. The formula for calculating WUE is the annual water usage (in litres) consumed by the entire datacom facility, divided by the IT equipment energy usage (in kWh).

PUE is an efficiency metric developed and popularized by The Green Grid. Since the concept was introduced (see, e.g., Rawson et al. [2007]), the metric has been revised to make it more understandable and the methods and reporting of measurement numbers more reliable, culminating in the 2014 release of a joint TGG/ASHRAE TC 9.9 publication (ASHRAE 2014b).

PUE measures how effectively an operating datacom facility delivers energy to the datacom equipment inside. The formula for calculating PUE is simply the energy consumed by the entire datacom facility (measured at the meter for the facility or room) divided by the energy consumed by the facility’s datacom equipment.

It is important to understand that the PUE metric was developed as a means for individual operations to monitor and track their own energy efficiencies. It was never intended as a means of comparing the efficiencies of different data centers, because too many conditions, including climate zone, can affect the number. It is also important to understand that an enterprise can take steps to reduce its total energy consumption, yet achieve a worse PUE. Extensive consolidation and virtualization, for example, and the purchase of ENERGY STAR^® rated servers, could significantly lower the datacom equipment energy number in the denominator of the PUE equation. However, unless a massive renovation of the power and cooling systems was also done, which would probably not be justifiable in most facilities, the energy consumption of those systems would not likely be reduced in the same proportion as the datacom loads. Although that would result in a larger PUE quotient, it should still be recognized as a very positive step, because total energy use has still been reduced.

It should also be recognized that the PUE metric is impractical to use as a means of quantifying projected energy efficiency in the design stage of a datacom facility. The number of calculations of electrical path efficiencies and losses, and the precision energy modeling that would be necessary to develop a realistic number would be overwhelming, and would still not result in a number likely to be realized when the facility is put into operation, potentially misleading owners into expecting something unachievable. It is for these reasons that different metrics were developed for use in the design stage, as set forth in the ASHRAE Standard 90.4.

A datacom facility is dynamic in terms of electrical and mechanical loading. The design of a datacom facility cooling system, whether single plant or modular, must be based on the maximum anticipated datacom equipment load of the space. However in reality, this maximum load is rarely, and sometimes never, achieved.

Even if the maximum design load is someday realized, the day-one load at move-in will be much lower than the ultimate design load in order to provide for long-term growth. Additionally, over the course of its lifetime (which may be 10 to 20 years or more), the datacom facility load constantly fluctuates. The load also changes density and location within the datacom space as systems are installed in one location and decommissioned in another.

These below-peak, fluctuating loads mean that the cooling plant operates in part-load conditions almost all of the time. It is therefore critical to ensure that the cooling plant selection has good part-load efficiency.

Typically, the primary energy users in a datacom facility cooling system are refrigerant compressors. Economizers, which leverage favorable ambient conditions to provide cooling without using compressors, are commonly integrated into cooling systems to minimize annual compressor use. There are three main classifications of economizers used on datacom facilities: dry, wet, and dual-mode.

A dry economizer can provide economizer cooling whenever the ambient dry-bulb conditions are suitable. This type of economizer can operate to provide cooling for a portion of the year in most climates.

A wet economizer consumes water to provide economizer cooling, leveraging the ambient wet-bulb conditions. This type of economizer can typically operate all year to provide some or all of the cooling required.

A hybrid economizer is able to operate either wet or dry, and can be designed to transition as the ambient conditions change. Hybrid economizers offer both the energy-saving benefits of a wet economizer and the water-saving and freeze protection advantages of a dry economizer. This type of economizer can be designed to optimize either water or power consumption in the datacom facility.

Water-Side Economizers. For systems that use a water or glycol loop to remove heat from a datacom space, a water-side economizer can be incorporated. These systems are typically designed as indirect fluid economizers to minimize coil fouling. In such cases, to ensure continuous flow and take advantage of the most hours of economization, the heat exchanger should be placed in series with the chiller. Condenser water can still be the primary source of cooling when ambient conditions allow. This arrangement provides a continuous flow of water through the system, and valve stroke time does not become a point of failure. Figure 14 shows a schematic diagram of a typical water-side economizer.

Air-Side Economizers. For systems where the room air is the primary transport medium of the heat load, an air-side economizer may be implemented. Air-side economizers for datacom facilities are separated into two general categories: direct and indirect. Schematic diagrams of these two categories are shown in Figures 15 and 16.

Figure 14. Schematic of Typical Water-Side Economizer (ASHRAE 2009b)

Figure 15. Schematic of Typical Direct Air-Side Economizer

Direct air-side economizers (DASEs) introduce ambient air directly into the space so that it flows through the datacom equipment to remove the heat. Indirect air-side economizers (IASEs) use ambient air to remove heat from recirculated cooling air by air-to-air heat exchangers.

Figure 16. Schematic of Typical Indirect Air-Side Economizer

Figure 17. Schematic of Typical Dual-Compressor Refrigerant-Side Economizer

Either solution may incorporate evaporative cooling to extend the number of economizer hours and, in some instances, reduce the capacity of the compressorized cooling equipment (trim cooling). See Chapter 41 of the 2020 ASHRAE Handbook—HVAC Systems and Equipment for more information on evaporative air-cooling systems.

Refrigerant-Side Economizers. More recently, systems with remote air-cooled condensers have integrated a refrigerant-side economizer by adding valves and a refrigerant pump to the refrigerant circuit. As ambient conditions allow, the compressor(s) are shut off and the pump activates to move the refrigerant between the indoor evaporator coil and the outdoor condenser coil. A typical system has multiple circuits, which allow for partial economizer cooling. See Figure 17 for a schematic of a typical system.