From a datacom power and cooling infrastructure planning perspective, the two common means of maximum load characterization are watts per square metre and kilowatts per rack. The datacom industry sometimes uses granularity as a means of describing the unit size.

Many in the datacom industry think that kilowatts per rack is superior to watts per square metre. However, at the start of a project, there might not be sufficient information about the quantity of racks, making the metric too granular. Professional judgment is critical to deciding which maximum load characterization to use.

Of equal importance is characterizing the minimum load as well as the load variation. The time increment for load variation can be very short (e.g., seconds, minutes) or very long. It is important to obtain or develop a detailed load profile including future possibilities.

Servers tend to be the most common equipment within a datacom space; there are many different types of servers and any type of server can go into any type of datacom facility. Although there are no set rules regarding what constitutes any specific server type, the following classifications help provide some guidance:

Standardized nomenclature defining the cooling airflow paths for datacom equipment have remained unaltered since 2004 (Figure 4). Most datacom equipment now uses the front-to-rear protocol. The exceptions are some legacy telecommunications equipment and some network switches. These may use a side-to-side protocol, or a mix of side-to-side-to-top and/or to-rear air flows, that are not shown.

The increasing heat densities of modern electronics are stretching the ability of air to adequately cool the electronic components within servers. Liquid cooling is therefore becoming more prevalent.

Figure 4. Equipment Airflow (ASHRAE 2012b)

Liquid cooling is defined as the process where a liquid (rather than air) is used to provide a heat removal (i.e., cooling) function. There are many different liquid-cooling solutions for datacom rooms. The most common implementations are

These definitions do not limit the cooling fluid to water. Various liquids could be considered for application, including some that could be in a vapor phase in part of the cooling loop.

Figure 5 depicts one example of liquid-cooled datacom equipment where a liquid loop internal to the rack is used to cool the components in the rack. In this case, the heat exchange is with a liquid-to-facility-water heat exchanger. Typically, liquid circulating in the rack is kept above dew point to eliminate any condensation concerns.

Figure 5. Internal Liquid-Cooling Loop Exchanging Heat with Liquid-Cooling Loop External to Racks (ASHRAE 2012b)

Most datacom facilities are well designed and are geographically located in areas with relatively clean environments. Therefore, they do not have significant contamination concerns. However, the overall cleanliness of the environment is only one consideration for the location of a datacom facility. It is often not considered to be a major driver.

Some datacom facilities may have harmful environments arising from the ingress of outdoor particulate and/or gaseous contamination. In some rare instances, contamination has been generated within the datacom facility itself.

If a datacom facility serving a critical application happens to be susceptible to gaseous or particulate contamination, the consequences could be severe. As a result, it is important to address the potential for contamination and mitigate the risk as much as practical.

A number of factors can result in an increased failure rate. Changes in solder type (lead based to lead free), and an ongoing miniaturizing of datacom equipment components increase the risk. Changes in datacom room temperature and humidity operating conditions combined with a lower priority consideration for the surrounding air quality are others.

Normal electrical power circuits typically fail with either an “open” or a “short.” Datacom equipment circuits are a much smaller physical size (can be smaller than a human hair) but fail in a similar manner. Two common datacom equipment circuit failures are termed copper creep corrosion and silver creep corrosion.

Detection of airborne dust contaminants can be determined from detailed visual inspections of filters in the air handling systems. Gaseous contaminant presence may require a seasonal or periodical monitoring and measurement through the use of copper and silver coupon testing in the datacom room. The coupons react when exposed to various gases (a typical exposure period is around one month). A subsequent lab analysis of the coupons can quantify the level of contaminants present.

Filtration systems (particulate filtration or gas filtration units) can be used to mitigate the risk of contaminants in the datacom facility. More information on this topic can be found in ASHRAE (2014).

The first edition of ASHRAE’s Thermal Guidelines for Datacom Processing Environments in 2004 created a common design point: the inlet temperature for datacom equipment. The 2008 edition expanded the recommended thermal envelope, and the 2011 edition increased the datacom class definitions from two to four, with wider thermal ranges (ASHRAE 2012a). All of these changes were made after a great deal of industry study. Important considerations include the following.

Recommended Environmental Range. To achieve both energy efficiency and equipment operating reliability and longevity, facilities must be designed to provide acceptable datacom equipment inlet conditions within the ASHRAE recommended temperature and humidity ranges. See Table 1 for this range, or use the process defined in ASHRAE (2012a).

Allowable Environmental Range. The allowable envelope is where datacom equipment manufacturers test their equipment to verify that it will function within those environmental boundaries. Typically, datacom equipment manufacturers perform tests before product announcement, to verify that it meets all functional requirements within this environmental envelope. This is not a statement of reliability but one of functionality of the datacom equipment. In addition to the allowable dry-bulb temperature and relative humidity ranges, the maximum dew point and maximum elevation values are part of the allowable operating environment definitions.

Practical Application. Prolonged exposure of operating equipment to conditions outside its recommended range, especially approaching the extremes of the allowable operating environment, can result in decreased equipment reliability and longevity [server reliability values versus inlet air temperatures are provided in ASHRAE (2012a) to provide some guidance on operating outside the recommended range]. Exposure of operating equipment to conditions outside its allowable operating environment risks catastrophic equipment failure.

Environmental Class Definitions for Air-Cooled Equipment. In order for any piece of datacom equipment to comply with a particular environmental class (ASHRAE 2012a), it must be able to reliably provide its full operational capabilities over the entire allowable environmental range, based on nonfailure conditions. The recommended and allowable ranges for each datacom equipment Class are given in tabular format in Table 1. The allowable environmental ranges for the four datacom equipment classes are illustrated in psychrometric format in Figure 6:

Dry-bulb temperature must be derated based on altitude for all classes. Refer to ASHRAE (2012a) for more information on derating methodology.

The latest guidelines were developed with a focus on providing as much information as possible to datacom facility operators so they can maximize energy efficiency without sacrificing the reliability required by their businesses. This assumes that the designs enable them to take advantage of reduced energy operation.

Figure 6. Environmental Classes for Datacom Equipment Classes (ASHRAE 2012a)

For any piece of datacom equipment to comply with a particular environmental class, it must be able to reliably provide its full operational capabilities over the entire classification temperature range based on non-failure conditions.

For datacom equipment that meets the higher supply temperatures as referenced by the ASHRAE classes in Table 2, enhanced thermal designs are required to keep liquid-cooled components within the desired temperature limits. Generally, the higher the supply water temperature, the lower the cost of the datacom facility cooling solution.

For classes W1 and W2, the datacom equipment should accommodate facility water supply temperatures that may be set by a campus-wide operational requirement. In these cases, condensation prevention is a must.

Availability of datacom equipment rated for classes W3 to W5 is limited. It is anticipated that future designs in these classes may involve trade-offs between IT cost and performance. However, these classes allow lower-cost data center infrastructure in some locations, as well as significantly reduced operating expense.

Facility water flow rate requirements and pressure drop values of the datacom equipment vary. Manufacturers typically provide configuration-specific flow rate and pressure differential requirements that are based on a given facility water supply temperature and rack heat dissipation to the water. Conformance with the water quality requirements for each cooling solution is important to long-term reliability.

A power supply nameplate rating indicates the maximum power draw for the datacom equipment’s safety and regulatory approval. A nameplate rating does not represent actual power draw during usage and should not be used as a measurement of datacom equipment heat release.

Manufacturers that follow ASHRAE guidelines follow a template for each product that tabulates heat release based on configuration and use. In addition, most major datacom equipment manufacturers have online tools that can provide even more specific and detailed heat release and airflow information to the datacom facility and datacom equipment community for use in datacom facility planning and design.

Datacom equipment manufacturers compete to create equipment that balances power and performance based on the markets and workloads they are targeting. Generally, the highest-performing servers continue to increase power density to address customers’ needs for uncompromised performance. ASHRAE (2012b) captures the upper limits of power density for various classifications of equipment (Figure 7).

When appropriately applied, datacom equipment power trends can be a powerful tool in considering what future loads might be in a facility or space. Future load is a critical component to planning, designing, constructing, and operating facilities to avoid ineffective expenditures, premature obsolescence, stranded cost or assets, etc.

Refer to ASHRAE (2012b) for details on how the trends were created along with how to apply them.

Figure 7. ASHRAE Projected Power Trends for Datacom Hardware (ASHRAE 2012b)

The goal of a good datacom facility cooling design is to match cooling capacity to actual heat load. This requires a correct and realistic assessment of the heat release of the projected datacom equipment. Even when actual datacom equipment is known, this can be challenging, and is often done incorrectly. A basic understanding of datacom equipment thermal design is therefore valuable, to comprehend how the datacom equipment interacts with the data center and vice versa.

The thermal design must ensure that the temperatures of all datacom equipment components (e.g., processors, memory, storage, I/O, power supplies) are maintained between the high and low limits of their specifications. Datacom equipment components have functional, reliability, and damage temperature specifications. Maximum functional temperature limits for silicon components are generally in the 85 to 105°C range.

The thermal management system (Figure 8) in the datacom equipment must take the appropriate actions to ensure compliance with these specifications. This ensures data integrity and maximizes equipment service life.

A well-designed thermal management implementation balances component temperatures, datacom equipment performance, acoustics and power consumption.

Air-cooled solutions are currently the most common approach for datacom equipment. The information described here is applicable to most mainstream, air-cooled volume servers; however, the principles apply to most types of datacom equipment.

Typical datacom equipment relies on variable-speed, forced-convection cooling to maintain the required temperatures. Component temperature is driven by one of three factors in an air-cooled system (Figure 9):

Datacom equipment manufacturers develop component- and equipment-level cooling solutions that balance cost, performance, and energy consumption, including trade-offs between air movers and heat sink designs. Cooling performance, power consumption, acoustic signature, fan reliability, and redundancy features are also important characteristics that must factor into the overall solution.

Figure 8. System Thermal Management

Figure 9. Example Component in System and Rack

Fan or cooling zones are often used to precisely adjust specific fans to the needs of the components most coupled with those fans. Cooling zones can be proximity based or physically separated. By using a fan zone approach, total fan power and acoustic output can be minimized. Fans in a nonstressed zone can run at lower speeds than those in a more highly stressed zone.

Thermal control enables optimization of datacom equipment system performance as a function of usage or workload, configuration, cooling capability, and environment. Underlying this optimization is the use of fan speed control and power management operating in parallel. Optimization for differential air temperature ΔT through the datacom equipment is generally not a significant design consideration because of the more critical requirement of ensuring that functional limits are maintained.

Components and their specifications are the primary drivers in a server’s thermal design (e.g., heat sink, fan selection, airflow management).

Power management features enable all components to stay within temperature limits while minimizing overall power consumption during periods of low activity.

Highly advanced control algorithms vary the datacom equipment fan speed and airflow, and tune the fan speed based on the datacom equipment’s usage model. Multiple algorithms can be used simultaneously, with the final fan speed determined by comparing the results of these algorithms.

Sensors (and proxy sensors) create the data necessary to trigger power management, and are the basis of a cohesive thermal management implementation.

With increasingly dense datacom equipment packaging, some components may require liquid cooling to maintain the environmental specifications required by the manufacturer.

Liquids considered for cooling electronic equipment are dielectric, engineered fluids, water, oils, or refrigerants. The transfer of heat from the liquid-cooled datacom equipment or components to the datacom facility generally takes place through a liquid-to-liquid heat exchanger.

Some liquid-cooling solutions include immersion of the datacom equipment components directly in a dielectric fluid, in either single- or two-phase applications. Dielectric fluids include mineral oil and fluoroketones.

A datacom facility can be a dedicated building, or be part of a general purpose building that houses other business functions or tenants. Regardless of the type of building in which it is housed, or its location within a building structure, a datacom facility is comprised of a number of spaces having different but interrelated functions (see Figure 1).

The main computing area is identified by different names: computer room; machine room; raised floor area; or white space. This chapter uses the term datacom room to differentiate it from the other areas that support it, and which comprise the complete datacom facility.

Determining the appropriate size of a datacom room is more challenging than it has ever been. There are three major reasons: the ever-increasing demand for computing services; the consolidation, virtualization, and increasing density of datacom equipment; and the moving of many computing services to the cloud or to leased co-location facilities.

It is unfortunately all too common to underestimate the amount of electrical/mechanical space required to support a datacom facility. Just the need for reliability in these critical facilities dictates a requirement for maintenance space. Overcrowding, even if minimum legal or manufacturer-dictated service clearances are maintained, can lead to inadvertent interruption of one system while servicing another. A rule of thumb, to be used as a starting point only, is to base minimum electrical/mechanical support space requirements on a percentage of the datacom room area:

Every increase in reliability requirements also increases the need for more redundant pieces of equipment, which in turn requires yet more support space. Further, highly redundant facilities require physical compartmentalization of duplicate or parallel systems by fire-rated walls, further increasing support space requirements.

The structure enclosing a datacom room should provide good thermal separation from the surrounding areas, whether they are exterior or interior spaces. The main concern with the overhead structure, regardless of its construction or intended use, is that it not be a source of particulate contamination or water leakage.

The overhead structure must be cleanly finished and sealed to avoid concrete dust. If it is a roof structure, take extra precautions to preclude leakage. In highly critical spaces, a double roof structure is used for insurance. Gaps and joints should be caulked.

Suspended ceiling tiles must be either metal pan or plastic encapsulated on both sides to prevent flake-off. This is particularly important when the above-ceiling plenum is used to convey return air. Cut edges must be sealed with spray paint or similar. Any suspension rods that penetrate the tiles should also be sealed at the penetrations. Metals used above a return air plenum ceiling should be either hot-dip galvanized or of a type that will not grow zinc whiskers.

Walls surrounding a datacom room should be well insulated to avoid both cooling loss and heat infiltration. All cracks should be sealed, which is mandatory if the room is also protected by a gas-based fire protection system.

Although most datacom equipment can accept a broad range of allowable humidity levels, consider installing vapor barriers for datacom spaces.

Windows should generally be avoided in a datacom room, but if they are necessary, they should be double-glazed and sealed. If covering the windows is allowed, and replacement is possible from the inside only, it will be necessary to make the coverings removable and to avoid blocking access with large pieces of mechanical/electrical equipment.

Raised access floors are no longer a standard requirement for datacom rooms. It is not only possible, but now relatively common, to put the entire power, cooling, and network infrastructure overhead, particularly when close-coupled cooling is used. Therefore, the raised floor is not necessary to convey air.

However, with the amount of piping often used to service in-row, rear-door, and direct water cooling, raised floors are often used anyway to avoid concerns about overhead water, as well as to minimize congestion above cabinets. When power, cable tray, and lighting are all run overhead, the vertical space can become congested and difficult to coordinate.

There are several advantages and disadvantages to using raised access floors, regardless of their purpose, and once it is determined that a raised access floor will be used, several factors should be considered in its selection and design.

The most obvious advantages of raised access floors are to provide a space for permanent infrastructure such as power, piping and cabling. Raised floors can also be used to convey cooling air through the plenum space. For slab variations too large to be leveled with patching and unrealistic for self-leveling cement, they can provide a level floor.

However, a raised access floor adds total mass to the structure. It must also be maintained, which includes releveling every few years, particularly if technicians do not take care in replacing tiles where they were removed, or open too many tiles in a row and destabilize the floor. The plenum space can also become a tangle of wire and cable if care is not used in installing new cable and removing old. If the floor is used to convey air, masses of unmaintained cable can reduce or totally block airflow. (Best practice is to not locate cables in floor plenum space.)

The height of a raised access floor is determined by its purpose. If it is used to convey cooling air it must be high enough to deliver the required air quantity while maintaining the necessary static pressure as evenly as possible across the floor area.

Piping, power systems or cable tray that will also occupy the space must be taken into consideration in determining the floor plenum height and its effect on air flow. It is generally accepted today that a raised access floor used to convey cooling air needs to be at least 450 to 600 mm high to be effective, and that even higher is better.

After height, the biggest consideration is floor structural strength. Raised floors for datacom rooms should use bolted stringer substructures to increase load capacity and to make it easy to remove and replace tiles without destabilizing the floor.

Newer computing equipment cabinets are usually rated for 1100 to 1360 kg, which is significantly higher than most legacy cabinets. Even if they are not full of the heaviest available equipment, floor loading must be planned as if they will be to address potential maximum loading in future.

However, cabinets with these load ratings also tend to be larger than legacy 600 by 600 mm cabinets, so the load is spread over more than one floor tile. It is not unrealistic to specify raised-floor systems designed for 700 kPa or 1360 kg rated tile capacity.

In selecting floor strengths, it is particularly important to examine the rolling load characteristics along with the static structural ratings, because equipment must often be moved into position on small integral wheels. The rolling load tests are generally performed for 10 passes and 10 000 passes with mass on test wheels of particular sizes in accordance with testing methods established by the Ceilings & Interior Systems Construction Association (CISCA) (CISCA 2007). However, not all tests are performed with the same wheel sizes, and the results can be misleading.

It is always safest to put thick hardboard or plywood over the floor when particularly heavy loads must be moved. This is especially important when rolling equipment through cool aisles with perforated airflow tiles, many of which do not have a rolling load specification and are easily deformed.

The most common surface material for raised access floor panels is high-pressure laminate (HPL). This material holds up well to heavy rolling loads without deforming or cracking, has good static dissipative characteristics when properly bonded to a grounded surface, is available in light colors to maximize lighting effectiveness, and is easy to maintain with damp mopping. Heavy scrubbing, buffing, or waxing should never occur in a datacom room.

This precludes the use of vinyl composite tile (VCT), pure vinyl tile (which can also be easily deformed under rolling loads), or linoleum (which is also too easily damaged). Carpeting, of course, should never be used in a datacom room, even if it is antistatic, because it both accumulates and generates particulate contaminants. (Note: it is generally accepted that the ground resistance of an installed raised-floor panel, when properly connected to a robust grounding system, should be in the range of 10⁴ to 10⁶ Ω τo minimize any potential for static generation.)

One of the most challenging decisions in selecting materials for air plenum raised access floors is the airflow panels. A range of types is now available, including legacy perforated tiles (25% open), grate-style cast aluminum tiles (56 to 63% open), and tiles incorporating directional vanes, air boost fans and automatic air flow control.

All air plenum raised floors leak air, and because cool air is expensive to produce and requires considerable fan energy to distribute through the plenum, this wastes energy. A good-quality raised-floor installation should leak no more than 2% air.

For datacom rooms designed without a raised access floor, the primary concern is that all power, cooling, and network infrastructure must be routed overhead. Depending mainly on the cooling method used, this can create a congested overhead space that requires careful design coordination and exacting installation. The cost of overhead infrastructure, particularly if extensive ductwork is necessary, can be very similar to the cost of a raised access floor.

Space must be allocated within a datacom facility for storing components and material, support equipment, and operating and servicing the datacom equipment. Some ancillary spaces may require environmental conditions comparable to those of the datacom equipment, whereas others may have less stringent requirements. Continuous operation of some support spaces is often vital to the facility’s proper functioning.

Electrical power distribution equipment can typically tolerate more variation and a wider range of temperature and humidity than datacom equipment. Equipment in this category includes incoming service/distribution switchgear, switchboard, automatic transfer switches, panel boards, and transformers. Manufacturers’ data should be checked to determine the amount of heat release and design conditions for satisfactory operation.

Uninterruptible power supplies (UPSs) come in various configurations, but most use batteries as the energy storage medium. They are usually configured to provide redundancy for the central power buses, and typically operate continuously at less than full-load capacity. They must be air conditioned with sufficient redundancy and diversity to provide an operable system throughout an emergency or accident.

UPS power monitoring and conditioning (rectifier and inverter) equipment is usually the primary source of heat release. This equipment usually has self-contained cooling fans that draw intake air from floor level or the equipment face, and discharge heated air at the top of the equipment. Air-distribution system design should take into account the position of the UPS air intakes and discharges.

Installation of secondary battery plants as a temporary back-up power source should be in accordance with IEEE Standard 1187 and NFPA Standard 70. Refer to other applicable standards, in addition to a design review with the local code official. Other relevant sources of guidance are NFPA Standards 70E and 76.

Temperature in a battery area is crucial to the life expectancy and operation of the batteries. The optimum space temperature for lead-calcium batteries is 25°C. If higher temperatures are maintained, it will reduce battery life; if lower temperatures are maintained, it may reduce the batteries’ ability to hold a charge (IEEE Standard 484).

Engine-driven generators used for primary or standby power typically have air-cooled radiators and require large volumes of outdoor air when running. Designs should ensure that engine exhaust air does not recirculate back to any building ventilation air intakes. Commonly, up to 72 h of fuel oil storage is required, so fuel oil storage tanks and distribution systems need to be integrated in the overall facility design and planning. The governing codes often mandate specific requirements for containment, location of fuel oil storage, fire resistance ratings, etc.

Fire Protection. Datacom fire protection involves a combination of strategies starting with prevention and continuing through detection, suppression, and response to a fire event. The National Fire Protection Association (NFPA) has several standards addressing design, installation, maintenance, and operation of fire protection systems in datacom facilities. Worldwide, additional fire protection standards may apply as well; consult local governments. Major NFPA standards include the following:

NFPA Standards 75 and 76 offer both prescriptive and performance-based approaches. Most designers defer to the prescriptive path, but a growing number of firms provide performance-based designs. These offer more flexibility, and can be tailored to a company’s specific risk and business models. As another alternative, some companies apply provisions from both standards, and often exceed one or more portions of either standard based on their own risk assessments or experiences.

There are several options for providing fire suppression in datacom rooms. Many older (and even some newer) datacom facilities use a code exemption to suppression. More commonly, however, datacom facilities are equipped with either a sprinkler or gaseous suppression system for a combination of life, structure, asset, and service protection. The conventional wisdom, invoked by many code authorities, is that gas protects equipment but sprinklers protect people and structures.

Air containment, either hot aisle or cool aisle, has become a common method of improving cooling performance and reducing energy usage. However, when containment systems are retrofit or designed into a new facility, effects on the required detection, suppression, release system, materials of construction, and prevention of fire must be considered. These important considerations are addressed in detail in the NFPA standards. The added obstructions often necessitate modifications to the suppression systems (sprinklers or gaseous agent nozzles) to ensure proper suppression release and dispersion.

Water Concerns. Water damage is always a concern in a datacom facility. It is best to locate above grade if possible, but this is not always practical.

There are other sources of water leakage as well: designs must consider the possibility of leaks from overhead. Datacom rooms and supporting electrical equipment should not be located below bathrooms, pantries, laboratories, or the like. If unavoidable, the space above should have waterproof membrane floors. Liquid piping should also be routed around the datacom room, but if this is not possible, should be provided with drip pans and leak detectors. Leak detectors should also be provided anywhere water can infiltrate, particularly if it could affect electrical infrastructure.

Acoustics. The rapid increase in density and power draw of datacom equipment has brought with it commensurate increases in required cooling. Air cooling requires substantial volumes of air movement, which generates sound levels that can be problematic for worker health and might require a hearing protection plan.

Sound level exposure limits in datacom rooms and their associated mechanical/electrical plant facilities are governed in the United States by the Occupational Safety and Health Administration (OSHA)’s General Industry Standard 1910.95. Similar regulations exist in other countries.

Sound emissions from heat rejection equipment (cooling towers and/or air-cooled chillers) as well as emergency and/or prime power-generating equipment for the datacom facility’s mechanical/electrical plant must also be considered. Noise generated outside the building, typically from rooftop chillers and cooling towers, must also be mitigated so that sound levels in the building are conducive to conducting normal business activities.

Community sound levels, mostly from exterior heat rejection and power-generating equipment, must typically comply with state, regional, or local noise codes, ordinances, guidelines, and/or regulations. Community sound level limits are typically cited at property lines and/or anywhere on the property of a potential complainant.

Sound levels of exterior equipment during normal, emergency, and test operation should allow for relatively easy communication among service personnel, as well as auditory awareness of vehicle and general service activities in the area. A sound level at or below 70 dBA in service areas and equipment yards, with all equipment operating, is an ideal goal.

Vibration. Vibration levels in datacom facilities must be considered as well. The greatest vibration concern in datacom installations is usually roof-mounted support equipment, such as air handlers, cooling towers, chillers, and generators, although similar equipment mounted inside the building can also create vibration issues. See the ASHRAE TC 9.9 datacom book series, especially ASHRAE (2008), for additional information.

Some datacom equipment can be sensitive to vibration, although disk drives have generally become far more tolerant of shock and vibration than in the past. Wherever there is concern, vibration specifications should be obtained from manufacturers, and the datacom facility floor’s vibration dynamics studied to determine compliance with vibration limits.

Many locations in the world are considered seismic zones, requiring special bracing and safety restraints for much of the infrastructure. However, the critical nature of many datacom facility operations mandates consideration of special structural supports and restraints even where seismic regulations are minimal or do not exist.

As described in ASHRAE (2008), it is important for both the owner and the designer to understand the potential hazards, including seismic, wind, etc., of the region where the facility is located.

Clear operational criteria should be established and used in system designs. These may include recommendations for structural restraints and bracing beyond what is required by law. Local code requirements must be identified and understood, as well as the requirements set forth in ASCE Standard 7, which provide further information and direction.

Lighting. In datacom rooms, lighting should usually be centered in the aisles, not over equipment cabinets or cable trays, where much of the light energy is wasted. Fixtures should also be suspended 2.68 to 2.75 m above the floor so as to deliver maximum illumination over the heads of technicians and into cabinets. Higher mountings may be necessary to clear other overhead infrastructure, but this disperses more light energy over the tops of cabinets and other obstructions. Although lighting is a small part of datacom room energy consumption, proximity sensors should be used to ensure they are not left on when there is no activity.

The photometric curves of most architectural luminaires are inappropriate for datacom facility lighting. Fixtures with wide horizontal dispersion patterns are needed. This requirement is very similar to the lighting of library book stacks, where the purpose is more to support the reading of titles on the books than to provide for reading books in the aisles. An illumination level of 325 lx on the vertical surfaces of cabinets is generally sufficient.

It is axiomatic that redundant systems should improve the reliability of a facility, but how much redundancy is justified is always a question, for business and operational needs as well as economic justification. Unfortunately, redundancy alone does not guarantee increased reliability.

It is not uncommon for large investments to be made in duplicate power and cooling systems that have been configured or installed in ways that defeat or greatly compromise their purpose. Consequently, a careful analysis of all possible failure modes should be an integral part of the design phase of any datacom facility.

The primary goal of redundancy should be to provide for concurrent maintainability. This requires a design that allows any item in the power and cooling infrastructure to be shut down and removed from service for maintenance without compromising the computing systems on which the infrastructure depends. This level of redundancy is commonly known as N+1, meaning that every system has at least one extra component and pathway.

Higher levels of redundancy require some degree of duplicate systems, such as two identical and fully load-sharing chiller plants with duplicate piping systems. This is known as 2N redundancy. An even more stringent design would have duplicate systems, but with additional redundant components in each. Depending on how the additional redundancy is configured, the systems may be known as 2N+1, in which an additional unit (e.g., a chiller module) is made available to either of the duplicate systems, or 2(N+1) in which both redundant systems each have their own redundant modules. Several methods have been developed to classify levels of redundancy and their resulting reliabilities and uptimes (e.g. ANSI/TIA Standard TIA-942A).

It is standard practice to power datacom equipment from an uninterruptable power supply (UPS). UPSs have two main purposes: to isolate the datacom equipment from power line disturbances; and to maintain ride-through power to the datacom equipment until back-up generators start, or long enough to accomplish an orderly shutdown.

With today’s heat densities, datacom systems cannot be maintained for very long on UPS alone. The usual maximum back-up time is 15 to 20 min, before thermal rise causes a shutdown of the datacom equipment and/or the UPS. High-performance computers may shut down in minutes or even seconds if cooling is interrupted. It is therefore necessary to have a means of maintaining cooling for critical systems until either generators start or they can be shut down properly.

It is generally impractical to run large cooling systems on UPSs. If this must be done, the cooling equipment’s electrical characteristics make it prudent to use a separate UPS. Further, the substantial power draws and high in-rush currents on compressor startup and cycling require large and expensive UPS systems.

In most datacom rooms it is not necessary to maintain full cooling for an extended period. If cooling can be continued to the most critical computing systems, this should suffice until both generators and full cooling restart. If a chilled-water system and close-coupled liquid-based cooling have been selected, this can be relatively easy to accomplish. There may be sufficient residual water in header pipes to cool critical systems for several minutes. If not, additional water can be stored in tanks.

Long battery life is of no value if the UPS it supports is without cooling. A UPS generates substantial heat under load, as do the batteries when they take full load after a power failure. Batteries also emit heat as they recharge once the generators start. This heat generation should be considered when choosing the location for the UPS, which is often relegated to a location that is less desirable for personnel. This is sometimes in an electrical or mechanical room that generates additional heat, in a corner of a parking garage, or even in a roof penthouse that is exposed to high sun loads. These kinds of locations can dramatically shorten the actual back-up duration of the UPS, particularly if the batteries are also exposed to continuous heat. The cooling system in the UPS room should have the same level of redundancy as the cooling for the datacom room.

Datacom equipment rooms can be conditioned with a wide variety of systems, including packaged computer room air-conditioning units and central-station air-handling systems. Air-handling and refrigeration equipment may be located either inside or outside the datacom equipment rooms.

The following system configurations are some of the most commonly used solutions to providing sufficient cooling to air-cooled datacom equipment.

Traditionally, telecommunication spaces had no raised floor and used overhead ducted air delivery, whereas datacom facilities used raised-flooring systems as supply air plenums.

Underfloor Air Delivery. The interstitial space under the raised floor creates a large-volume air plenum that, if properly configured, can deliver relatively uniform air pressures across the entire room area. However, because the floor plenum is also often used for piping, power and cable, there are many potential interruptions to airflow that can be challenging to mitigate.

Underfloor air delivery to cold aisles is relatively easy to provide and balance using a range of airflow tiles. Even distribution of air through the airflow panels is a function of the evenness of the static pressure below the floor. Leakage between the floor tiles and through any floor cutouts (cable or chilled-water lines) in the raised-floor plenum that are not correctly sealed will reduce the expected airflow.

Overhead Air Delivery. Delivering air overhead requires ducts large enough to convey the air volume needed to cool the equipment in each aisle, at velocities and pressures that enable air flow to be easily adjusted and balanced in each aisle, with even air availability over the full duct length. Often, ductwork is not considered adaptable enough to effectively accommodate the changes and upgrades that occur with the datacom equipment.

Effects of Air Mixing. Air mixing occurs in two ways: (1) when hot air discharged from computing equipment recirculates back to the air intakes, thereby increasing supply air temperature; and (2) when cool supply air bypasses the computing equipment and mixes with hot discharge air, thereby lowering return air temperature. Reduced return air temperature decreases the cooling capacity of the air conditioner coils.

If the supply air temperature has been set toward the upper limits of the ASHRAE recommended envelope, hot-air recirculation may result in equipment seeing inlet air that is warmer than the design temperature. Avoiding or minimizing air mixing requires separating the supply air from the return air and the datacom equipment intake air from the datacom equipment discharge air. The more complete the separation, the more effective and energy efficient the cooling system will be.

Hot Aisle/Cold Aisle. The first step in avoiding air mixing is to arrange cabinets in hot aisle/cold aisle configuration. This means that racks and cabinets are installed facing back-to-back and front-to-front. This arrangement keeps the hot air discharge from one row of cabinets from directly entering the intakes of cabinets in the next row.

If air paths exist through or between the datacom equipment racks, then some of the cool supply air will bypass the datacom equipment, and some of the discharge air will recirculate to the front equipment intakes. Use blanking or filler panels to minimize air mixing.

Containment. Containment further segregates the supply and return airflow paths by preventing mixing at the top of the equipment racks and at the end of equipment rows. There are several types of containment, including hot-aisle containment (HAC) and cold-aisle containment (CAC), either of which can be full or partial; and rack-based containment, commonly associated with active or passive chimneys. These main types of containment are illustrated in Figure 10.

One of the main challenges to maintaining the high availability required for datacom rooms is delivering cooling effectively and efficiently to all the equipment, wherever it is in the room. Complexities created by widely variable heat densities make it difficult to envision air movement in the space.

CFD simulations are a useful tool for predicting actual cooling performance. It requires building a 3D computer generated model of the datacom room. Of most practical importance is the way in which the user defines it: a model is only as good as the input data, regardless of the program’s sophistication. The model needs to represent the physical room geometry, and anything that might add or stimulate airflow or heat transfer, such as fans and vents. It must also include items that impede airflow, such as underfloor pipes and cables, and interactions with the surrounding environment (the boundary conditions).

Several simulations are commonly completed for datacom rooms. These may be based either on assumed datacom equipment layouts and projected heat densities, or on actual datacom equipment installations:

Although CFD is a powerful tool, it is also easy for it to be misapplied. Data centers are complex, and infrastructure and equipment must be simplified for models to be practical. It is critical, therefore, that the modeler understands the key elements of the data center and the fundamentals of CFD modeling for the application of CFD to be successful.

Figure 10. Examples of Main Types of Containment

In conceptual design of most enterprise facilities, the modeler will probably not know detailed information about the datacom equipment type or detailed configuration. Similarly, the precise location and routing of cables and other physical infrastructure may not be known, or even the cooling system manufacturer or model. In such an instance, there is little point in excessive detail in modeling, but at the same time the modeler must interpret the results accordingly: that is, understand that the predictions are limited to high-level system design decisions and recognize that performance will likely be a best-case solution because best practice has been assumed.

Where real facilities are being modeled, the models need to be more representative of the actual installation. This normally means basing the model on a physical survey of the facility, infrastructure, and datacom equipment configuration. Even so, the real infrastructure and equipment cannot be represented in ultimate detail. For example, a bundle of cables will be represented by an approximate obstruction or resistance to airflow rather than explicitly modeling each and every cable.

To ensure these judgments are made appropriately and the model is accurate, compare simulation results with measurements of airflow and temperature. Then, and only then, should the model be used for sensitivity studies to upgrade the facility, troubleshoot problems, or make deployment decisions.

Although CFD’s primary focus for datacom facilities is determining the effectiveness and efficiency of cooling delivery to the computing equipment, it can also be used to analyze such things as airflow around air-cooled chillers, generators, and other critical equipment. A very good use is evaluating redundant cooling system effectiveness by simulating failure modes.

Liquid-cooling equipment may be integrated with a facility-level cooling system in various ways, including the following.

Modular Room-Based Systems. The most common liquid cooling requires that facility chilled water be delivered to a heat exchanger [often called a cooling distribution unit (CDU)] located in or adjacent to the datacom room. The CDU has piping that connects to the datacom equipment; this is termed the technology cooling system (TCS). The TCS connections may be to a centralized heat exchanger at the datacom equipment rack or may connect with the datacom equipment itself (e.g., multiple connections per rack). An example of this configuration is shown in Figure 11.

The fluid in the technology cooling system may be chilled water, deionized water, refrigerant, or other liquids. The cooling distribution unit typically also contains pumps, valves, temperature monitoring and control, and operating software. Refrigerant-based systems have many of the same components as well as compressors and/or pumps and related control components. When datacom equipment cooling system (DECS) loops are required, they are considered beyond the scope of the facility-level installation.

Figure 11. Typical Liquid Cooling Systems/Loops Within a Datacom Facility

It is important to understand that many liquid-cooled datacom equipment solutions are not entirely cooled by liquid. Often, the datacom room needs to support a hybrid of air cooling and liquid cooling.

Direct Component Liquid Cooling. This type of system delivers the cooling medium directly to the individual datacom equipment, and often straight to the components. These systems are typically used in high-performance computing (HPC) or supercomputing platforms and have limited applications for typical commercial installations. They require completely dedicated piping distribution installations, as well as specialized heat exchangers, and related components between the liquid cooling equipment and the facility climate control systems.

Figure 12. Example of Chilled-Water Distribution Piping System (ASHRAE 2012b)

Immersion Cooling. In this type of system, the datacom chassis are fully immersed in a liquid bath. The cooling medium completely surrounds the devices, and circulates through the datacom enclosures or individual chassis subsystems. The pumped fluid transfers the heat to a dedicated coolant-to-water heat exchanger, which is connected to the facility chilled-water loop.

Facility water distribution systems that serve datacom equipment should be designed to the same standards or quality, reliability, and flexibility as other datacom room support systems. This means that it is important to configure the system so that it can be expanded or modified as needed to accommodate changes in datacom equipment without needing extensive system shutdowns. Further, the impact of servicing valves in the distribution system should be considered.

Figure 12 illustrates a looped chilled-water distribution system with sectional valves and multiple valved branch connections. The branches could serve air handlers or liquid-cooled datacom equipment. The valves allow modifications or repairs without a complete system shutdown.

Additional piping concepts are detailed in ASHRAE (2012b).

PUE is an efficiency metric developed and popularized by The Green Grid (TGG; http://www.thegreengrid.org/). Since the concept was introduced [see, e.g., Rawson et al. (2007)], the metric has been revised to make it more understandable and the methods and reporting of measurement numbers more reliable, culminating in the 2014 release of a joint TGG/ASHRAE TC 9.9 publication (ASHRAE 2013).

PUE measures how effectively a datacom facility delivers energy to the datacom equipment inside of it. The formula for calculating PUE is simply the energy consumed by the entire datacom facility (measured at the meter for the facility or room) divided by the energy consumed by the facility’s datacom equipment.

A datacom facility is a dynamic building in terms of electrical and mechanical loading. The design of a datacom facility cooling system, whether single plant or modular, is based on the maximum anticipated datacom equipment load of the space. In reality, this maximum load is rarely, and sometimes never, achieved, but the design must anticipate it anyway.

The day-one load at move-in is much lower than the ultimate design load, to provide for long-term growth. Over the course of its lifetime, which may be 10 to 20 years or more, the datacom facility load constantly fluctuates. The load also changes density and location as systems are installed in one place and decommissioned in another.

These below-peak, fluctuating loads mean that the cooling plant operates in part-load conditions almost all of the time. It is therefore critical to ensure that the cooling plant selection has good part-load efficiency.

To ensure continuous flow and take advantage of economization, the heat exchanger should be placed in series with the chiller. Condenser water can still be the primary source of cooling when ambient conditions allow. This arrangement provides a continuous flow of water through the system, and valve stroke time does not become a point of failure. Figure 13 shows a schematic diagram of a typical water-side economizer.

The increases in recommended and allowable inlet temperatures to datacom equipment have spurred an increased use of air-side economizer designs. Air-side economizers for datacom facilities are separated into two general categories: direct and indirect. Schematic diagrams of these two categories are shown in Figures 14 and 15.

Direct air-side economizers (DASEs) introduce ambient air directly into the space so that it flows through the interstices of datacom equipment to remove the heat. Indirect air-side economizers (IASEs) use ambient air to remove heat from recirculated cooling air by air-to-air heat exchangers.

Figure 14. Schematic of Typical Direct Air-Side Economizer

Figure 15. Schematic of Typical Indirect Air-Side Economizer

Either solution may incorporate indirect evaporative air cooling to reduce the number of operating hours and in some instances, the capacity of the compressorized cooling equipment. See Chapter 41 of the 2016 ASHRAE Handbook—HVAC Systems and Equipment for more information on indirect evaporative air-cooling systems.