Ackerman-Practicon
801 E. Charleston Road
Palo Alto, Ca. 94303
Phone (650) 965-1000
www.apcts.com
jpettegrew@apcts.com
 

Abstract

The next generation of High Density Data Center cooling systems must focus greater attention on the following two areas:  the control of the in-room cooling systems, and enhanced communication between the hardware vendor, the end users, and the Engineer of Record. Cooling delivered automatically in response to the thermal needs of computer hardware will increase the utilization efficiency and overall operating efficiency of the cooling systems.

Summary:

High Density Data Center (HDDC) cooling systems and Class 1 clean rooms have one thing in common. Their success depends on the effective management and control of large volumes of air within their respective spaces. In clean rooms the objective is to remove particulate matter and in the HDDC it is the removal of heat. Reversing the air pattern and creating an upside down clean room offers a unique way to deliver high volumes of cooling to the data center floor from below. Ackerman-Practicon used this innovative approach recently in the design of the mechanical systems for the new Terascale Facility at Lawrence Livermore Laboratory. The lower level of the building shell served a dual purpose. It was used to convey  cooling to the data center located above and was also used for the installation of the mechanical and  electrical support systems serving the data center.

The above is an example of a unique engineering approach that goes beyond the traditional design thinking for data center infrastructure. The increased demands being placed on HDDC cooling systems will require fresh design approaches and not merely the scaling up of past designs. The next generation of in-room HDDC cooling systems will be designed to accommodate the varying heat density applications with cooling provided automatically in response to the thermal needs of the computer hardware.

This discussion centers on the management and control of the in-room air distribution systems with emphasis on the following design and design process related areas:
 

• A greater emphasis needs to be placed on the design and implementation process for the in-room cooling systems. Air distribution by the manual deployment of supply air floor tile will no longer be adequate. The role of the Engineer of Record needs to be expanded to include the design of the in-room cooling systems with the design process mirroring that of any other cooling design application.  Computer hardware vendors, final users, and the Engineer of Record need to work as a team to develop the cooling criteria requirements for the various hardware applications within the data center.

• Increase the utilization efficiency and flexibility of the in-room cooling systems through the use of engineered air management and control systems. Cold cells, secondary cooling modules, and capacity controlled air grating, integrated into equipment layouts, will provide temperature controlled cooling zones to deliver cooling in response to the actual thermal needs of computer hardware.

The upward trend in the thermal densities of computer hardware, increased equipment compaction rates and a greater awareness of energy usage are all putting increased demands on the performance of data center cooling systems. Data center “hot spots,” which are becoming more prevalent, are the result of trying to meet the cooling requirements of today’s high heat density hardware with cooling system design philosophies that are more than 20 years old.

Data center cooling systems are designed, in general, to deliver uniform cooling over a specified area and do not address the variations in thermal densities that will ultimately occur within the data center. The more complex issue is the design of an in-room cooling system that is capable of reacting to equipment thermal footprints ranging from less than 100 watts/sf to over 2000 watts/sf. Temperature control is still mainly achieved by the manual manipulation of perforated floor tile in an effort to distribute the cooling. This will work to a degree when the initial computer hardware installation begins. However, as the equipment population of the room grows delivering cooling capacity to where it is needed will become impossible unless automatic control systems and cooling hardware capable of providing concentrated cooling are integrated into the design of the cooling systems. Design of the air management and control systems within the room becomes increasingly important as the range of heat densities throughout the room increases.

The basic data center cooling system, using air as the cooling medium, can be divided into the following three major components:
 

• The primary infrastructure generates and distributes the cooling for the data center. This includes refrigeration units and the piping infrastructure to distribute the cooling energy to primary air distribution units in or adjacent to the data center. There are several variations in equipment and system types depending on economics and the business objectives of the project. This design should be energy efficient and modular to accommodate future changes and expansion at a reasonable cost.

• The primary air delivery system conveys cooling from the air distribution units to the data center equipment. This air distribution system can be an overhead system or a supply plenum below a raised access floor. The use of the below floor supply air plenum is a cost effective and flexible method of conveying large amounts cooling energy that can be accessed at any point in the data center. It forms an energy reservoir that is available to meet the needs of any given area. Although there may be several cooling units supplying the below floor supply air plenum, the system is basically one large zone of temperature control.

• The secondary air delivery system is the controlled delivery of cooling from the supply plenum to the data center computer equipment. Static pressure in the supply air plenum forces air through perforated floor tiles or floor grates into the data center. Cooling capacities available to the room depend on the amount of pressure that can be developed in the below floor supply air plenum and the primary supply air temperature. Variations in the static pressure and temperature in the below floor supply plenum will result in unpredictable and largely uncontrollable cooling system performance. The distribution of cooling within the room has become the weak link in the overall cooling system performance. It directly affects the utilization efficiency and operating efficiency of the cooling systems. There are many existing data centers today that have adequate total capacity but are still experiencing cooling problems because the cooling can not be distributed to where it is needed.

In the past there has been a communication disconnect between the cooling system designer, the computer hardware vendor and the end user. It is not uncommon for the services of the Engineer of Record to end with the design of the basic cooling system infrastructure. When the equipment heat densities were lower this lack of communication between the various parties was less of an issue. Manual manipulation of supply air floor tiles provided a degree of temperature control. The cooling systems were more forgiving then and as a result the bad implementation habits noted above were established that continue today.

The design of the in-room air distribution and control systems needs to be approached like any other cooling design application. The Engineer of Record needs to work with hardware designers to define the cooling requirements necessary for optimum computer hardware performance.  Equipment locations, heat dissipation rates, air flows of computer hardware, and recommended operating temperatures are all parameters that need to be accounted for to determine the final distribution of cooling within the room.

The design and control of the secondary air delivery systems will depend on the peak and range of thermal densities in the room. Options available include the standard perforated floor tile, floor grating, and packaged secondary cooling modules. When fitted with control dampers the floor supply air tiles are limited in capacity. Without control dampers high capacity supply air floor tiles will make it difficult to maintain plenum pressure.

The predictable and reliable performance of the in-room air distribution and control systems starts with a stable and reliable static pressure in the below floor supply air plenum. With a stable static pressure under the floor, the secondary air delivery systems can be designed to meet the thermal needs of the computer hardware. Without this stability the performance of the cooling systems will be unpredictable and unreliable. Changes in one part of the system will influence the performance in other areas.

The following concepts for implementing secondary air delivery devices are offered for consideration. The design objective is to provide cooling automatically in response to the thermal needs of the computer hardware with minimal mixing of warm room air into the cooling air stream. The concepts are directed to those applications using open racks or free standing servers with density ranges given to describe the concept. With higher capacity controllable devices, equipment cold aisles can be reduced and equipment compaction rates increased. Final density ranges and the management and control of air distribution within the room would need to be engineered for specific applications.
 

• For low heat density applications up to approximately 3 KW per rack, standard perforated floor tile is an economical choice. Capacities of the various tile systems are readily available and the number of tiles and their placement can be easily determined. The higher pressure drops of these units will have minimal impact on the supply plenum static pressure.

• For applications between 3 and 8 KW per rack, higher capacity floor tiles or floor grates with air flow dampers should be considered. When installed with controls these units will deliver cooling only as required.

• For applications greater than 8 KW per rack, temperature controlled fan powered secondary cooling modules should be considered to provide separate zones of temperature control within the data center. Integrated into new designs these fan powered modules would deliver concentrated cooling to the computer hardware air intakes from the below floor supply air plenum. Temperature sensors would provide automatic cooling capacity in response to the thermal needs of the computer hardware. These multiple fan units drop into the standard 2 X 2 access floor and provide flexibility, scalability, reliability, and N+1 redundancy.

• For computer hardware mounted in enclosed racks and certain open rack applications, the development of a “cold cell” configuration within the data center should be considered. This configuration is similar to the bay and chase concept commonly used in clean rooms and takes the data center hot aisle/cold aisle concept to the next level. Cold cells are easily integrated into computer equipment layouts and provide a means for managing the air flow distribution within the data center. An architectural enclosure from the top of the equipment racks to a ceiling over the equipment aisles along with access doors at the ends of the aisles defines the cold cell envelop. This concept delivers supply plenum temperature air directly to the air intakes of all of the computer hardware served by a given cold cell. Mixing of warm room air into the supply air stream, as it moves through the data center, is no longer an issue. Computer hardware would be installed with the air intakes of the cooling fans having direct access to the below floor supply plenum temperature air in the cold cell. Hardware installed at the top of the rack would receive the same temperature of cold air as equipment installed at the bottom of the rack. The cold cell concept will allow increased equipment compaction rates, increased utilization efficiency of the installed cooling capacity, and increased operating efficiency of the cooling systems by distributing cold air directly to and through the computer hardware.

Cooling systems for new facilities will continue to utilize air as a cooling medium in the near term. Supply air temperatures need to be adjusted downward to achieve the greatest amount of cooling per pound of air delivered to the space. Variable flow fixed temperature supply air systems will increase the utilization efficiency and operational efficiency of the cooling systems. Developing new techniques for the management and control of the air distribution systems will be essential to providing a proper equipment operating environment. Utilizing a below floor supply air plenum provides an efficient means to deliver large amounts of cooling energy. It forms a big reservoir that can be tapped into as needed to address the varying heat densities that will occur within the data center. The height of the supply air plenum needs to be engineered for specific applications and will depend on the design load of the room along with other physical and operational parameters. It can range from a raised access floor within a single story building to a full story of building height where it shares space with mechanical and electrical equipment.

Successful cooling systems, for data centers of the future, will challenge current designs and design processes. Communication between hardware designers and cooling system designers will lead to cooling system designs that deliver cooling automatically in response to the thermal needs of the computer hardware. Control of the in-room cooling systems will increase the utilization efficiency and operating efficiency allowing owners to realize the full potential of their cooling system capital investment.

About the Author: James E. Pettegrew P.E. is a Mechanical Engineer with the consulting firm of Ackerman-Practicon located in Palo Alto, Ca.