How to Pick a Cooling Solution for Your Servers? A Tech Guide by GIGABYTE
As CPUs and GPUs continue to advance, they consume more power and generate more heat. It is vital to keep temperature control in mind when purchasing servers. A good cooling solution keeps things running smoothly without hiking up the energy bill or requiring persistent maintenance. GIGABYTE Technology, an industry leader in high-performance servers, presents this tech guide to help you choose a suitable cooling solution. We analyze three popular options—air, liquid, immersion—and demonstrate what GIGABYTE can do for you.
Air is the primary medium of transferring heat, which is why the rule of thumb in an air-based system is to improve the airflow. This is applicable to an entire data center or a single server.
In a data center, the airflow follows a specific path designed to disperse heat efficiently. First, cool air is pumped through floor vents into the data center’s “cold aisle”. Then, the cool air is channeled through the server racks, transferring heat to the “hot aisle”. Finally, the heated air is drawn into ceiling vents and chilled by the computer room air conditioning (CRAC) unit, thus renewing the cycle.
It is also important to facilitate airflow in the server itself. Cool air must flow past every key component to disperse the thermal energy. GIGABYTE offers various innovative hardware designs and programming to improve the airflow while preventing the ingress of airborne contaminants, making it a breeze to cool your servers.
A side view diagram demonstrating how heat is dissipated in an air-cooled data center. Servers must also be designed to facilitate airflow. They can be customized to provide greater protection against airborne contaminants that may cause equipment failure.
● Hardware: An Airflow-Friendly Design
All of GIGABYTE’s air-cooled servers come with a proprietary airflow-friendly hardware design. This means the overall airflow direction of the chassis has been evaluated with simulation software, and then fine-tuned to optimize ventilation. Powerful fans and high performance heat sinks are installed to maximize contact between the cool air and the surface area of key components, enhancing heat dissipation. Special air ducts are attached to the heat sinks to further improve airflow. The components are made from heat-resistant materials to prevent them from conducting heat to the rest of the machine, so the server can beat the heat even when dealing with heavy workloads.
GIGABYTE’s air-cooled servers perform admirably even under the hefty workloads of pressure tests. This is because the chassis design has been optimized for ventilation; powerful fans and heat sinks have been installed at key positions; and the components are made from heat-resistant materials.
● Programming: Automatic Fan Speed Control
Besides the hardware design, the programming is equally sophisticated. GIGABYTE servers are shipped with automatic fan speed control as standard. Sensors are placed in the chassis to monitor the temperatures of the CPU, GPU, memory, and hard drive. If the baseboard management controller (BMC) detects a change at certain critical locations, the speed of the corresponding fan (or fans) will adjust automatically. On some models, fan speed profiles may be created or edited to accommodate specific requirements, allowing users to achieve an optimal balance between temperature control and power efficiency. 《Glossary: What is BMC?》
● Adaptive Customization to Protect Against Contaminants
By default, an air-cooled server is susceptible to damage from the elements, such as moisture and dust. Modern data centers may install air purifiers and adhere to a strict cleaning schedule, but the risk of airborne contaminants getting in is ever-present.
As an industry leader, GIGABYTE possesses the know-how to adapt our products for demanding environments. For example, GIGABYTE provided server solutions for the smart distribution center of a major logistics company. The constant stream of inbound envelopes and packages made it difficult to keep the servers clean. By studying the environment, it was discovered that microscopic fibers from papers and cardboard posed the biggest threat. GIGABYTE customized the mechanical design to achieve an ingress protection rating of IP5X, making the devices resistant to dust, while remaining conducive to airflow. This was done by adding a dust screen over the fans, dust covers over the exterior buttons, and other inventive customizations.
Liquid cooling, also known as direct liquid cooling (DLC) or direct-to-chip (D2C) liquid cooling, uses sealed tubes (called cooling loops) filled with coolant to disperse heat. Thermal energy is transferred from the components to the coolant through cold plates; then, a heat exchanger removes the heat from the coolant, allowing it to circulate back into the server and repeat the cycle.
A liquid-cooled server uses sealed cooling loops filled with coolant to dissipate heat. Thermal energy is transferred from the components to the coolant through cold plates; then, a heat exchanger removes the heat from the coolant to repeat the cycle.
A liquid-based system is not only more effective at dissipating heat, it omits the need for air-based cooling components like heat sinks or fans. The extra space can be used to add more CPUs or GPUs. Because of this, high density servers and high performance computing (HPC) solutions often utilize some form of liquid cooling, whether it expels heat via a hybrid “liquid-to-air” system or a pure “liquid-to-liquid” heat exchanger.
This hybrid system is ideal for air-cooled data centers that want to dip a toe into liquid cooling without completely overhauling the infrastructure. Liquid-cooled servers are installed on standard server racks; the cooling loops are connected to heat exchangers fitted on the same rack. Heat is expelled into the data center’s “hot aisle”, similar to how an air-cooled server expels heat. This makes it possible to have a mix of air-cooled and liquid-cooled servers in the same facility. Though heat exchangers take up some rack space, liquid-cooled servers can house more processors, so the overall computing power is increased.
With a liquid-to-air cooling solution, heat exchangers are fitted on the same rack as the liquid-cooled servers, and they transfer heat from the coolant into the “hot aisle” of the data center. This allows processors to be packed closely together, resulting in greater computing power. It also allows air-cooled and liquid-cooled servers to coexist in the same facility.
● Liquid-to-Liquid Cooling
A liquid-to-liquid system can remove more heat than liquid-to-air, but it comes with a caveat: it must be used in data centers with existing liquid cooling infrastructure. Rather than expelling heat into the “hot aisle”, the rack-mounted heat exchangers transfer energy to the facility’s liquid-based cooling loop, which is more efficient at dispersing heat than a standard CRAC. Because of this, server density can be increased exponentially, resulting in even greater computing power.
A liquid-to-liquid cooling system transfers heat from the coolant to the data center’s liquid supply, making it possible to pack servers even more densely together.
GIGABYTE has partnered up with two leading cooling solutions providers—Asetek and CoolIT Systems—to offer different choices for liquid-to-air and liquid-to-liquid cooling. The solutions come in many sizes: from the localized liquid-cooling of just one server rack to an entire liquid-cooled data center (around 750kW of heat dissipation).
Liquid-cooled servers offer increased computing power, greater reliability, and they can be installed where space is limited. Due to this, they are popular in a variety of sectors. For example, edge data centers can use liquid-cooled high density servers to squeeze more computing power into a smaller space. The financial sector, which uses computer programs to engage in high frequency trading (HFT), can also benefit from fast and stable liquid-cooled servers.《Glossary: What is Edge Computing?》
Another sector where liquid cooling is making waves is “green computing”. One of GIGABYTE’s clients is the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt, abbreviated as DLR), the national center for aerospace, energy, and transportation research in Germany. The team at DLR—including some literal rocket scientists—undertook to build a green data center to support space exploration, as well as other research programs. They specifically asked for servers that can operate in a data center with an ambient temperature of 40°C, and with no air conditioning equipment.
GIGABYTE recommended the liquid-cooled version of the H261-Z60 High Density Server. Paired with optimized power supplies, the solution could reduce power demand by 15% compared with competing products. Since more processors can be housed in a liquid-cooled server, the footprint was reduced by 50% while providing up to twice the maximum computing power of competing products. The liquid-cooled server solution fulfilled the DLR’s goal to deploy eco-friendly, high speed computing in a limited space.
The H261-Z60 is not the only server eligible for liquid cooling. Other models like G481-S80 and R161-R12 can be modified. To use the G481-S80 as an example, outfitting it with a liquid-to-air heat exchanger can create energy savings of up to 70%. Additional models may be made available upon request.
Immersion cooling is the pinnacle of liquid cooling. By submerging the servers directly in a bath of non-conductive fluids, heat can be removed without cooling mechanisms like heat sinks or cold plates. Thermal energy is transferred out of the liquid in one of two ways: through a heat exchanger, as is the case with “single-phase” immersion cooling; or through a cycle of vaporization and condensation, as is the case with “two-phase” immersion cooling. 《Glossary: What is Immersion Cooling?》
GIGABYTE has joined forces with LiquidStack and 3M to offer a two-phase immersion cooling solution. The servers are placed inside a sealed tank partially filled with an “open bath” of 3M™ Fluorinert™ Electronic Liquid FC-72—a transparent, dielectric chemical that’s non-conductive and non-corrosive. Because the thermally conductive fluid has an ultra-low boiling point of just 56°C, heat from the components naturally causes the liquid to undergo a low-temperature evaporation process, transferring heat out of the liquid. The vapor rises to the top of the tank, where it is cooled by a heat exchanger (in this case, a condenser coil) to revert it back to liquid form. The liquid flows back into the bath to be reused.
Immersion cooling places servers directly in a bath of non-conductive liquid with a very low boiling point. Heat from the components causes the fluids to vaporize and disperse heat. The vapor is then cooled by a coil condenser and reverts to liquid form, which flows back into the bath to be reused. (Image provided by LiquidStack)
Two-phase immersion cooling has a lot of advantages. For one, circulation inside the tank happens naturally without consuming any extra energy, so power usage efficiency (PUE) is increased. By some estimates, replacing air cooling with two-phase immersion cooling can lead to energy savings of up to 90%.《Glossary: What is PUE?》
What’s more, the removal of parts like pumps and jets means less maintenance and a lower chance of equipment failure. System stability is improved while operating costs go down. This is not only better for your bottom line, it is better for the environment.
The high number of processors that can be packed in a small space with immersion cooling makes it ideal for the 5G era. Telecom companies need to establish edge data centers to provide high speed processing capabilities and reduce data transfer latency. Such centers are often located in crowded urban areas with exorbitant property prices, so greater server density is essential. 《Glossary: What is 5G?》
Immersion cooling can also be used to develop artificial intelligence (AI). Algorithms are trained through a process known as deep learning to implement groundbreaking applications, such as autonomous vehicles. The process requires a great deal of parallel computing, which in turn relies on dense GPU deployments. Powerful processors generate a lot of heat and may send energy bills through the roof. Replacing an air-based system with two-phase immersion cooling is more cost-efficient, not to mention more environmentally friendly.
GIGABYTE has built an Immersion Cooling Proof of Concept (POC) Unit for testing and validation. It is fully compatible with 2U form factors and can be modified on request to suit 1U or 4U models. Many servers in GIGABYTE’s GPU Server product line are compatible with the POC unit: such as G291-281, a high density model that can house an unrivalled eight double-slot GPGPUs or co-processor cards within a 2U chassis; or G481-HA0, a 4U dual root server that supports 5G network infrastructure.
GIGABYTE’s Immersion Cooling POC Unit. It is compatible with 2U form factors and can be modified on request to be used with 1U or 4U servers.
Summary: Three Steps to Help You Pick a Suitable Cooling Solution
After learning about three ways to control your servers’ temperature, we come to the crux of the problem: how do you pick the cooling solution that’s right for you?
The fact of the matter is, no two clients are alike. You are highly encouraged to contact our sales representatives with your specific requirements for expert consultation and tailor-made solutions.
However, there are three quick questions you can ask yourself to get a clearer picture of what you might need. Not only will these questions help you focus on the more feasible options, it will make it easier to formulate long-term plans about your data center in general, and the cooling system that acts as the backbone in particular.
1. Energy consumption and heat dissipation
As mentioned earlier, the efficacy of a cooling solution is measured in kilowatts (kW) of heat dissipation. Roughly speaking, to use a standard server rack as a basis of comparison, air cooling offers 20kW of heat dissipation; liquid cooling offers 60kW of heat dissipation; immersion cooling offers 200kW of heat dissipation.
In other words, your energy demand determines which cooling solution is most suitable for you. You should have a clear picture of which processors your servers will be using, and how much power they will consume. In general, processors used for HPC or parallel computing expend more energy and expel more heat. If that’s the job your servers will be doing most of the time, then it makes sense for you to invest in liquid cooling, or even immersion cooling.
2. Availability of space and other resources
Another factor to consider is how much space is available. Since air cooling relies on good airflow, the servers cannot be packed too tightly together. This limitation may affect the implementation of edge computing, which is why liquid and immersion cooling methods are gaining popularity.
It has been stated that PUE can be vastly improved by adopting liquid or immersion cooling. But water use efficiency (WUE) should also be evaluated. Good WUE and a reliable water supply can help ensure the smooth running of data centers that are cooled by such systems.
Liquid-to-liquid and immersion cooling solutions must be used in conjunction with the facility’s liquid supply and cooling loop. In other words, the data center must be built for such solutions. This is one of the biggest hurdles to adopting the more advanced forms of cooling.
Using a liquid-to-air system is a nice middle ground if you’d like to deploy liquid-cooled servers in an air-cooled data center. However, do keep in mind that liquid cooling and immersion cooling are catching on, and successful data centers should be scalable; they need to grow alongside the IT demand of their users, and they should adopt the latest technological improvements. A liquid-based infrastructure can support more advanced methods of cooling, and they can house servers that boast an impressive amount of computing power while taking up less space. A data center with these characteristics will be better prepared to keep up with the ever-growing sea of data. 《Glossary: What is Scalability?》
Ultimately, whether you opt to use air cooling, liquid cooling, or two-phase immersion cooling, GIGABYTE has the solutions you need. We encourage you to reach out to our sales representatives at email@example.com for consultation.