How to Pick a Cooling Solution for Your Servers? A Tech Guide by GIGABYTE

Learn More:
《Immersion cooling has been chosen by the industry forerunner as part of its ESG initiatives》

Air is the primary medium of transferring heat, which is why the rule of thumb in an air-based system is to improve the airflow. This is applicable to an entire data center or a single server.

In a data center, the airflow follows a specific path designed to disperse heat efficiently. First, cool air is pumped through floor vents into the data center’s “cold aisle”. Then, the cool air is channeled through the server racks, transferring heat to the “hot aisle”. Finally, the heated air is drawn into ceiling vents and chilled by the computer room air conditioning (CRAC) unit, thus renewing the cycle.

It is also important to facilitate airflow in the server itself. Cool air must flow past every key component to disperse the thermal energy. GIGABYTE offers various innovative hardware designs and programming to improve the airflow while preventing the ingress of airborne contaminants, making it a breeze to cool your servers.

Learn More:
《Glossary: What is a Data Center?》

 ● Hardware: An Airflow-Friendly Design

● Programming: Automatic Fan Speed Control

Besides the hardware design, the programming is equally sophisticated. GIGABYTE servers are shipped with automatic fan speed control as standard. Sensors are placed in the chassis to monitor the temperatures of the CPU, GPU, memory, and hard drive. If the baseboard management controller (BMC) detects a change at certain critical locations, the speed of the corresponding fan (or fans) will adjust automatically. On some models, fan speed profiles may be created or edited to accommodate specific requirements, allowing users to achieve an optimal balance between temperature control and power efficiency. 《Glossary: What is BMC?》

● Adaptive Customization to Protect Against Contaminants

Liquid cooling, also known as direct liquid cooling (DLC) or direct-to-chip (D2C) liquid cooling, uses sealed tubes (called cooling loops) filled with coolant to disperse heat. Thermal energy is transferred from the components to the coolant through cold plates; then, a heat exchanger removes the heat from the coolant, allowing it to circulate back into the server and repeat the cycle.

A liquid-based system is not only more effective at dissipating heat, it omits the need for air-based cooling components like heat sinks or fans. The extra space can be used to add more CPUs or GPUs. Because of this, high density servers and high performance computing (HPC) solutions often utilize some form of liquid cooling, whether it expels heat via a hybrid “liquid-to-air” system or a pure “liquid-to-liquid” heat exchanger.

Glossary:
《What is Liquid Cooling?》
《What is HPC?》

● Liquid-to-Air Cooling

This hybrid system is ideal for air-cooled data centers that want to dip a toe into liquid cooling without completely overhauling the infrastructure. Liquid-cooled servers are installed on standard server racks; the cooling loops are connected to heat exchangers fitted on the same rack. Heat is expelled into the data center’s “hot aisle”, similar to how an air-cooled server expels heat. This makes it possible to have a mix of air-cooled and liquid-cooled servers in the same facility. Though heat exchangers take up some rack space, liquid-cooled servers can house more processors, so the overall computing power is increased.

● Liquid-to-Liquid Cooling

A liquid-to-liquid system can remove more heat than liquid-to-air, but it comes with a caveat: it must be used in data centers with existing liquid cooling infrastructure. Rather than expelling heat into the “hot aisle”, the rack-mounted heat exchangers transfer energy to the facility’s liquid-based cooling loop, which is more efficient at dispersing heat than a standard CRAC. Because of this, server density can be increased exponentially, resulting in even greater computing power.

GIGABYTE has partnered up with two leading cooling solutions providers—Asetek and CoolIT Systems—to offer different choices for liquid-to-air and liquid-to-liquid cooling. The solutions come in many sizes: from the localized liquid-cooling of just one server rack to an entire liquid-cooled data center (around 750kW of heat dissipation).

Learn More:
《Discover CoolIT’s Liquid Cooling Solution》
《When Air isn’t Cutting it, Try Liquid-Cooled Servers》

Liquid-cooled servers offer increased computing power, greater reliability, and they can be installed where space is limited. Due to this, they are popular in a variety of sectors. For example, edge data centers can use liquid-cooled high density servers to squeeze more computing power into a smaller space. The financial sector, which uses computer programs to engage in high frequency trading (HFT), can also benefit from fast and stable liquid-cooled servers.《Glossary: What is Edge Computing?》

Another sector where liquid cooling is making waves is “green computing”. One of GIGABYTE’s clients is the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt, abbreviated as DLR), the national center for aerospace, energy, and transportation research in Germany. The team at DLR—including some literal rocket scientists—undertook to build a green data center to support space exploration, as well as other research programs. They specifically asked for servers that can operate in a data center with an ambient temperature of 40°C, and with no air conditioning equipment.

GIGABYTE recommended the liquid-cooled version of the H261-Z60 High Density Server. Paired with optimized power supplies, the solution could reduce power demand by 15% compared with competing products. Since more processors can be housed in a liquid-cooled server, the footprint was reduced by 50% while providing up to twice the maximum computing power of competing products. The liquid-cooled server solution fulfilled the DLR’s goal to deploy eco-friendly, high speed computing in a limited space.

Learn More:
《Success Story: DLR Builds Green Data Center with GIGABYTE’s Liquid Cooled Servers》
《More information about GIGABYTE's High Density Servers》

The H261-Z60 is not the only server eligible for liquid cooling. Other models like G481-S80 and R161-R12 can be modified. To use the G481-S80 as an example, outfitting it with a liquid-to-air heat exchanger can create energy savings of up to 70%. Additional models may be made available upon request.

Learn More:
《More information about GIGABYTE's GPU Servers》
《More information about GIGABYTE's Rack Servers》

Immersion cooling is the pinnacle of liquid cooling. By submerging the servers directly in a bath of non-conductive fluids, heat can be removed without cooling mechanisms like heat sinks or cold plates. Thermal energy is transferred out of the liquid in one of two ways: through a heat exchanger, as is the case with “single-phase” immersion cooling; or through a cycle of vaporization and condensation, as is the case with “two-phase” immersion cooling. 《Glossary: What is Immersion Cooling?》

GIGABYTE has joined forces with LiquidStack and 3M to offer a two-phase immersion cooling solution. The servers are placed inside a sealed tank partially filled with an “open bath” of 3M™ Fluorinert™ Electronic Liquid FC-72—a transparent, dielectric chemical that’s non-conductive and non-corrosive. Because the thermally conductive fluid has an ultra-low boiling point of just 56°C, heat from the components naturally causes the liquid to undergo a low-temperature evaporation process, transferring heat out of the liquid. The vapor rises to the top of the tank, where it is cooled by a heat exchanger (in this case, a condenser coil) to revert it back to liquid form. The liquid flows back into the bath to be reused.

Two-phase immersion cooling has a lot of advantages. For one, circulation inside the tank happens naturally without consuming any extra energy, so power usage efficiency (PUE) is increased. By some estimates, replacing air cooling with two-phase immersion cooling can lead to energy savings of up to 90%.《Glossary: What is PUE?》

What’s more, the removal of parts like pumps and jets means less maintenance and a lower chance of equipment failure. System stability is improved while operating costs go down. This is not only better for your bottom line, it is better for the environment.

The high number of processors that can be packed in a small space with immersion cooling makes it ideal for the 5G era. Telecom companies need to establish edge data centers to provide high speed processing capabilities and reduce data transfer latency. Such centers are often located in crowded urban areas with exorbitant property prices, so greater server density is essential. 
《Glossary: What is 5G?》

Immersion cooling can also be used to develop artificial intelligence (AI). Algorithms are trained through a process known as deep learning to implement groundbreaking applications, such as autonomous vehicles. The process requires a great deal of parallel computing, which in turn relies on dense GPU deployments. Powerful processors generate a lot of heat and may send energy bills through the roof. Replacing an air-based system with two-phase immersion cooling is more cost-efficient, not to mention more environmentally friendly.

Learn More:
《Glossary: What is Artificial Intelligence?》
《Glossary: What is Deep Learning?》
《Glossary: What is Parallel Computing?》
《Success Story: GIGABYTE Servers Pave the Way for Tomorrow’s Self-Driving Cars》

GIGABYTE has built an Immersion Cooling Proof of Concept (POC) Unit for testing and validation. It is fully compatible with 2U form factors and can be modified on request to suit 1U or 4U models. Many servers in GIGABYTE’s GPU Server product line are compatible with the POC unit: such as G291-281, a high density model that can house an unrivalled eight double-slot GPGPUs or co-processor cards within a 2U chassis; or G481-HA0, a 4U dual root server that supports 5G network infrastructure.

Learn More:
《Dive into GIGABYTE’s Two-Phase Liquid Immersion Cooling Solution》

After learning about three ways to control your servers’ temperature, we come to the crux of the problem: how do you pick the cooling solution that’s right for you?

The fact of the matter is, no two clients are alike. You are highly encouraged to contact our sales representatives with your specific requirements for expert consultation and tailor-made solutions.

However, there are three quick questions you can ask yourself to get a clearer picture of what you might need. Not only will these questions help you focus on the more feasible options, it will make it easier to formulate long-term plans about your data center in general, and the cooling system that acts as the backbone in particular.

As mentioned earlier, the efficacy of a cooling solution is measured in kilowatts (kW) of heat dissipation. Roughly speaking, to use a standard server rack as a basis of comparison, air cooling offers 20kW of heat dissipation; liquid cooling offers 60kW of heat dissipation; immersion cooling offers 200kW of heat dissipation.

In other words, your energy demand determines which cooling solution is most suitable for you. You should have a clear picture of which processors your servers will be using, and how much power they will consume. In general, processors used for HPC or parallel computing expend more energy and expel more heat. If that’s the job your servers will be doing most of the time, then it makes sense for you to invest in liquid cooling, or even immersion cooling.

Another factor to consider is how much space is available. Since air cooling relies on good airflow, the servers cannot be packed too tightly together. This limitation may affect the implementation of edge computing, which is why liquid and immersion cooling methods are gaining popularity.

It has been stated that PUE can be vastly improved by adopting liquid or immersion cooling. But water use efficiency (WUE) should also be evaluated. Good WUE and a reliable water supply can help ensure the smooth running of data centers that are cooled by such systems.

Liquid-to-liquid and immersion cooling solutions must be used in conjunction with the facility’s liquid supply and cooling loop. In other words, the data center must be built for such solutions. This is one of the biggest hurdles to adopting the more advanced forms of cooling.

Using a liquid-to-air system is a nice middle ground if you’d like to deploy liquid-cooled servers in an air-cooled data center. However, do keep in mind that liquid cooling and immersion cooling are catching on, and successful data centers should be scalable; they need to grow alongside the IT demand of their users, and they should adopt the latest technological improvements. A liquid-based infrastructure can support more advanced methods of cooling, and they can house servers that boast an impressive amount of computing power while taking up less space. A data center with these characteristics will be better prepared to keep up with the ever-growing sea of data. 《Glossary: What is Scalability?》

Ultimately, whether you opt to use air cooling, liquid cooling, or two-phase immersion cooling, GIGABYTE has the solutions you need. We encourage you to reach out to our sales representatives at server.grp@gigacomputing.com for consultation.