Tech-Guide

How to Pick a Cooling Solution for Your Servers? A Tech Guide by GIGABYTE

Jul 18, 2023 by GIGABYTE

As CPUs and GPUs continue to advance, they consume more power and generate more heat. It is vital to keep temperature control in mind when purchasing servers. A good cooling solution keeps things running smoothly without hiking up the energy bill or requiring persistent maintenance. GIGABYTE Technology, an industry leader in high-performance servers, presents this Tech Guide to help you choose a suitable cooling solution. We analyze three popular options—air, liquid, immersion—and demonstrate what GIGABYTE can do for you.

1. Air Cooling

Air is the primary medium of transferring heat, which is why the rule of thumb in an air-based system is to improve the airflow. This is applicable to an entire data center or a single server.

Glossary:
《What is a Data Center?》
《What is a Server?》

In a data center, the airflow follows a specific path designed to disperse heat efficiently. First, cool air is pumped through floor vents into the data center’s “cold aisle”. Then, the cool air is channeled through the server racks, transferring heat to the “hot aisle”. Finally, the heated air is drawn into ceiling vents and chilled by the computer room air conditioning (CRAC) unit, thus renewing the cycle.

It is also important to facilitate airflow in the server itself. Cool air must flow past every key component to disperse the thermal energy. GIGABYTE offers various innovative hardware designs and programming to improve the airflow while preventing the ingress of airborne contaminants, making it a breeze to cool your servers.

A side view diagram demonstrating how heat is dissipated in an air-cooled data center. Servers must also be designed to facilitate airflow. They can be customized to provide greater protection against airborne contaminants that may cause equipment failure.

● Hardware: An Airflow-Friendly Design

All of GIGABYTE’s air-cooled servers come with a proprietary airflow-friendly hardware design. This means the overall airflow direction of the chassis has been evaluated with simulation software, and then fine-tuned to optimize ventilation. Powerful fans and high-performance heat sinks are installed to maximize contact between the cool air and the surface area of key components, enhancing heat dissipation. Special air ducts are attached to the heat sinks to further improve airflow. The components are made from heat-resistant materials to prevent them from conducting heat to the rest of the machine, so the server can beat the heat even when dealing with heavy workloads.

GIGABYTE’s air-cooled servers perform admirably even under the hefty workloads of pressure tests. This is because the chassis design has been optimized for ventilation; powerful fans and heat sinks have been installed at key positions; and the components are made from heat-resistant materials.

● Programming: Automatic Fan Speed Control

Besides the hardware design, the programming is equally sophisticated. GIGABYTE servers are shipped with automatic fan speed control as standard. Sensors are placed in the chassis to monitor the temperatures of the CPU, GPU, memory, and hard drive. If the baseboard management controller (BMC) detects a change at certain critical locations, the speed of the corresponding fan (or fans) will adjust automatically. On some models, fan speed profiles may be created or edited to accommodate specific requirements, allowing users to achieve an optimal balance between temperature control and power efficiency.

Glossary:

《What is a CPU?》

《What is a GPU?》

● Adaptive Customization to Protect Against Contaminants

By default, an air-cooled server is susceptible to damage from the elements, such as moisture and dust. Modern data centers may install air purifiers and adhere to a strict cleaning schedule, but the risk of airborne contaminants getting in is ever-present.

As an industry leader, GIGABYTE possesses the know-how to adapt our products for demanding environments. For example, GIGABYTE provided server solutions for the smart distribution center of a major logistics company. The constant stream of inbound envelopes and packages made it difficult to keep the servers clean. By studying the environment, it was discovered that microscopic fibers from papers and cardboard posed the biggest threat. GIGABYTE customized the mechanical design to achieve an ingress protection rating of IP5X, making the devices resistant to dust, while remaining conducive to airflow. This was done by adding a dust screen over the fans, dust covers over the exterior buttons, and other inventive customizations.

Learn More:
《Success Story: GIGABYTE’s customized solution for North American logistics leader》

2. Liquid Cooling

A liquid-cooled server uses sealed cooling loops filled with coolant to dissipate heat. Thermal energy is transferred from the components to the coolant through cold plates; then, a heat exchanger removes the heat from the coolant to repeat the cycle.

A liquid-based system is not only more effective at dissipating heat, but it also omits the need for air-based cooling components like heat sinks or fans. The extra space can be used to add more CPUs or GPUs. Because of this, high density servers and high-performance computing (HPC) solutions often utilize some form of liquid cooling, whether it expels heat via a hybrid “liquid-to-air” system or a pure “liquid-to-liquid” heat exchanger.

Glossary:
《What is Liquid Cooling?》
《What is HPC?》

● Liquid-to-Air Cooling

This hybrid system is ideal for air-cooled data centers that want to dip a toe into liquid cooling without completely overhauling the infrastructure. Liquid-cooled servers are installed on standard server racks; the cooling loops are connected to heat exchangers fitted on the same rack. Heat is expelled into the data center’s “hot aisle”, similar to how an air-cooled server expels heat. This makes it possible to have a mix of air-cooled and liquid-cooled servers in the same facility. Though heat exchangers take up some rack space, liquid-cooled servers can house more processors, so the overall computing power is increased.

With a liquid-to-air cooling solution, heat exchangers are fitted on the same rack as the liquid-cooled servers, and they transfer heat from the coolant into the “hot aisle” of the data center. This allows processors to be packed closely together, resulting in greater computing power. It also allows air-cooled and liquid-cooled servers to coexist in the same facility.

● Liquid-to-Liquid Cooling

A liquid-to-liquid system can remove more heat than liquid-to-air, but it comes with a caveat: it must be used in data centers with existing liquid cooling infrastructure. Rather than expelling heat into the “hot aisle”, the rack-mounted heat exchangers transfer energy to the facility’s liquid-based cooling loop, which is more efficient at dispersing heat than a standard CRAC. Because of this, server density can be increased exponentially, resulting in even greater computing power.

A liquid-to-liquid cooling system transfers heat from the coolant to the data center’s liquid supply, making it possible to pack servers even more densely together.

GIGABYTE has partnered up with the leading cooling solutions provider CoolIT Systems to provide high-density, factory-installed, drop-in ready liquid cooled servers for both the liquid-to-air and liquid-to-liquid configurations. The modular, rack-based cooling solution allows for dramatic increases in rack densities, component performance, and power efficiency. By using passive cold plate loops to eliminate separate pumping mechanisms, the rate of system failure is minimized. The rack manifold is easy to install and occupies only a single PDU space on each side of the rack. It features a reliable stainless steel design, and 100% dry-break quick disconnects for quick maintenance of any server node. Last but not least, the CDU provides an intelligent control system that manages supply temperature and flow of coolant to servers.

Learn More:
《Discover CoolIT’s Liquid Cooling Solution》
《When Air isn’t Cutting it, Try Liquid-Cooled Servers》

Liquid-cooled servers offer increased computing power and greater reliability, and they can be installed where space is limited. Due to this, they are popular in a variety of sectors. For example, edge data centers can use liquid-cooled high-density servers to squeeze more computing power into a smaller space. The financial sector, which uses computer programs to engage in high-frequency trading (HFT), can also benefit from fast and stable liquid-cooled servers.《Glossary: What is Edge Computing?》

Another sector where liquid cooling is making waves is “green computing”. One of GIGABYTE’s clients is the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt, abbreviated as DLR), the national center for aerospace, energy, and transportation research in Germany. The team at DLR—including some literal rocket scientists—undertook to build a green data center to support space exploration, as well as other research programs. They specifically asked for servers that can operate in a data center with an ambient temperature of 40°C, and with no air conditioning equipment.

GIGABYTE recommended the liquid-cooled version of the H261-Z60 High Density Server. Paired with optimized power supplies, the solution could reduce power demand by 15% compared with competing products. Since more processors can be housed in a liquid-cooled server, the footprint was reduced by 50% while providing up to twice the maximum computing power of competing products. The liquid-cooled server solution fulfilled the DLR’s goal to deploy eco-friendly, high-speed computing in a limited space.

Learn More:
《Success Story: DLR Builds Green Data Center with GIGABYTE’s Liquid Cooled Servers》
《More information about GIGABYTE's High Density Servers》
《More information about GIGABYTE's GPU Servers》
《More information about GIGABYTE's Rack Servers》

3. Immersion Cooling

Immersion cooling is the pinnacle of liquid cooling. By submerging the servers directly in a bath of non-conductive fluids, heat can be removed without cooling mechanisms like fans or even cold plates. Thermal energy is transferred out of the liquid in one of two ways: through a heat exchanger, as is the case with “single-phase” immersion cooling; or through a cycle of vaporization and condensation, as is the case with “two-phase” immersion cooling.

Learn More:
《Glossary: What is Immersion Cooling?》
《Dive into GIGABYTE’s Immersion Cooling Solutions》

In the single-phase variant, the heat given off by CPUs, GPUs, and other components is transferred directly to the liquid via a heatsink. The warm liquid is piped to a heat exchanger—in this case, a coolant distribution unit (CDU)—where a second transfer occurs as the heat is dissipated to the outside air or a water loop. The coolant is then pumped back into the immersion tank to continue the cycle. As the name implies, the fluid in the immersion tank does not change state, so there is no evaporation or condensation occurring.

In single-phase immersion cooling, warm coolant is pumped out to an external CDU, which chills the coolant through a variety of methods. The coolant is then pumped back into the tank to continue the cycle. As the name implies, the coolant does not change state throughout the process.

GIGABYTE provides a complete product line for single-phase immersion cooling, including GIGABYTE’s own immersion tanks for EIA or OCP-standard servers, as well as peripherals like lifts, dry racks, and coolants. GIGABYTE’s immersion-cooled servers are also compatible with solutions provided by Asperitas, GRC, and Submer. Japan’s leading telecom provider KDDI uses single-phase immersion cooling with GIGABYTE servers in their innovative green computing edge data centers.

Learn More:
《Discover GIGABYTE’s Single-phase Immersion Cooling Solution》
《Success Story: Japanese Telco Leader KDDI Invents Immersion Cooling Small Data Center with GIGABYTE》

GIGABYTE also offers two-phase immersion cooling solutions, which submerge the servers in a sealed tank partially filled with a bath of non-conductive and nontoxic fluids that have an ultra-low boiling point. Heat from the chips causes the bath to naturally undergo low-temperature vaporization, transferring heat out of the liquid. The vapor rises to the top of the tank, where it is cooled by a heat exchanger, such as a condenser coil, so that it reverts to liquid form. The fluids flow back into the bath to be reused.

Two-phase immersion cooling places servers directly in a bath of non-conductive liquid with a very low boiling point. Heat from the components causes the fluids to vaporize and disperse heat. The vapor is then cooled by a coil condenser and reverts to liquid form, which flows back into the bath to be reused.

Two-phase immersion cooling has a lot of advantages. For one, circulation inside the tank happens naturally without consuming any extra energy, so power usage effectiveness (PUE) is improved. By some estimates, replacing air cooling with two-phase immersion cooling can lead to energy savings of up to 90%. What’s more, the removal of parts like pumps and jets means less maintenance and a lower chance of equipment failure. System stability is enhanced while operating costs go down. Success cases include the world’s foremost semiconductor giant, which selected GIGABYTE’s two-phase immersion cooling solution to build “green HPC data centers” capable of pushing PUE below 1.08 and reducing the data center’s total power consumption by up to 30%.

Learn More:
《Glossary: What is PUE?》
《Success Story: Semiconductor Giant Selects GIGABYTE’s Two-Phase Immersion Cooling Solution》

Whether you go with single-phase or two-phase immersion cooling, GIGABYTE servers are modular designed, which means that air-cooled servers may be modified for use in immersion cooling. Servers used in these solutions are installed with brackets to support vertical installation and removal for easy management. GIGABYTE's immersion tanks are also self-contained, which means they require less servicing. They are equipped with sensors to maintain ideal operating temperature, as well as a redundant CDU to provide greater operation stability.

Summary: Three Steps to Help You Pick a Suitable Cooling Solution

After learning about three ways to control your servers’ temperature, we come to the crux of the problem: how do you pick the cooling solution that’s right for you?

The fact of the matter is, no two clients are alike. You are highly encouraged to contact our sales representatives with your specific requirements for expert consultation and tailor-made solutions.

However, there are three quick questions you can ask yourself to get a clearer picture of what you might need. Not only will these questions help you focus on the more feasible options, but it will also make it easier to formulate long-term plans about your data center in general, and the cooling system that acts as the backbone in particular.

1. Energy consumption and heat dissipation

The efficacy of a cooling solution is measured in kilowatts (kW) of heat dissipation. Roughly speaking, to use a standard server rack as a basis of comparison, air cooling offers 20kW of heat dissipation; liquid cooling offers 60kW of heat dissipation; immersion cooling offers 200kW of heat dissipation.

In other words, your energy demand determines which cooling solution is most suitable for you. You should have a clear picture of which processors your servers will be using, and how much power they will consume. In general, processors used for HPC or parallel computing expend more energy and expel more heat. If that’s the job your servers will be doing most of the time, then it makes sense for you to invest in liquid cooling, or even immersion cooling.《Glossary: What is Parallel Computing?》

2. Availability of space and other resources

Another factor to consider is how much space is available. Since air cooling relies on good airflow, the servers cannot be packed too tightly together. This limitation may affect the implementation of edge computing, which is why liquid and immersion cooling methods are gaining popularity.

It has been stated that PUE can be vastly improved by adopting liquid or immersion cooling. But water use efficiency (WUE) should also be evaluated. Good WUE and a reliable water supply can help ensure the smooth running of data centers that are cooled by such systems.

3. Infrastructure

Liquid-to-liquid and immersion cooling solutions must be used in conjunction with the facility’s liquid supply and cooling loop. In other words, the data center must be built for such solutions. This is one of the biggest hurdles to adopting the more advanced forms of cooling.

Using a liquid-to-air system is a nice middle ground if you’d like to deploy liquid-cooled servers in an air-cooled data center. However, do keep in mind that liquid cooling and immersion cooling are catching on, and successful data centers should be scalable; they need to grow alongside the IT demand of their users, and they should adopt the latest technological improvements. A liquid-based infrastructure can support more advanced methods of cooling, and they can house servers that boast an impressive amount of computing power while taking up less space. A data center with these characteristics will be better prepared to keep up with the ever-growing sea of data. 《Glossary: What is Scalability?》

Ultimately, whether you opt to use air cooling, liquid cooling, or immersion cooling, GIGABYTE has the solutions you need. We encourage you to reach out to our sales representatives at marketing@gigacomputing.com for consultation.

More GIGABYTE Tech Guides:
《To Harness Generative AI, You Must Learn About “Training” & “Inferencing”》
《CPU vs. GPU: Which Processor is Right for You?》