Single-phase Immersion Cooling with GRC

Data centers can achieve high performance and efficiency even as the power per square meter increases.

First-rate Cooling in Data Centers

Data centers are heating up. High performance and AI workloads demand a level of compute density that we haven't seen before. The power requirements demanded by modern server processors, such as AMD EPYC and Intel Xeon Scalable, has more than doubled in a decade. Adding to this, accelerators have made great progress in data analytics and AI, but they too add to the power envelope in such a way that the data center just can't keep up. It's time to go back to the drawing board to create a new design.

GIGABYTE has partnered with GRC (Green Revolution Cooling) to offer a Single-Phase Liquid Immersion Cooling solution that delivers optimal system performance, hardware reliability, and high efficiency. By deploying this cooling solution customers can expect rapid ROI. As less energy is needed, TCO is reduced as well, helping data centers unlock greater value in the utilization of resources for HPC and AI workloads.

Air Cooling Falls Short

Data centers face increasing challenges from the amount of heat that is generated by components that have a high-power draw, such as CPUs and GPUs. These high temperatures can be detrimental to the overall performance of a system. Systems may have to throttle down to lower clock speeds to maintain availability or huge investments must be made to expand the air cooling system.

Additionally, besides power hungry components, modern high-performance systems have also increased server rack density, causing yet another heat issue. High density servers can house up to eight CPUs in a single 2U server, or 8 GPUs and 2 CPUs in a single 2U server. Now when these servers are stacked upon each other we have further compounded the issue of how to deal with the heat removal. This may also place restrictions on how much power is able to be supplied to a processor without reaching a heat barrier, preventing you from utilizing top tier performance, especially when you also require a large number of drives that can interfere with air flow.

An alternative choice to maximize the use of high performance servers is to use a complex direct-to-chip liquid cooling solution, which comes with a hefty price tag, but can certainly remove the heat more efficiently than air cooling. However, direct-to-chip cooling still does not achieve the efficiency that is possible by immersion cooling and often it still dumps the heat into the data center.

Nowadays we commonly see the average rack power density in a data center approaching 20kW/rack, but HPC applications are going far beyond this, requiring a solution to handle a 30kW/rack, or more, and one that does not rely solely on a facility's air conditioning. Enter immersion cooling. To transfer heat quickly, immersion cooling uses a dielectric coolant that is an excellent conductor of heat and has a greater heat-carrying capacity compared to air. The direct contact between the coolant and the processors (occasionally heatsinks too) allows for faster heat transfer from the hot components compared to using a heatsink and air flow to pull the heat away. Because of these immersion cooling solutions it is possible to have a self-contained system that can handle up to 100kW/rack, which is a density almost unheard of until now.

Immersion cooling with GRC and GIGABYTE

Liquid cooling has been in use in data center applications for over 30 years. Of the two most common immersion cooling methods, single-phase immersion cooling is the far simpler and quicker to get data centers up and running. In this instance, servers are placed vertically in a tank with an environmentally friendly synthetic fluid, aka coolant, that is an excellent conductor of heat but not electricity. From here the cooling process starts with a circulating loop that uses a coolant distribution unit (CDU) to pump in the cool or slightly warm coolant into the tank's bath. As the components heat up, the direct contact with the coolant transfers the heat from components to the coolant. This warm coolant is then piped out to the CDU where a pump and heat exchanger remove the heat using a cooling tower, dry cooler, or a chiller and then feed it back into the tank. All in all, the process maintains high availability and efficiency thanks to collaborations with immersion cooling leaders.

Year by year, GIGABYTE continues to add new industry leading technology partners that have a solid reputation, thorough know-how, and a successful track record. In GRC's case, the company is the acknowledged authority in immersion cooling and has been developing quick to deploy, cost saving, and efficient liquid immersion cooling systems for over a decade. In order for this joint venture to get off the ground GIGABYTE engineers worked with GRC to ensure that select GIGABYTE servers meet the specification requirements to be deployed in one of GRC's ICE or Hash lines of products.

ICE Line of Products
	ICEraQ	ICEtank
Racks	1 x 24U to 4 x 42U	4 x 42U to 8 x 42U
Deployment	Data Center	Container
Cooling Capacity	25-368kW	184-368kW
Applications	HPC, AI, Private/Cloud, Edge, Geosciences, and High Frequency Trading

Hash Line of Products
	HashTank	HashRaQ
# of Racks	1 or 2	2 x 52U or 6 x 52U
Deployed	Data Center	Container
Cooling Capacity	200kW	200kW
Applications	Bitcoin Mining or Blockchain Computing

Coolant - GRC ElectroSafe®

Physical & Chemical Properties	GRC ElectroSafe
Fluid Type	Synthetic (hydrocarbon based)
Chemical State	Inert liquid
Health Concerns	Safe to breathe
Evaporation Rate	Nil
Percent Volatile	Nil (NFPA 701 rating: 0-1-0)
Auto-Ignition Temperature	> 343°C (650°F)
Density	7.10-7.13 lbs/gal
Specific Gravity	0.80-0.88
Lifespan of coolant	15+ years

Liquids, when compared to air, have a much higher thermal capacity and excel at transferring heat. The direct contact between components and liquid facilitate fast heat transfer. GRC liquid immersion cooling systems utilize ElectroSafe coolants. ElectroSafe fluids represent a broad spectrum of high-performance, Earth-friendly, synthetic coolants that have undergone a meticulous selection and testing process. These coolants are dielectric, which is an absolute necessity, and can withstand the heat without degradation. They will outlast the components and are eco-friendly, non-toxic, odorless, non-evaporative, and electrically non-conductive. ElectroSafe fluids are globally available and have been deployed with GRC immersion cooling systems in 20 countries around the world, ensuring maximum performance, reliability and compatibility with virtually every GIGABYTE server and IT component available.

How GIGABYTE Servers Excel in GRC Tanks

Reduce Costs

Fanless server designs deliver 10-20% lower power draw, air-cooling infrastructure is reduced, and cuts cooling energy by up to 95%, dramatically reducing CAPEX and TCO.

Increase Thermal Efficiency (PUE)

Single-phase immersion cooling is more power-efficient than traditional air cooling. Expect 1.03 PUE for single-phase vs 1.7-2.0 PUE for air cooling.

Increase Hardware Reliability

Components will operate at lower temps and hot spots are remedied for high performance. Also, sealed tanks avoid contamination and eliminate vibrations.

Increase Compute per Square Meter

Workloads that demand power will run at optimal speeds while increasing both compute per rack and per square meter with top-tier performance.

Low Maintenance

Self-contained tanks need less servicing, which can be very beneficial for remote locations. Lower temps also reduce the chance of component failure.

Environmentally Friendly Coolant

GRC's ElectroSafe single-phase coolants are non-toxic and compliant with all regulatory or building codes, and long-lasting and fully recyclable.

Applications

HPC/AI

Training algorithms for AI applications or high-performance computing workloads require a powerful GPU. And with it comes a need to supply more power and to cool the components. Single-phase immersion cooling is cost efficient and environmentally friendly.

Edge

As compute is pushed closer to the source of the data, edge computing comes with space restrictions and a need for dense computing. Single-phase cooling excels at this without requiring significant air cooling infrastructure while GIGABYTE software aids with remote monitoring.

Cloud/Hyperscaler

The ability to quickly scale up is a critical function for cloud and hyperscalers. The GRC cooling tanks offer the complete solution and come in multiple sizes from a single 24U rack to a unit with four 42U racks.