Japanese Telco Leader KDDI Invents Immersion Cooling Small Data Center with GIGABYTE

KDDI Corporation, which ranks alongside SoftBank Group Corp. and NTT Docomo Inc. as one of Japan’s top three telecommunications companies, started work in July of 2020 on a new, experimental design for data centers that may set the trend for edge computing and “net zero” green computing. These modular, containerized micro data centers, which KDDI refers to as “container-type immersion cooling small data centers” (コンテナ型液浸スモールデータセンター), are expected to drastically reduce both the carbon footprint and the installation lead time of data centers.

Learn More:
《Visit GIGABYTE’s CSR Homepage to See How We are Giving Back to the World!》
《Glossary: What is Data Center?》
《Glossary: What is Edge Computing?》

There are many benefits to such a design. For one thing, because these data centers are portable and convenient to install, they can quickly respond to a burst in demand in a specific location (such as an outdoor music festival—think Coachella) or an emergency situation (such as an earthquake). What’s more, since these data centers are environmentally friendly and sustainable, they can help us build the smart, interconnected world of tomorrow—without having an adverse effect on the environment.

This second point is worth going deeper into. As our lives become more digitized, the international community is building more and more data centers. They are an indelible part of our modern IT infrastructure, because they contribute to every high tech invention you can think of, from extended reality (XR) and the metaverse to self-driving cars and space exploration. However, data centers eat up a lot of energy. By some estimates, data centers consume 2% of the world’s electricity and emit roughly as much carbon dioxide (CO2) as the entire aviation industry. How to build new data centers without harming the environment or dumping more carbon into the atmosphere—that is the million-dollar question (or billion-dollar, if you adjust for inflation) that IT experts everywhere are grappling with.

Learn More:
《Glossary: What is IT?》
《Glossary: What is Metaverse?》
《ArchiFiction Achieves Naked-Eye 3D Virtual Reality with GIGABYTE Workstations》
《GIGABYTE’s ARM Server Boosts Development of Smart Traffic Solution by 200%》
《Lowell Observatory Looks for Habitable Exoplanets with GIGABYTE Servers》

KDDI may have hit upon a solution: a revolutionary method of data center cooling known as “immersion cooling”. Immersion cooling is a liquid-based thermal management system that submerges the servers directly in a bath of dielectric, nonconductive coolant. Since this is more energy-efficient than air cooling, the data center’s PUE, which is its total energy consumption divided by its computing equipment’s energy consumption, can be lowered below 1.07, by KDDI’s estimates. That is a 43% reduction in electricity use when compared with air-cooled data centers, which may have an average PUE of around 1.7.《Glossary: What is PUE?》

By adopting immersion cooling in its micro data centers, KDDI hopes to create a win-win situation that fulfills two important aspects of its corporate vision. There are two important reasons why KDDI is exploring possible new designs for data centers. The first is that such a design may contribute to the digital transformation (DX) of Japan. The second is to meet the carbon neutrality goals outlined in "KDDI GREEN PLAN 2030", which is the company’s roadmap to combating climate change, protecting biodiversity, and ushering in the circular economy.

Learn More:
《Visit GIGABYTE’s Dedicated Solutions Page to Learn All about Immersion Cooling》
《How to Pick a Cooling Solution for Your Servers? A Tech Guide by GIGABYTE》

In fact, there are two types of immersion cooling. In the “two-phase” solution, the coolant undergoes a continuous cycle of vaporization and condensation within the sealed tank, and the heat is extracted from the vapors through a condenser at the top of the tank. In the “single-phase” solution, the coolant does not vaporize; instead, it is pumped to a coolant distribution unit (CDU) to remove the heat. GIGABYTE Technology, a leading brand of data center and server solutions, has already built a two-phase immersion cooling solution for a prominent IC foundry giant that leads the world in advanced process technology.

Learn More:
《Press Release: Building a Two-phase Immersion Cooling Solution for IC Foundry Giant, GIGABYTE Is Looking to Set New Standards for Modern Data Center》
《As Good as Gold: Immersion Cooling Accelerates Oil Extraction》

KDDI Corporation opted for single-phase immersion cooling. It invited GIGABYTE Technology to participate in its “container-type immersion cooling small data centers” project. GIGABYTE provided one R282-Z93 and two R182-Z91’s from its state-of-the-art line of Rack Servers to serve as the management and GPU computing nodes. Benefiting from GIGABYTE’s server products and expert knowledge, KDDI’s vision of “carbon neutral digital transformation” may be one step closer to reality.

Learn More:
《More information about GIGABYTE’s Rack Server》
《Glossary: What is GPU?》
《Glossary: What is Node?》

In Mitsubishi Heavy Industries’ Yokohama Hardtech Hub (YHH) in Yokohama, Japan, KDDI erected a 12ft shipping container measuring 3.66 meters in length, 2.44 meters in width, and 2.90 meters in height. Within, KDDI installed a micro-modular, rack-based immersion cooling system from GRC (Green Revolution Cooling) that offers 24 rack units (24U) of space for servers, and can cool up to 50kVA of heat. The coolant system is made by Mitsubishi Heavy Industries. The Japanese system integrator NEC Networks & System Integration Corp. (NESIC) also helped to set up KDDI’s innovative data center.

Learn More:
《See It Now! GIGABYTE Provides Single-Phase Immersion Cooling with GRC》
《Glossary: What is Rack Unit?》

Here is how single-phase immersion cooling keeps the data center’s temperature in check: during operation, heat generated by the servers is absorbed by the coolant, which is then piped to the CDU installed in an outward-facing closet partitioned into one end of the container. The heat exchanger in the CDU transfers the thermal energy from the coolant to a chilled water loop, which is kept cool by an external radiator. This is more energy-efficient than air or even direct-to-chip (D2C) liquid cooling. By KDDI’s estimates, when the data center is operating at its maximum capacity, the servers will use 50kVA of power, while the data center’s total power consumption will only be around 53.5kVA—therefore lowering the PUE below 1.07.《Glossary: What is Liquid Cooling?》

In addition to being more energy-efficient, there are other advantages, as well. Because immersion cooling can support a higher density of servers, the entire system fits snugly within a 12ft container. Immersion cooling can help to protect the servers from the environment, whether it is high temperatures or salt and dusts in the air. Thanks to the modular design, a number of installation site restrictions can be eliminated; therefore, it is expected that the new data centers will be able to overcome the difficulties of space allocation and material delivery, making them more flexible, mobile, and space-saving.

With the help of GIGABYTE and other suppliers, KDDI may be close to achieving its goal of deploying micro data centers that are not only mobile and eco-friendly, but can expedite edge computing across Japan. The adoption of immersion cooling is a valuable learning experience for KDDI’s IT experts, and it may become a role model that can be emulated by other ICT companies in Japan—because it represents the best of many important tech trends, such as carbon-neutral computing, edge computing, and small-scale data centers.

“We are impressed that the GIGABYTE servers were able to adapt to our liquid-based cooling system so smoothly,” says Mr. Shintaro Kitayama, Assistant Manager and Server Infrastructure Engineer at KDDI. “GIGABYTE leveraged its past experience in the telco industry to overcome the challenges of immersion cooling, which is very beneficial to our project.”

The single R282-Z93 that GIGABYTE provided for KDDI contains dual AMD EPYC™ 7643 CPUs and triple NVIDIA® A100 Tensor Core GPUs in a 2U chassis, while the two R182-Z91’s contain AMD EPYC™ 7713 CPUs and NVIDIA® T4 Tensor Core GPUs in 1U chassis. Based on GIGABYTE’s years of experience providing enterprise server solutions for the telco sector, this combination of CPUs and GPUs (or more accurately, GPGPUs) in small form factors are eminently suitable for the management and GPU computing nodes in a micro data center.

Learn More:
《GIGABYTE Solutions: NVIDIA A100 Tensor Core GPU Offers Unprecedented Acceleration at Every Scale》
《Glossary: What is GPGPU?》

The synergy between CPUs and GPGPUs, which is sometimes referred to as heterogeneous computing, is crucial for a data center that supports the public network. Not only are there a large number of users connecting to the data center at all times, the sheer amount of data in graphical form—photos, videos, even data that may be used by computer vision—is staggering. While the 3rd Gen AMD EPYC™ processors already contain an impressive number of cores and threads, it is still necessary to boost CPU performance with the NVIDIA® A100 and T4 GPUs. PCIe Gen 4.0 is used to enable faster data transmission between the processors.

Glossary:
《What is Heterogeneous Computing?》
《What is Computer Vision?》
《What is Core?》
《What is Thread?》
《What is PCIe?》

Mr. Kitayama explains, “We thought it necessary to supplement the CPUs with powerful GPUs to provide enough computing power on the data center and the edge site in a highly aggregated state, which will allow us to offer a rich variety of services for our customers. Immersion cooling opens the door to the most advanced GPUs, such as the A100, without encountering limitations in power consumption or cooling.”

The form factors of the R282-Z93 and R182-Z91, which can support a highly dense and scalable configuration of CPUs and GPUs, is another reason why the GIGABYTE servers are suitable for micro data centers. Immersion cooling has made it possible for the telco giant to experiment with high-density computing, which refers to the aggregation of a lot of computing resources within limited space. KDDI is able to reduce the data center’s physical footprint without giving up an iota of computing power, all the while retaining the option to scale out if necessary. The small size of the data center also makes it more convenient to load onto the back of a truck, where it can be carted around Japan to the locations that need it the most.《Glossary: What is Scale Out?》

Before the R282-Z93 and R182-Z91’s were shipped to Yokohama, the GIGABYTE team in Taiwan dedicated weeks of time to modifying and optimizing the servers for immersion cooling. Air-based cooling components, such as the fans, heat sinks, and air shrouds were removed. The servers’ temperature sensors, which were calibrated for the lower ambient temperatures of an air-cooled data center, were also replaced. GIGABYTE was happy to draw upon its experience with data center cooling solutions to help KDDI achieve the vision of a containerized small data center that utilizes immersion cooling.

After the servers arrived in Japan, KDDI conducted a round of operation tests, performance tests, and heat resistance tests to evaluate the cooling effectiveness and structural integrity of the servers. This was done to see if the servers could maintain peak performance without running into issues. The configurable TDP (thermal design power) of the CPUs could go as high as 240W—this is why, in an air-cooled data center, the ambient temperature is usually limited to 30°C or 35°C. However, because immersion cooling is much more efficient at dissipating heat, a much higher ambient temperature may be acceptable. KDDI tested the whole set-up extensively to make sure the micro data center would be ready for field deployment.《Glossary: What is TDP?》

In the course of testing, KDDI spotted other details to consider when adopting immersion cooling. For example, KDDI discovered that ink on labels may smudge when it comes in contact with the oil-based coolant, and this may form a pollutant that may be detrimental to the cooling system. This example proves that there are many things to watch out for when optimizing the servers for immersion cooling. Fortunately, GIGABYTE was glad to share its treasure trove of expert knowledge, as well as its extensive line of server products which are compatible with air, liquid, and immersion cooling, to help KDDI achieve its vision of the container-type immersion cooling small data center.

Learn More:
《Waseda University Decodes the Storm with GIGABYTE’s Computing Cluster》
《Using GIGABYTE, NIPA Cloud Soars Among CSP Giants in Thailand》

KDDI Corporation has embarked on a bold venture that may revolutionize the way we design and build data centers. The outlook is optimistic. As of April of 2022, KDDI is already testing immersion cooling in the KDDI Oyama Technical Center (KDDI Oyama TC) in the Tochigi Prefecture, about 90 kilometers north of Tokyo. The system is being tested against the “High Availability” standards of Tier 4 data centers, as outlined by ANSI/TIA-942-A. It may also be tested to see if immersion cooling may contribute to overcoming the problems of space limitations and heat dissipation and evaluate the operability for construction and maintenance. If all goes well, it is hoped that the container-type immersion cooling small data center may eventually see widespread deployment across Japan.

Learn More:
《Glossary: What is High Availability?》
《Glossary: What is 5G?》
《Free Downloadable Tech Guide: How to Build Your Data Center with GIGABYTE?》

“We are grateful for GIGABYTE’s active participation as a server manufacturer in our new cooling technology challenge, and for their provision of equipment and technical support,” says KDDI’s Mr. Masato Katou, Expert at Platform Technology Department, DX Solution Engineering Division, “As a manufacturer of IT equipment in the immersion field, KDDI hopes that GIGABYTE will lead the industry in the future with GIGABYTE’s advanced technological capabilities and know-how.”

GIGABYTE Technology is pleased to receive KDDI’s kind words of commendation. GIGABYTE also anticipates the widespread adoption of data center cooling solutions that will help industry leaders reduce carbon emissions, achieve carbon neutrality, and move the world toward a greener and more sustainable “net zero” future. This is in line with GIGABYTE Technology’s long-term CSR, ESG, and SDGs-related efforts, as well as its vision: “Upgrade Your Life”, which is focused on utilizing innovative high tech to deliver better business results while building a smarter, better world of tomorrow.

Learn More:
《More GIGABYTE Tech Guides: What is HPC, and How Can You Stand to Benefit?》