What Is Infiniband Network And Its Architecture? A Complete Guide

As central processing units (CPUs) and graphic processing units (GPUs) scale in computing power, traditional network bottlenecks have become the primary limitation in modern data centers. To feed data to processors fast enough, network architects are increasingly turning to a specialized High-Speed Interconnect (HSI) known as InfiniBand.

But what exactly is InfiniBand, and how does its architecture differ from the traditional TCP/IP Ethernet networks we have used for decades?

Whether you are designing a high-performance computing (HPC) cluster, deploying machine learning workloads, or studying modern distributed architectures, this guide breaks down the core components, the five-layer architecture, and the engineering principles that make InfiniBand the gold standard for AI data centers.

What is an InfiniBand Network?

InfiniBand is an open-standard network communications protocol specifically designed for high-throughput, extremely low-latency inter-node communication.

Unlike traditional networks designed for general-purpose internet traffic, InfiniBand was engineered from the ground up to connect servers to other servers, and servers to storage devices, within a tightly coupled environment (such as a supercomputer or a massive AI training cluster).

Its core vision revolves around three engineering goals:

1. Maximum Network Utilization: Eliminating dropped packets and congestion.

2. Maximum CPU/GPU Utilization: Freeing the processors from managing network traffic.

3. Minimum Latency: Moving data from Node A to Node B in nanoseconds, not milliseconds.

The Core Differentiator: RDMA vs. TCP/IP

To understand the InfiniBand architecture, you must understand the problem it solves: Operating System Overhead.

In a traditional TCP/IP model, data must pass through the operating system (OS) kernel. The CPU has to context-switch, copy data from the application buffer to the OS buffer, segment it, and push it to the network card. This software intervention creates micro-delays (latency) and burns valuable CPU cycles.

InfiniBand solves this using RDMA (Remote Direct Memory Access). RDMA bypasses the OS kernel entirely. An application on Server A can write data directly into the application memory space of Server B over the network. The hardware (the network adapter) handles the entire transfer. This “kernel bypass” mechanism is what enables InfiniBand to achieve near-zero latency and massive throughput.

The 4 Hardware Components of an InfiniBand Network

Before diving into the protocol layers, an InfiniBand subnet physically consists of four primary components:

1.Host Channel Adapters (HCAs): The InfiniBand equivalent of an Ethernet NIC. HCAs sit inside the servers and physically execute the transport layer and RDMA operations in hardware.

2.InfiniBand Switches: High-speed switching fabrics that move traffic based on local routing tables, completely independent of the OS.

3.The Subnet Manager (SM): Unlike Ethernet (which uses broadcast protocols like ARP to find devices), InfiniBand uses a centralized Subnet Manager. The SM discovers all nodes, assigns addresses (LIDs), and programs the routing tables directly into the switches, eliminating network-clogging broadcast storms.

4.Physical Cabling: The optical or copper links connecting the HCAs to the switches (e.g., DAC, AOC, or Optical Transceivers).

The InfiniBand Architecture: The 5 Layers Explained

The InfiniBand Architecture (IBA) is structured into five layers. While it looks somewhat similar to the OSI model, it operates entirely differently to support hardware-level offloading.

1. Upper Layer (The Consumers)

This layer defines how applications access the InfiniBand messaging services. Instead of web browsers, the “consumers” here are high-performance computing libraries. Examples include MPI (Message Passing Interface) for parallel computing, and NCCL (NVIDIA Collective Communication Library) used to synchronize multiple GPUs during AI training.

2. Transport Layer (Hardware Execution)

In TCP/IP, the transport layer (TCP) is managed by software in the OS. In InfiniBand, the transport layer is hardcoded directly into the HCA hardware. It establishes the end-to-end “virtual channel,” handles segmentation and reassembly natively, and delivers the data payload directly into the receiving application’s memory buffer.

3. Network Layer (Routing Between Subnets)

The network layer is only utilized when traffic needs to leave a specific InfiniBand subnet to reach another. It uses Global IDs (GIDs) and InfiniBand Routers to scale networks across different physical locations while maintaining traffic isolation.

4. Link Layer (The Lossless Magic)

The Link Layer is responsible for moving packets across local switches. It assigns Local IDs (LIDs) to nodes. More importantly, this layer handles Credit-Based Flow Control. Traditional Ethernet is a “lossy” network—if a switch buffer is full, it drops packets. InfiniBand is lossless. The receiving port constantly tells the sending port exactly how much buffer space it has left. If the receiver is full, the sender physically pauses transmission. Packets are never dropped due to congestion.

5. Physical Layer (The Copper and Glass)

The physical layer specifies the signaling protocols, port shapes, and the actual cables carrying the data. Modern InfiniBand networks operate at incredible bandwidths, including HDR (200G), NDR (400G), and XDR (800G).

Because the physical layer directly dictates the raw bandwidth of the AI cluster, selecting the correct interconnects is critical. Data center architects typically utilize:

• DAC (Direct Attach Copper): For cost-effective, zero-power links within the same server rack.

• AOC (Active Optical Cables): For high-speed, lightweight connections between racks (Row-to-Row).

• Optical Transceivers (SR4/DR4): Paired with MPO fiber trunks for longer spine-leaf topologies.

As an OEM manufacturer operating directly from Optical Valley, Wolontek provides the strict-tolerance physical layer required for high-performance networks. We engineer and supply factory-direct, fully compatible InfiniBand DACs, AOCs, and Optical Transceivers (100G to 800G) designed to support lossless topologies without the massive vendor markups.