PCIe

computer

PCIe (Peripheral Component Interconnect Express) is the high-speed serial bus standard that connects a computer’s CPU/chipset to peripherals like GPUs, NVMe SSDs, and network cards. Given your NVMe/NVMeOF work, it’s the underlying transport your NVMe drives actually sit on

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
graph TB
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px;
    classDef node fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
    classDef pcie fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px;
    classDef memory fill:#fff3e0,stroke:#e65100,stroke-width:2px;
    classDef dev fill:#fce4ec,stroke:#880e4f,stroke-width:2px;
    classDef procLocal fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,stroke-dasharray: 5 5;
    classDef procRemote fill:#ffebee,stroke:#c62828,stroke-width:2px,stroke-dasharray: 5 5;
    classDef crosslink stroke:#f44336,stroke-width:2px,stroke-dasharray: 5 5;

    subgraph "Server Architecture"
        direction TB
        
        subgraph "NUMA Node 0 (Socket 0)"
            CPU0[CPU 0]:::node
            MEM0[(Local Memory)]:::memory
            RC0[PCIe Root Complex 0]:::pcie
            
            CPU0 --- MEM0
            CPU0 --- RC0
            
            NIC[InfiniBand NIC]:::dev
            NVME0[NVMe SSD 0]:::dev
            
            RC0 --- NIC
            RC0 --- NVME0
            
            P1((Process Pinned to 0)):::procLocal
            P1 -.->|Optimal: Direct Access| MEM0
            P1 -.->|Optimal: Local PCIe| NIC
        end
        
        CPU0 <-->|UPI/Infinity Fabric<br/>Cross-Socket Link| CPU1:::crosslink
        
        subgraph "NUMA Node 1 (Socket 1)"
            CPU1[CPU 1]:::node
            MEM1[(Local Memory)]:::memory
            RC1[PCIe Root Complex 1]:::pcie
            
            CPU1 --- MEM1
            CPU1 --- RC1
            
            NVME1[NVMe SSD 1]:::dev
            RC1 --- NVME1
            
            P2((Process Pinned to 1)):::procRemote
            P2 -.->|Optimal: Direct Access| MEM1
            P2 -.->|Sub-optimal: UPI Penalty| NIC
        end
    end

Server Architecture Topology

NUMA Node 0 (Socket 0)
The CPU acts as the memory controller for its local RAM banks and provides PCIe lanes via its internal Root Complex. The InfiniBand NIC and NVMe SSD 0 plug directly into slots governed by this CPU.

  • Process A (Pinned to Node 0): When this process runs, it has fast, direct access to the local memory. When it initiates RDMA via the InfiniBand NIC, the PCIe traffic stays entirely within Node 0.

Cross-Socket Link (UPI / Infinity Fabric)
This is the bridge connecting the two CPUs. It carries memory reads/writes and PCIe traffic when a component on one side needs to talk to the other side.

NUMA Node 1 (Socket 1)
The second CPU has its own memory banks and its own PCIe Root Complex. NVMe SSD 1 plugs in here.

  • Process B (Pinned to Node 1): This process has fast access to its local memory. However, if it tries to use the InfiniBand NIC (which is physically attached to Node 0), the data must travel across the UPI link, incurring a cross-socket latency penalty.