Kubernetes Volume

Kubernetes 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
sequenceDiagram
    autonumber
    participant U as User
    participant A as API Server
    participant P as external-provisioner
    participant C as CSI Controller
    participant K as Kubelet
    participant N as CSI Node Plugin
    participant H as Host (LVM)
    participant D as containerd/runc

    rect rgb(230, 245, 255)
    Note over U,H: PROVISION & BIND
    U->>A: Create PVC (StorageClass: lvm-csi)
    P->>C: CreateVolume
    C->>H: lvcreate -n lv-xxx vg0
    C-->>P: volume_id
    P->>A: Create PV -> PV/PVC Bound
    U->>A: Create Pod (uses PVC)
    A->>K: Pod scheduled to node
    end

    rect rgb(235, 255, 235)
    Note over K,H: MOUNT (per pod)
    K->>N: NodeStageVolume
    N->>H: lvchange -ay + mkfs + mount staging<br/>/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-name/globalmount
    K->>N: NodePublishVolume
    N->>H: mount --bind staging -> target<br/>/var/lib/kubelet/pods/uid/volumes/kubernetes.io~csi/pvc/mount
    end

    rect rgb(255, 245, 225)
    Note over K,D: CONTAINER START
    K->>D: CRI CreateContainer (Mounts: host target -> mountPath)
    D->>D: runc bind-mounts into rootfs<br/>/run/containerd/io.containerd.runtime.v2.task/k8s.io/cid/rootfs/mountPath
    D-->>K: Container running
    end

    rect rgb(255, 235, 235)
    Note over U,H: TEARDOWN (reverse order)
    U->>A: Delete Pod
    K->>D: StopContainer -> task destroyed, rootfs path gone
    K->>N: NodeUnpublishVolume -> umount pod target path
    K->>N: NodeUnstageVolume -> umount staging, lvchange -an
    U->>A: Delete PVC
    P->>C: DeleteVolume
    C->>H: lvremove vg0/lv-xxx
    end

command

1
2
3
4
5
6
7
8
9
10
# 1. Get the full container ID for a pod's container
CID=$(crictl ps --name <container-name> -q | head -1)          # truncated 13-char
CID=$(crictl ps --name <container-name> --no-trunc -q | head -1)  # full 64-char

# or filter by pod ID
PODID=$(crictl pods --name <pod-name> --namespace <ns> -q)
CID=$(crictl ps --pod $PODID --no-trunc -q)

# 2. Build the rootfs path
echo /run/containerd/io.containerd.runtime.v2.task/k8s.io/$CID/rootfs

Here’s the end-to-end reference flow of how a Pod’s volume target path is created in Kubernetes with a CSI driver (using your LVM-on-host + containerd scenario), from PVC all the way to the containerd rootfs view.

The Big Picture

There are two distinct “target” concepts people often conflate: the kubelet target path (/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<vol-name>/mount) created by the CSI node plugin, and the containerd task rootfs (/run/containerd/io.containerd.runtime.v2.task/k8s.io/<container-id>/rootfs) created later by the runtime when the container starts. The CSI driver never touches the containerd path directly — the kubelet hands the CSI target path to the CRI, and containerd/runc bind-mounts it into the container’s mount namespace.

Stage 1 — Provision and Bind

  1. You create a PVC referencing a StorageClass with provisioner: <your-csi-driver>.
  2. The external-provisioner sidecar watches PVCs and calls CreateVolume on the CSI controller plugin. For an LVM driver this typically creates the LV on the target host (or records intent for topology-aware late binding with volumeBindingMode: WaitForFirstConsumer).
  3. The provisioner creates the PV object; the PersistentVolume controller binds PV↔PVC.
  4. Pod is created referencing the PVC; scheduler picks a node (with WaitForFirstConsumer, provisioning happens after scheduling so the LV lands on the right node).

Stage 2 — Attach (ControllerPublishVolume)

If the driver reports attach_required: true, the Attach/Detach controller creates a VolumeAttachment object, and the external-attacher calls ControllerPublishVolume to “attach” the volume to the node. For pure local LVM drivers this is usually a no-op or skipped entirely (CSIDriver.spec.attachRequired: false), since the LV is already local.

Stage 3 — NodeStageVolume (Staging Path)

Once the pod lands on the node, the kubelet’s Volume Manager (reconciler loop comparing desired vs. actual state of world) drives the mount:[2]

  • Kubelet’s in-tree CSI mounter calls NodeStageVolume over the driver’s UNIX socket.
  • Staging path: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv-name>/globalmount (for block volumes: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/<pv-name>).
  • Your LVM driver here: lvchange -ay vg/lv, format if the LV has no filesystem, then mount /dev/vg/lv <staging-path>.
  • Staging happens once per PV per node, even if multiple pods use it.

Stage 4 — NodePublishVolume (Target Path)

This is where the per-pod target path is born:

  • Kubelet calls NodePublishVolume with target_path = /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<pvc-name>/mount.
  • The driver performs a bind mount: mount --bind <staging-path> <target-path>.
  • The path is stable for the pod UID; kubelet’s reconciler will re-issue NodePublishVolume if the mount is lost, and drivers with RequiresRepublish get repeated calls.

So on the host you now have:

1
2
/dev/mapper/vg-lv  →  /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv>/globalmount
                    →  (bind) /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount

Stage 5 — CRI → containerd → the rootfs Path

Now the runtime enters:

  1. Kubelet calls CRI CreateContainer, passing Mounts: host_path = /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount, container_path = <volumeMounts.mountPath>.
  2. Containerd’s CRI plugin builds the OCI spec (rootfs from the snapshotter + the mount list) and starts a runtime v2 shim task.
  3. runc creates the container’s mount namespace and bind-mounts the kubelet target path into the rootfs at mountPath before pivot_root.
  4. Containerd exposes the running container’s rootfs on the host at /run/containerd/io.containerd.runtime.v2.task/k8s.io/<container-id>/rootfs — this is the path you see in config.json / runc spec output.

Because that rootfs bind shares the same underlying directory the mounts were applied to, you can see the LV’s content from the host at:

1
/run/containerd/io.containerd.runtime.v2.task/k8s.io/ca47433f.../rootfs/<mountPath>

This rootfs path is also what vulnerability scanners and debug tooling walk when inspecting container filesystems.

Path Ownership Map

Path Created by Lifetime
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv>/globalmount CSI node plugin (NodeStageVolume) Per PV per node
/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount CSI node plugin (NodePublishVolume, bind mount) Per pod UID
/run/containerd/io.containerd.runtime.v2.task/k8s.io/<cid>/rootfs containerd shim / runc Per container task
/var/log/pods/<ns>_<pod>_<uid>/<container>/*.log → symlinks into containerd log dir kubelet Per pod

Verify on the Host

1
2
3
4
5
6
7
8
9
10
11
# find the CSI staging + target mounts for a pod
findmnt -R /var/lib/kubelet/pods/<pod-uid>/volumes
findmnt /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv-name>/globalmount

# map containerd task → rootfs and confirm the volume inside
ctr -n k8s.io tasks ls
findmnt /run/containerd/io.containerd.runtime.v2.task/k8s.io/<cid>/rootfs
ls /run/containerd/io.containerd.runtime.v2.task/k8s.io/<cid>/rootfs/<mountPath>

# confirm the LV backing it
lvs -o lv_name,vg_name,lv_path,devices

Teardown is the exact reverse: NodeUnpublishVolume (unmount target) → container task destroyed (rootfs path disappears) → NodeUnstageVolume (unmount staging, optionally lvchange -an) → ControllerUnpublishVolume → DeleteVolume when the PV is released.[11][1]