Kubernetes Volume

Kubernetes

overview

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
sequenceDiagram
    autonumber
    participant U as User
    participant A as API Server
    participant P as external-provisioner
    participant C as CSI Controller
    participant K as Kubelet
    participant N as CSI Node Plugin
    participant H as Host (LVM)
    participant D as containerd/runc

    rect rgb(230, 245, 255)
    Note over U,H: PROVISION & BIND
    U->>A: Create PVC (StorageClass: lvm-csi)
    P->>C: CreateVolume
    C->>H: lvcreate -n lv-xxx vg0
    C-->>P: volume_id
    P->>A: Create PV -> PV/PVC Bound
    U->>A: Create Pod (uses PVC)
    A->>K: Pod scheduled to node
    end

    rect rgb(235, 255, 235)
    Note over K,H: MOUNT (per pod)
    K->>N: NodeStageVolume
    N->>H: lvchange -ay + mkfs + mount staging<br/>/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pv-name/globalmount
    K->>N: NodePublishVolume
    N->>H: mount --bind staging -> target<br/>/var/lib/kubelet/pods/uid/volumes/kubernetes.io~csi/pvc/mount
    end

    rect rgb(255, 245, 225)
    Note over K,D: CONTAINER START
    K->>D: CRI CreateContainer (Mounts: host target -> mountPath)
    D->>D: runc bind-mounts into rootfs<br/>/run/containerd/io.containerd.runtime.v2.task/k8s.io/cid/rootfs/mountPath
    D-->>K: Container running
    end

    rect rgb(255, 235, 235)
    Note over U,H: TEARDOWN (reverse order)
    U->>A: Delete Pod
    K->>D: StopContainer -> task destroyed, rootfs path gone
    K->>N: NodeUnpublishVolume -> umount pod target path
    K->>N: NodeUnstageVolume -> umount staging, lvchange -an
    U->>A: Delete PVC
    P->>C: DeleteVolume
    C->>H: lvremove vg0/lv-xxx
    end

command

1
2
3
4
5
6
7
8
9
10
11
12
13
# 1. Get the full container ID for a pod's container
CID=$(crictl ps --name <container-name> -q | head -1)          # truncated 13-char
CID=$(crictl ps --name <container-name> --no-trunc -q | head -1)  # full 64-char

# or filter by pod ID
PODID=$(crictl pods --name <pod-name> --namespace <ns> -q)
CID=$(crictl ps --pod $PODID --no-trunc -q)

# 2. Build the rootfs path
echo /run/containerd/io.containerd.runtime.v2.task/k8s.io/$CID/rootfs

# (tip) check pod metadata base on CID
crictl inspect $CID | jq '.status.labels'

Here’s the end-to-end reference flow of how a Pod’s volume target path is created in Kubernetes with a CSI driver (using your LVM-on-host + containerd scenario), from PVC all the way to the containerd rootfs view.

There are two distinct “target” concepts people often conflate:

the kubelet target path (/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<vol-name>/mount) created by the CSI node plugin

and

the containerd task rootfs (/run/containerd/io.containerd.runtime.v2.task/k8s.io/<container-id>/rootfs) created later by the runtime when the container starts.

The CSI driver never touches the containerd path directly — the kubelet hands the CSI target path to the CRI, and containerd/runc bind-mounts it into the container’s mount namespace.

Stage 1 — Provision and Bind

  1. You create a PVC referencing a StorageClass with provisioner: <your-csi-driver>.
  2. The external-provisioner sidecar watches PVCs and calls CreateVolume on the CSI controller plugin. For an LVM driver this typically creates the LV on the target host (or records intent for topology-aware late binding with volumeBindingMode: WaitForFirstConsumer).
  3. The provisioner creates the PV object; the PersistentVolume controller binds PV↔PVC.
  4. Pod is created referencing the PVC; scheduler picks a node (with WaitForFirstConsumer, provisioning happens after scheduling so the LV lands on the right node).

Stage 2 — Attach (ControllerPublishVolume)

If the driver reports attach_required: true, the Attach/Detach controller creates a VolumeAttachment object, and the external-attacher calls ControllerPublishVolume to “attach” the volume to the node. For pure local LVM drivers this is usually a no-op or skipped entirely (CSIDriver.spec.attachRequired: false), since the LV is already local.

Stage 3 — NodeStageVolume (Staging Path)

Once the pod lands on the node, the kubelet’s Volume Manager (reconciler loop comparing desired vs. actual state of world) drives the mount:

  • Kubelet’s in-tree CSI mounter calls NodeStageVolume over the driver’s UNIX socket.
  • Staging path: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv-name>/globalmount (for block volumes: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/<pv-name>).
  • Your LVM driver here: lvchange -ay vg/lv, format if the LV has no filesystem, then mount /dev/vg/lv <staging-path>.
  • Staging happens once per PV per node, even if multiple pods use it.

Stage 4 — NodePublishVolume (Target Path)

This is where the per-pod target path is born:

  • Kubelet calls NodePublishVolume with target_path = /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<pvc-name>/mount.
  • The driver performs a bind mount: mount --bind <staging-path> <target-path>.
  • The path is stable for the pod UID; kubelet’s reconciler will re-issue NodePublishVolume if the mount is lost, and drivers with RequiresRepublish get repeated calls.

So on the host you now have:

1
2
/dev/mapper/vg-lv  →  /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv>/globalmount  
                    →  (bind) /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount

Stage 5 — CRI → containerd → the rootfs Path

Now the runtime enters:

  1. Kubelet calls CRI CreateContainer, passing Mounts: host_path = /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount, container_path = <volumeMounts.mountPath>.
  2. Containerd’s CRI plugin builds the OCI spec (rootfs from the snapshotter + the mount list) and starts a runtime v2 shim task.
  3. runc creates the container’s mount namespace and bind-mounts the kubelet target path into the rootfs at mountPath before pivot_root.
  4. Containerd exposes the running container’s rootfs on the host at /run/containerd/io.containerd.runtime.v2.task/k8s.io/<container-id>/rootfs — this is the path you see in config.json / runc spec output.

Because that rootfs bind shares the same underlying directory the mounts were applied to, you can see the LV’s content from the host at:

1
/run/containerd/io.containerd.runtime.v2.task/k8s.io/ca47433f.../rootfs/<mountPath>

This rootfs path is also what vulnerability scanners and debug tooling walk when inspecting container filesystems.

Path Ownership Map

Path Created by Lifetime
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv>/globalmount CSI node plugin (NodeStageVolume) Per PV per node
/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pvc>/mount CSI node plugin (NodePublishVolume, bind mount) Per pod UID
/run/containerd/io.containerd.runtime.v2.task/k8s.io/<cid>/rootfs containerd shim / runc Per container task
/var/log/pods/<ns>_<pod>_<uid>/<container>/*.log → symlinks into containerd log dir kubelet Per pod

Verify on the Host

1
2
3
4
5
6
7
8
9
10
11
# find the CSI staging + target mounts for a pod
findmnt -R /var/lib/kubelet/pods/<pod-uid>/volumes
findmnt /var/lib/kubelet/plugins/kubernetes.io/csi/pv/<pv-name>/globalmount

# map containerd task → rootfs and confirm the volume inside
ctr -n k8s.io tasks ls
findmnt /run/containerd/io.containerd.runtime.v2.task/k8s.io/<cid>/rootfs
ls /run/containerd/io.containerd.runtime.v2.task/k8s.io/<cid>/rootfs/<mountPath>

# confirm the LV backing it
lvs -o lv_name,vg_name,lv_path,devices

Teardown is the exact reverse: NodeUnpublishVolume (unmount target) → container task destroyed (rootfs path disappears) → NodeUnstageVolume (unmount staging, optionally lvchange -an) → ControllerUnpublishVolume → DeleteVolume when the PV is released.

OverlayFS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
flowchart TD
    A["LVM Logical Volume<br>/dev/mapper/csi--lvm-pvc--85dfd566...<br>(block device from csi-lvm driver)"]
    B["XFS Mount<br>/var/lib/kubelet/pods/dfd4c191.../<br>volumes/kubernetes.io~csi/pvc-85dfd566.../mount"]
    C["mount/rootfs/fs<br>overlay upperdir (writable)"]
    D["mount/rootfs/work<br>overlay workdir"]
    E["Container Image Layers<br>lowerdir=482/fs:481/fs:...:404/fs<br>(read-only)"]
    F["OverlayFS Merged Mount<br>/run/containerd/io.containerd.runtime.v2.task/<br>k8s.io/52b9bde58b98.../rootfs"]
    G["Container 52b9bde58b989<br>sees merged rootfs as /<br>(bin etc usr code package workspace ...)"]

    A --> B
    B --> C
    B --> D
    E --> F
    C --> F
    D --> F
    F --> G
    G -. "all writes land here" .-> C

OverlayFS is the union filesystem that stitches the read-only container image and writable PVC-backed directory into a single coherent root filesystem for the container. It’s the “glue” that lets one container see a complete Linux filesystem while the actual data lives in two very different places.

Component host mount Role
lowerdir 482/fs:481/fs:...:404/fs The container image layers — read-only, shared, immutable base content
upperdir .../pvc-85dfd566.../mount/rootfs/fs The writable layer — every change the container makes lands here, persisted on your LVM volume
workdir .../pvc-85dfd566.../mount/rootfs/work Kernel-internal scratch space for atomic copy-up operations; must be on the same filesystem as upperdir
merged mount /run/containerd/.../52b9bde58b98.../rootfs The unified view — what the container actually sees as /
  • Reads: when the container opens a file, OverlayFS checks the upperdir first; if not found, it walks the lowerdirs top-to-bottom. Unmodified files are served straight from the image layers at native speed — after open, operations go directly to the underlying filesystem.
  • New writes: files the container creates go directly into the upperdir — in your case, onto the XFS filesystem on the LVM LV. Nothing touches the image layers.
  • Modifying an image file (copy-up): if the container writes to a file that only exists in a lowerdir, the kernel first copies the entire file into the upperdir, then applies the write to that copy. The original in the image layer stays untouched. The workdir is used to stage this so a crash mid-copy leaves the old version intact rather than a half-written file.
  • Deletes: removing a lower-layer file doesn’t touch the image — OverlayFS drops a “whiteout” marker in the upperdir so the file simply disappears from the merged view.

So in one sentence: overlay is what lets the container believe it owns a normal writable root filesystem, while in reality the reads come from shared read-only image layers and every write is silently redirected into your LVM-backed PVC.

mount propagation

Mode Linux Container sees new host mounts Host sees mounts created in the container
None (default) rprivate No No
HostToContainer rslave Yes No
Bidirectional rshared Yes Yes