The RBD CSI plugin is able to provision new RBD images and attach and mount those to workloads.
Configuration
Available command line arguments:
| Option | Default value | Description |
|---|---|---|
--endpoint |
unix:///tmp/csi.sock |
CSI endpoint, must be a UNIX socket |
--csi-addons-endpoint |
unix:///tmp/csi-addons.sock |
CSI-Addons endpoint, must be a UNIX socket |
--drivername |
rbd.csi.ceph.com |
Name of the driver (Kubernetes: provisioner field in StorageClass must correspond to this value) |
--nodeid |
empty | This node’s ID |
--type |
empty | Driver type: [rbd/cephfs]. If the driver type is set to rbd it will act as a rbd plugin or if it’s set to cephfs will act as a cephfs plugin |
--instanceid |
“default” | Unique ID distinguishing this instance of Ceph CSI among other instances, when sharing Ceph clusters across CSI instances for provisioning |
--pidlimit |
0 | Configure the PID limit in cgroups. The container runtime can restrict the number of processes/tasks which can cause problems while provisioning (or deleting) a large number of volumes. A value of -1 configures the limit to the maximum, 0 does not configure limits at all. |
--metricsport |
8080 |
TCP port for liveness metrics requests |
--metricspath |
"/metrics" |
Path of prometheus endpoint where metrics will be available |
--polltime |
"60s" |
Time interval in between each poll |
--timeout |
"3s" |
Probe timeout in seconds |
--clustername |
empty | Cluster name to set on RBD image |
--domainlabels |
empty | Kubernetes node labels to use as CSI domain labels for topology aware provisioning, should be a comma separated value (ex:= “failure-domain/region,failure-domain/zone”) |
--rbdhardmaxclonedepth |
8 |
Hard limit for maximum number of nested volume clones that are taken before a flatten occurs |
--rbdsoftmaxclonedepth |
4 |
Soft limit for maximum number of nested volume clones that are taken before a flatten occurs |
--skipforceflatten |
false |
skip image flattening on kernel < 5.2 which support mapping of rbd images which has the deep-flatten feature |
--maxsnapshotsonimage |
450 |
Maximum number of snapshots allowed on rbd image without flattening |
--setmetadata |
true |
Deprecated. Set metadata on volume. This flag will be removed in a future release. |
--enable-read-affinity |
false |
enable read affinity |
--crush-location-labels |
empty | Kubernetes node labels that determine the CRUSH location the node belongs to, separated by ‘,’. Note: These labels will be replaced if crush location labels are defined in the ceph-csi-config ConfigMap for the specific cluster. |
--logslowopinterval |
30s |
Log slow operations at the specified rate. Operation is considered slow if it outlives its deadline. |
--feature-gates |
empty | Comma-separated list of feature gates (e.g., SlowGRPCRestart=false). Available gates: SlowGRPCRestart (default: true) — restart the process when a unary gRPC call is stuck for more than 10 minutes. |
Available volume parameters:
| Parameter | Required | Description |
|---|---|---|
clusterID |
yes | String representing a Ceph cluster, must be unique across all Ceph clusters in use for provisioning, cannot be greater than 36 bytes in length, and should remain immutable for the lifetime of the Ceph cluster in use |
pool |
yes | Ceph pool into which the RBD image shall be created |
dataPool |
no | Ceph pool used for the data of the RBD images. |
volumeNamePrefix |
no | Prefix to use for naming RBD images (defaults to csi-vol-). |
snapshotNamePrefix |
no | Prefix to use for naming RBD snapshot images (defaults to csi-snap-). |
imageFeatures |
no | RBD image features. CSI RBD currently supports layering, journaling, exclusive-lock, object-map, fast-diff, deep-flatten features. deep-flatten is added for cloned images. Refer https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#image-features for image feature dependencies. |
mkfsOptions |
no | Options to pass to the mkfs command while creating the filesystem on the RBD device. Check the man-page for the mkfs command for the filesystem for more details. When mkfsOptions is set here, the defaults will not be used, consider including them in this parameter. |
tryOtherMounters |
no | Specifies whether to try other mounters in case if the current mounter fails to mount the rbd image for any reason |
mapOptions |
no | Map options to use when mapping rbd image. See krbd and nbd options. |
unmapOptions |
no | Unmap options to use when unmapping rbd image. See krbd and nbd options. |
csi.storage.k8s.io/provisioner-secret-name, csi.storage.k8s.io/node-stage-secret-name |
yes (for Kubernetes) | name of the Kubernetes Secret object containing Ceph client credentials. Both parameters should have the same value |
csi.storage.k8s.io/provisioner-secret-namespace, csi.storage.k8s.io/node-stage-secret-namespace |
yes (for Kubernetes) | namespaces of the above Secret objects |
mounter |
no | if set to rbd-nbd, use rbd-nbd on nodes that have rbd-nbd and nbd kernel modules to map rbd images |
encrypted |
no | disabled by default, use "true" to enable either LUKS or fscrypt encryption on PVC and "false" to disable it. Do not change for existing storageclasses |
encryptionSectorSize |
no | set the sector size that is used to perform I/O. Size must be a power of two and in between 512 and 4096. Typical values are 4096 and 512. |
encryptionCipher |
no | set the cipher that is used for the volume encryption. Supported Keywords: aes-xts-plain64 (default), serpent-xts-plain64, aes-xts-random, serpent-xts-random |
encryptionKeySize |
no | set the key size used for encryption. Typical key sizes are 128, 256 and 512. Default 256 (or 512 when using an xts cipher mode) |
integrityMode |
no | set and enable the integrity verification for volume encryption. Supported Keywords: hmac-sha256, hmac-sha512 |
encryptionKMSID |
no | required if encryption is enabled and a kms is used to store passphrases |
encryptionType |
no | Either block or file. If unset or block use LUKS block device encryption. If file use ext4 fscrypt to encrypt on the file system level (requires kernel support). |
stripeUnit |
no | stripe unit in bytes |
stripeCount |
no | objects to stripe over before looping |
objectSize |
no | object size in bytes |
baseIops |
no | the base limit of operations per second |
maxIops |
no | the max limit of operations per second |
baseReadIops |
no | the base limit of read operations per second |
maxReadIops |
no | the max limit of read operations per second |
baseWriteIops |
no | the base limit of write operations per second |
maxWriteIops |
no | the max limit of write operations per second |
baseBps |
no | the base limit of bytes per second |
maxBps |
no | the max limit of bytes per second |
baseReadBps |
no | the base limit of read bytes per second |
maxReadBps |
no | the max limit of read bytes per second |
baseWriteBps |
no | the base limit of write bytes per second |
maxWriteBps |
no | the max limit of write bytes per second |
iopsPerGiB |
no | the limit of operations per GiB |
readIopsPerGiB |
no | the limit of read operations per GiB |
writeIopsPerGiB |
no | the limit of write operations per GiB |
bpsPerGiB |
no | the limit of bytes per GiB |
readBpsPerGiB |
no | the limit of read bytes per GiB |
writeBpsPerGiB |
no | the limit of write bytes per GiB |
baseVolSizeBytes |
no | the min size of volume what use to calculate qos beased on capacity |
extraDeploy |
no | array of extra objects to deploy with the release |
NOTE: An accompanying CSI configuration file, needs to be provided to the running pods. Refer to Creating CSI configuration for more information.
NOTE: A suggested way to populate and retain uniqueness of the clusterID is to use the output of ceph fsid of the Ceph cluster to be used for provisioning.
Required secrets:
User credentials, with required access to the pool being used in the storage class, is required for provisioning new RBD images.
Deployment
Use the rbd templates(/path/to/ceph-csi/deploy/rbd/kubernetes)
Your Kubernetes cluster must allow privileged pods (i.e. --allow-privileged flag must be set to true for both the API server and the kubelet). Moreover, as stated in the mount propagation docs, the Docker daemon of the cluster nodes must allow shared mounts.
YAML manifests are located in deploy/rbd/kubernetes.
Create CSIDriver object:
1 | kubectl create -f csidriver.yaml |
Deploy RBACs for sidecar containers and node plugins:
1 | kubectl create -f csi-provisioner-rbac.yaml |
Those manifests deploy service accounts, cluster roles and cluster role bindings. These are shared for both RBD and CephFS CSI plugins, as they require the same permissions.
Deploy ConfigMap for CSI plugins:
1 | kubectl create -f csi-config-map.yaml |
The configmap deploys an empty CSI configuration that is mounted as a volume within the Ceph CSI plugin pods. To add a specific Ceph clusters configuration details, refer to Creating CSI configuration for RBD based provisioning for more information.
Deploy Ceph configuration ConfigMap for CSI pods:
1 | kubectl create -f ../../ceph-conf.yaml |
Deploy prerequisites for CSI Snapshot:
If you intend to use the snapshot functionality in Kubernetes cluster, please refer to snap-clone.md
Deploy CSI sidecar containers:
1 | kubectl create -f csi-rbdplugin-provisioner.yaml |
Deploys deployment of provision which includes external-provisioner ,external-attacher,csi-snapshotter sidecar containers and CSI RBD plugin.
Deploy RBD CSI driver:
1 | kubectl create -f csi-rbdplugin.yaml |
Deploys a daemon set with two containers: CSI node-driver-registrar and the CSI RBD driver.
NOTE: In case you want to use a different release version, replace canary with the release version in the provisioner and nodeplugin YAMLs.
1 | # for stable functionality replace canary with latest release version |
Check the release version here.
Verify
After successfully completing the steps above, you should see output similar to this:
1 | $ kubectl get all |
Once the CSI plugin configuration is updated with details from a Ceph cluster of choice, you can try deploying a demo pod from examples/rbd using the instructions provided to test the deployment further.
They are two halves of the same RBD CSI driver: node I/O vs cluster control plane. Both must run; a PVC needs the provisioner to create the image and the plugin to map it on the node.[1][2]
Split of work
csi-rbdplugin.yaml |
csi-rbdplugin-provisioner.yaml |
|
|---|---|---|
| Kind | DaemonSet | Deployment |
| Where | Every worker (ubuntu2, ubuntu3) |
2 replicas, not per node |
| CSI role | Node service | Controller service |
| Does | rbd map / unmap, mkfs, mount into the pod |
rbd create / delete, expand, snapshot, attach metadata |
| Host paths | /dev, /sys, /lib/modules, kubelet plugin dir |
Mostly ConfigMaps + emptyDir; /dev optional |
| Scale | One pod per node | 2 is enough (leader election) |
CreateVolume is the provisioner. NodeStageVolume / NodePublishVolume is the node plugin. That is why timezone /dev mounts belong on the plugin DaemonSet, not as a third provisioner replica.
csi-rbdplugin.yaml (node):
csi-rbdplugin— talks to kubelet via the CSI socket, maps the imagedriver-registrar— registersrbd.csi.ceph.comwith kubeletliveness-prometheus— health
Without this on a node, the PVC binds but the pod stays ContainerCreating (FailedMount).
csi-rbdplugin-provisioner.yaml (controller, your 7/7 pod):
csi-provisioner— watches PVCs, callsCreateVolume/DeleteVolumecsi-attacher— VolumeAttachmentcsi-resizer— PVC expandcsi-snapshotter— VolumeSnapshotcsi-rbdplugin— controller-side Ceph client (same binary, different flags)- extra sidecars (liveness / CSI-Addons)
Without this, PVCs stay Pending (waiting for a volume to be created).
storage class
1 | kubectl create -f secret.yaml |
PVC flows
1 | PVC → provisioner CreateVolume → RBD image in kubernetes-meta (data on kubernetes-ec) |