linux kernel

linux

kernel log

1
2
3
4
5
6
# check
dmesg -w          # follow the kernel ring buffer, like tail -f
dmesg -Tw         # same but with human-readable timestamps

journalctl -kf    # follow kernel messages via journald (better timestamping)
journalctl -kf -p warning   # only warnings and above

check log level

1
2
3
4
# See current values: console_loglevel default_message_loglevel ...
cat /proc/sys/kernel/printk
# print 
4       4       1       7

/proc/sys/kernel/printk controls the Linux kernel’s printk() logging thresholds—primarily which kernel messages are emitted to the console. It does not control whether messages are retained in the kernel log buffer, which you can inspect with dmesg.[1]

For this value:

1
4  4  1  7

the four integers mean:

Position Name Value Effect
1 console_loglevel 4 Console displays messages with priority numerically less than 4: KERN_EMERG (0) through KERN_ERR (3). Warnings (4) and below are not newly printed to the console.
2 default_message_loglevel 4 Default priority for kernel messages that omit an explicit log-level prefix: KERN_WARNING (4).
3 minimum_console_loglevel 1 Lowest console threshold allowed; protects against setting the console level below emergencies.
4 default_console_loglevel 7 Console log level used at boot/default reset: priorities 0–6, up to KERN_INFO.

Linux log priorities, from most to least severe, are:

1
2
3
4
5
6
7
8
0 emerg    System unusable
1 alert    Immediate action required
2 crit     Critical condition
3 err      Error
4 warning  Warning
5 notice   Significant normal event
6 info     Informational
7 debug    Debugging

To change the current console verbosity temporarily:

1
2
3
4
5
6
7
8
9
10
11
# Print errors and more severe messages only
sudo dmesg -n 4

# Include warnings
sudo dmesg -n 5

# Include normal informational kernel messages
sudo dmesg -n 7

# Include debug messages
sudo dmesg -n 8

Equivalently, writing the first value changes only the active console threshold:

1
echo 7 | sudo tee /proc/sys/kernel/printk

Restore the prior console threshold when finished

1
sudo dmesg -n 4

bd_openers

  bd_openers (block device openers)  is a kernel-level counter inside the  struct block_device  structure that tracks how many times a block device has been opened — and it’s exactly the number behind the  Open count  you saw in  dmsetup info .

 it’s a field in the kernel’s struct block_device, the in-memory object representing a block device (a disk, a partition, an LV mapping). It counts how many times that device is currently open — a plain integer refcount.

roles

In the Linux kernel, every block device has a struct block_device containing bd_openers. It’s incremented in blkdev_get_by_*() (via bdev_open_by_*() → blkdev_get_whole()) every time someone opens the device, and decremented in bdev_release() / blkdev_put() when the reference is closed:

1
2
3
open(path, O_RDONLY)          → bd_openers++
mount(), dm table activation  → bd_openers++
close / umount / dm remove    → bd_openers--

What increments it

Opener How
Mounting a filesystem mount opens the bdev and holds it for the entire mount’s life
open("/dev/dm-5") any process — dd, mkfs, blkid, lsblk, a database on raw block
Stacking a device dm/md/loop opening it as a lower layer
swapon holds it until swapoff
Kernel probes udev/blkid briefly, on uevents

This is what dmsetup info surfaces in the Open column

1
2
3
4
Name                     Maj Min Stat Open Targ Event  UUID
csi--lvm-pvc--01dec500-- 253   5 L--w    1    1      0 LVM-...
                                         ^
                                         bd_openers

Why it blocks your removal

dm_lock_for_deletion() refuses to tear down a mapping that something is still using — otherwise the kernel would free structures out from under a live opener and panic. So:

1
bd_openers != 0  →  remove ioctl returns EBUSY

That’s your Device or resource busy, verbatim. And it explains the three failures you hit:

  • --force — swaps the table to error first, then removes. The swap succeeded (hence your Buffer I/O error on dev dm-5 lines); the remove still hit the refcount.
  • --deferred — flags “remove when it reaches zero.” It never reaches zero, so nothing happens.
  • --retry — loops on the same ioctl. Same EBUSY each pass.

None of them decrement the counter. There is no userspace call that does. You can only close the thing holding it open.

so the one opener is a mounted XFS superblock. Normally you’d find its mount in some namespace and unmount it, which releases the reference. But your scans found no mount table entry anywhere and no process holding files on it.

Hence reboot — it’s the only thing that frees kernel memory holding an unreachable reference. Not a workaround; the actual mechanism.

One caveat worth keeping: Open 1 is the kernel’s own accounting and is reliable. But /sys/fs/xfs/dm-5 is keyed by kernel name, not dm UUID — so if a previous device had reused minor 5 and was force-removed while mounted, that kobject could be stale and misattributed. That’d matter if you ever see this error with Open 0. With Open 1 there’s a real opener, so it isn’t your situation.

In short: bd_openers is the kernel’s canonical “is this block device in use” counter; device-mapper consults it before allowing device removal, which is why it directly caused your deactivation failure.

solution

plan 1

bdopener_cli

plan 2

1
reboot

try

1
2
3
4
5
1. host machine
bash release_useless_lv_improved.sh pvc-xxx

2. csi plugin
lvchange -an /dev/csi-lvm/pvc-xxx

nsenter

nsenter  (“namespace enter”) is a util-linux tool that runs a program inside the Linux namespaces of another, already-running process — it’s essentially a command-line wrapper around the  setns(2)  system call.

Linux namespaces partition kernel resources (network, mounts, PIDs, etc.) so processes get isolated views of the system — this is the foundation of containers.  nsenter  lets you “step into” any of those isolated environments from the host, without attaching to or modifying the container.