lvmlocked sanlock

fs

LVM + lvmlockd (clvm replacement)

source code

1
2
3
4
5
6
7
8
9
10
https://sourceware.org/git/?p=lvm2.git
https://gitlab.com/lvmteam/lvm2
https://github.com/lvmteam/lvm2

https://android.googlesource.com/platform/external/lvm2/+/d44af0be2c6f4652eafd90a70e7ba5f24c0f6d5a/lib/locking/lvmlockd.c

# how lvmlockd  talks to  sanlock , look at the  daemons/lvmlockd/  directory in the source tree.
	•	Core Logic:  daemons/lvmlockd/lvmlockd-core.c 
	•	Sanlock Adapter:  daemons/lvmlockd/lvmlockd-sanlock.c 
	•	This file contains the C functions that translate LVM lock requests ( lock_lv ,  lock_vg ) into  sanlock_acquire and sanlock_release calls.

🚨prepare

  • ❗操作之后需要把之前冗余的 activate 的 lv 都设置成 inactive

    1
    2
    3
    4
    5
    # flite
    lvscan | grep '/csi-lvm/' | grep -v inactive

    # deactivate duplicate pv
    lvchange -an /dev/csi-lvm/pvc-xxx

install

1
2
# sanlock and lvmlocked
apt install lvm2 sanlock lvm2-lockd

lvmlocked

  • enable locked

    1
    2
    # vim /etc/lvm/lvm.conf
    use_lvmlockd = 1

  • check

    1
    lvmconfig --type current global/use_lvmlockd

  • restart lvmlockd

    1
    systemctl restart lvmlockd

sanlock

  • config host id

    1
    2
    3
    4
    5
    # vim /etc/lvm/lvmlocal.conf
    local {
        # Replace <nodeNum> with the unique integer for this node (e.g., 1, 2, 3...)
        host_id = <nodeNum>
    }

  • Restart

    1
    2
    3
    systemctl restart sanlock
    sleep 8
    systemctl restart lvmlockd

first node

Convert VG to sanlock

  • config sanlock

    1
    (optional) lvchange -an csi-lvm

  • skip global lockspace 

    1
    2
    3
    4
    5
    6
    7
    vgchange --lock-type sanlock --lockopt skipgl csi-lvm

    # print
    WARNING: skipping global lock in lvmlockd.
    Logical volume "lvmlock" created.
    device-mapper: remove ioctl on  (253:9) failed: Device or resource busy
    Volume group "csi-lvm" successfully changed

Start the Lockspace

Activate the lockspace for newly converted VG.This allows the local lvmlockd to manage it.

1
2
3
4
5
6
vgchange --lock-start csi-lvm

# print
Skipping global lock: lockspace not found or started
VG csi-lvm starting sanlock lockspace
Starting locking.  Waiting for sanlock may take 20 sec to 3 min...

Enable the Global Lock

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# enable 
lvmlockctl --gl-enable csi-lvm

# verify
lvmlockctl -i

# 正常打印
VG csi-lvm lock_type=sanlock UShLSg-JLRf-Agkb-YRQK-asNV-GSFn-IRqbDH
LS sanlock lvm_csi-lvm
LK VG un ver 0 🚨 初始化之后，本来为0，等创建lv之后（lvcreate -L 10G -n lv-temp-1 csi-lvm）会出现数字
LK GL un ver 0

# Verify the Global Lock
sanlock client status

# verify
vgs -o vg_name,lock_type
or
vgs -o+lock_type,lock_args

other node

Start the Lockspace

1
2
3
4
5
6
vgchange --lock-start csi-lvm

# Verify the Global Lock
sanlock client status

vgs -o vg_name,lock_type

init lock

1
lvcreate -L 10G -n lv-temp-1 csi-lvm

best practices

查看当前节点持有的锁

1
sanlock client status

check lvmlockd status

1
systemctl status lvmlockd