Kubernetes Pod动态mount nfs方式
背景
为Pod 挂在一个nfs目录, 临时处理数据使用。 确保pod不重启。
验证方式
准备
- nfs-server
[root@xxxxx /var/lib/paascontainer/nfs]# exportfs -s
/var/lib/paascontainer/nfs *(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
[root@xxxxx /var/lib/paascontainer/nfs]# showmount -e 172.16.0.8
Export list for 172.16.0.8:
/var/lib/paascontainer/nfs *
- 准备测试pod
1 apiVersion: apps/v1
2 kind: Deployment
3 metadata:
4 name: nfs-test
5 spec:
6 selector:
7 matchLabels:
8 app: nfs-test
10 template:
11 metadata:
12 labels:
13 app: nfs-test
14 spec:
15 volumes:
16 - name: task-pv-storage
17 hostPath:
18 path: /data/nfs
19 type: Directory
20 containers:
21 - name: nfs
22 command: [ "/bin/bash", "-c", "--" ]
23 args: [ "while true; do sleep 30; done;" ]
24 image: dongjiang1989/nfs-centos-7:main
25 volumeMounts:
26 - name: task-pv-storage
27 mountPath: /mnt/sample
先通过hostPath 挂载 local 下的一个目录
- 检查结果
dongjiang@MacBook Pro:~ $ kubectl get pod|grep nfs
nfs-test-7c75465fd-ghgdh 1/1 Running 0 18m
进行操作
希望将nfs mount到 nfs-test-7c75465fd-ghgdh pod 下的 hostPath下的一个目录中。
首先进入pod
[root@nfs-test-69cb66c555-nnbdw sample]# mount -t nfs 172.16.0.8:/var/lib/paascontainer/nfs /mnt/sample/nfs
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: Operation not permitted
[root@nfs-test-69cb66c555-nnbdw sample]# mount -vvv -t nfs 172.16.0.8:/var/lib/paascontainer/nfs /mnt/sample/nfs -o nolock
mount.nfs: timeout set for Tue Oct 17 02:48:50 2023
mount.nfs: trying text-based options 'nolock,vers=4.1,addr=172.16.0.8,clientaddr=172.19.162.59'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'nolock,addr=172.16.0.8'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.16.0.8 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 172.16.0.8 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Operation not permitted
mount.nfs: Operation not permitted
[root@nfs-test-69cb66c555-nnbdw sample]# rpcinfo -p 172.16.0.8
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 40872 status
100024 1 tcp 37863 status
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100021 1 udp 27331 nlockmgr
100021 3 udp 27331 nlockmgr
100021 4 udp 27331 nlockmgr
100021 1 tcp 11731 nlockmgr
100021 3 tcp 11731 nlockmgr
100021 4 tcp 11731 nlockmgr
调整 nfs 设置fsid=0 权限 和 cap_sys_admin权限
更改 nfs-server 权限
[root@xxxxx /var/lib/paascontainer/nfs]# vi /etc/exports //添加fsid=0
[root@xxxxx /var/lib/paascontainer/nfs]# exportfs -r //reload
[root@xxxxx /var/lib/paascontainer/nfs]# exportfs -s
/var/lib/paascontainer/nfs *(sync,wdelay,hide,no_subtree_check,insecure_locks,fsid=0,sec=sys,rw,insecure,no_root_squash,no_all_squash)
更改pod 权限, 添加 SYS_ADMIN 权限
1 apiVersion: apps/v1
2 kind: Deployment
3 metadata:
4 name: nfs-test
5 spec:
6 selector:
7 matchLabels:
8 app: nfs-test
10 template:
11 metadata:
12 labels:
13 app: nfs-test
14 spec:
15 volumes:
16 - name: task-pv-storage
17 hostPath:
18 path: /data/nfs
19 type: Directory
20 containers:
21 - name: nfs
22 command: [ "/bin/bash", "-c", "--" ]
23 args: [ "while true; do sleep 30; done;" ]
24 securityContext:
25 capabilities:
26 add:
27 - SYS_ADMIN #添加cap_sys_admin权限
28 image: dongjiang1989/nfs-centos-7:main
29 volumeMounts:
30 - name: task-pv-storage
31 mountPath: /mnt/sample
变更：
[root@nfs-test-7c75465fd-ghgdh /]# mount -vvv -t nfs 172.16.0.8:/var/lib/paascontainer/nfs /mnt/sample/nfs -o nolock
mount.nfs: mount point /mnt/sample/nfs does not exist
[root@nfs-test-7c75465fd-ghgdh /]# mkdir /mnt/sample/nfs
[root@nfs-test-7c75465fd-ghgdh /]# mount -vvv -t nfs 172.16.0.8:/var/lib/paascontainer/nfs /mnt/sample/nfs -o nolock
mount.nfs: timeout set for Tue Oct 17 03:02:46 2023
mount.nfs: trying text-based options 'nolock,vers=4.1,addr=172.16.0.8,clientaddr=172.19.141.24'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'nolock,addr=172.16.0.8'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.16.0.8 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 172.16.0.8 prog 100005 vers 3 prot UDP port 20048
验证
pod 中touch文件，并进行写入
[root@nfs-test-7c75465fd-ghgdh sample]# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 500G 68G 433G 14% /
tmpfs 64M 0 64M 0% /dev
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 99G 13G 82G 14% /mnt/sample
/dev/sdb 500G 68G 433G 14% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 7.8G 12K 7.8G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 7.8G 0 7.8G 0% /proc/acpi
tmpfs 7.8G 0 7.8G 0% /proc/scsi
tmpfs 7.8G 0 7.8G 0% /sys/firmware
172.16.0.8:/var/lib/paascontainer/nfs 500G 68G 433G 14% /mnt/sample/nfs
[root@nfs-test-7c75465fd-ghgdh sample]# cd /mnt/sample/nfs/
[root@nfs-test-7c75465fd-ghgdh nfs]# touch aaa
[root@nfs-test-7c75465fd-ghgdh nfs]# echo "aaa" > aaa
[root@nfs-test-7c75465fd-ghgdh nfs]# ls
在K8s node上可读写
[root@node-xxx ~]# mount -vvv -t nfs 172.16.0.8:/var/lib/paascontainer/nfs /data/nfs/nfs -o nolock
mount.nfs: timeout set for Wed Oct 18 09:57:45 2023
mount.nfs: trying text-based options 'nolock,vers=4.2,addr=172.16.0.8,clientaddr=172.16.0.8'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'nolock,addr=172.16.0.8'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.16.0.8 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 172.16.0.8 prog 100005 vers 3 prot UDP port 20048
[root@node-xxx ~]# ls
[root@node-xxx ~]# ll
total 0
[root@node-xxx ~]# cd /data/nfs/nfs
[root@node-xxx /data/nfs/nfs]# ls
aaa
[root@node-xxx /data/nfs/nfs]# ll
total 4
-rw-r--r-- 1 root root 4 Oct 18 09:51 aaa
[root@node-xxx /data/nfs/nfs]# vi aaa
[root@node-xxx /data/nfs/nfs]# touch bbb
[root@node-xxx /data/nfs/nfs]# echo "bbbb" > bbb
[root@node-xxx /data/nfs/nfs]# cat bbb
bbbb
在 pod中可读写:
[root@nfs-test-7c75465fd-ghgdh /]# cd mnt/sample/nfs/
[root@nfs-test-7c75465fd-ghgdh nfs]# ls
aaa bbb
[root@nfs-test-7c75465fd-ghgdh nfs]# ls
aaa bbb
[root@nfs-test-7c75465fd-ghgdh nfs]# ls -l
total 8
-rw-r--r-- 1 root root 4 Oct 18 01:51 aaa
-rw-r--r-- 1 root root 5 Oct 18 01:56 bbb
[root@nfs-test-7c75465fd-ghgdh nfs]# cat bbb
bbbb
在nfs-server node上读文件
[root@xxxxx /var/lib/paascontainer]# pwd
/var/lib/paascontainer
[root@xxxxx /var/lib/paascontainer]# ls
containerd docker etcd kubelet nfs
[root@xxxxx /var/lib/paascontainer]# cd nfs/
[root@xxxxx /var/lib/paascontainer/nfs]# ls
aaa
[root@xxxxx /var/lib/paascontainer/nfs]# cat aaa
aaa
结论
此做法非云原生做法， pod被驱逐等重启场景会丢失：
- 通过nfs pvc进行绑定
- 对远端nfs进行读写场景，可以在server上集成 nfs-client 进行操作
「如果这篇文章对你有用,请随意打赏」
如果这篇文章对你有用,请随意打赏
使用微信扫描二维码完成支付
