关于 Kubernetes中Volume的一些笔记（二）

写在前面

学习K8s，刚把这一块学完，整理笔记，理论很少，偏实战，适合温习

博文内容涉及：

常见nfs,hostPath,emptyDir数据卷类型

PV+PVC的创建

持久性存储及动态卷供应

男女情事,谁先动心谁吃亏,越吃亏越难忘,到最后,到底是喜欢对方呢,还是喜欢自己,都搞不清楚了,答案偏偏在对方身上,所以才说,由爱故生忧。--------《剑来》

持久性存储(Persistent Volume)

Volume是定义在Pod上的,属于“计算资源”的一部分,而实际上, “网络存储”是相对独立于“计算资源”而存在的一种实体资源。比如在使用虚拟机的情况下,我们通常会先定义一个网络存储,然后从中划出一个“网盘”并挂接到虚拟机上

Persistent Volume(简称PV)和与之相关联的Persistent Volume Claim (简称PVC)也起到了类似的作用。PV可以理解成

Kubernetes集群中的某个网络存储中对应的一块存储

,它与Volume很类似,但有以下区别。

这里也可以结合物理盘区和逻辑卷来理解，PV可以理解为物理卷，PVC可以理解为划分的逻辑卷。

pv的创建

PV的accessModes属性

, 目前有以下类型:

ReadWriteOnce:读写权限、并且只能被单个Node挂载。

ReadOnlyMany:只读权限、允许被多个Node挂载。

ReadWriteMany:读写权限、允许被多个Node挂载。

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pv No resources found ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$vim pod_volunms-pv.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle #storageClassName: slow mountOptions: - hard - nfsvers=4.1 nfs: path: /tmp server: vms81.liruilongs.github.io

┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$cat /etc/exports /liruilong *(rw,sync,no_root_squash) ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$echo "/tmp *(rw,sync,no_root_squash)" >>/etc/exports ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$cat /etc/exports /liruilong *(rw,sync,no_root_squash) /tmp *(rw,sync,no_root_squash) ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$ ┌──[root@vms81.liruilongs.github.io]-[~/ansible] └─$exportfs -avr exporting *:/tmp exporting *:/liruilong

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f pod_volunms-pv.yaml persistentvolume/pv0003 created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pv -o wide NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE pv0003 5Gi RWO Recycle Available 16s Filesystem

PVC的创建

如果某个Pod想申请某种类型的PV,则首先需要定义一个PersistentVolumeClaim (PVC)对象:

PVC是基于命名空间相互隔离的，不同命名空间的PVC相互隔离PVC通过accessModes和storage的约束关系来匹配PV，不需要显示定义，accessModes必须相同，storage必须小于等于。

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pvc No resources found in liruilong-volume-create namespace. ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$vim pod_volumes-pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mypvc01 spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 4Gi #storageClassName: slow

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f pod_volumes-pvc.yaml persistentvolumeclaim/mypvc01 created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pvc -o wide NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE mypvc01 Bound pv0003 5Gi RWO 10s Filesystem ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$

storageClassName 用于控制那个PVC能和PV绑定，只有在storageClassName相同的情况下才去匹配storage和accessModes

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$vim pod_volunms-pv.yaml

pod_volunms-pv.yaml

apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: slow mountOptions: - hard - nfsvers=4.1 nfs: path: /tmp server: vms81.liruilongs.github.io

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f pod_volunms-pv.yaml persistentvolume/pv0003 created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pv -A NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv0003 5Gi RWO Recycle Available slow 8s

pod_volumes-pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mypvc01 spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 4Gi storageClassName: slow

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pvc -A No resources found ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f pod_volumes-pvc.yaml persistentvolumeclaim/mypvc01 created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE liruilong-volume-create mypvc01 Bound pv0003 5Gi RWO slow 5s

使用持久性存储

在pod里面使用PVC

apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: podvolumepvc name: podvolumepvc spec: volumes: - name: volumes1 persistentVolumeClaim: claimName: mypvc01 containers: - image: nginx name: podvolumehostpath resources: {} volumeMounts: - mountPath: /liruilong name: volumes1 imagePullPolicy: IfNotPresent dnsPolicy: ClusterFirst restartPolicy: Always status: {}

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f pod_volumespvc.yaml pod/podvolumepvc created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pods -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES podvolumepvc 1/1 Running 0 15s 10.244.171.184 vms82.liruilongs.github.io ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl exec -it podvolumepvc -- sh # ls bin dev docker-entrypoint.sh home lib64 media opt root sbin sys usr boot docker-entrypoint.d etc lib liruilong mnt proc run srv tmp var # cd liruilong # ls runc-process838092734 systemd-private-66344110bb03430193d445f816f4f4c4-chronyd.service-SzL7id systemd-private-6cf1f72056ed4482a65bf89ec2a130a9-chronyd.service-5m7c2i systemd-private-b1dc4ffda1d74bb3bec5ab11e5832635-chronyd.service-cPC3Bv systemd-private-bb19f3d6802e46ab8dcb5b88a38b41b8-chronyd.service-cjnt04 #

pv回收策略

persistentVolumeReclaimPolicy: Recycle

会生成一个pod回收数据

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv0003 5Gi RWO Recycle Bound liruilong-volume-create/mypvc01 slow 131m ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl describe pv pv0003 .................. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RecyclerPod 53s persistentvolume-controller Recycler pod: Successfully assigned default/recycler-for-pv0003 to vms82.liruilongs.github.io Normal RecyclerPod 51s persistentvolume-controller Recycler pod: Pulling image "busybox:1.27" ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv0003 5Gi RWO Recycle Available slow 136m ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$

动态卷供应storageClass

通过storageClass来动态处理PV的创建，管理员只需要创建好storageClass就可以了，用户创建PVC时会自动的创建PV和PVC。当创建 pvc 的时候，系统会通知 storageClass，storageClass 会从它所关联的分配器来获取后端存储类型，然后动态的创建一个 pv 出来和此 pvc 进行关联

定义 storageClass 时必须要包含一个分配器(provisioner)，不同的分配器指定了动态创建 pv时使用什么后端存储。

分配器使用 aws 的 ebs 作为 pv 的后端存储

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: slow provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "10" fsType: ext4

分配器使用 lvm 作为 pv 的后端存储

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-lvm provisioner: lvmplugin.csi.alibabacloud.com parameters: vgName: volumegroup1 fsType: ext4 reclaimPolicy: Delete

使用 hostPath 作为 pv 的后端存储

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-hostpath-sc provisioner: hostpath.csi.k8s.io reclaimPolicy: Delete #volumeBindingMode: Immediate volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

上面 3 个例子里所使用的分配器中，有一些是 kubernetes 内置的分配器，比如kubernetes.io/aws-ebs，其他两个分配器不是 kubernetes 自带的。kubernetes 自带的分配器:

kubernetes.io/aws-ebs

kubernetes.io/gce-pd

kubernetes.io/glusterfs

kubernetes.io/cinder

kubernetes.io/vsphere-volume

kubernetes.io/rbd

kubernetes.io/quobyte

kubernetes.io/azure-disk

kubernetes.io/azure-file

kubernetes.io/portworx-volume

kubernetes.io/scaleio

kubernetes.io/storageos

kubernetes.io/no-provisioner

在动态创建 pv 的时候，根据使用不同的后端存储，应该选择一个合适的分配器。但是像lvmplugin.csi.alibabacloud.com 和 hostpath.csi.k8s.io 这样的分配器不是 kubernetes 自带的，称之为外部分配器，这些外部分配器由第三方提供，是通过自定义

CSIDriver(容器存储接口驱动)来实现的分配器

。

所以整个流程就是，管理员创建storageClass时会通过provisioner 字段指定分配器。创建好storageClass之后，用户在定义pvc时需要通过.spec.storageClassName指定使用哪个storageClass。

利用 nfs 创建动态卷供应

创建一个目录/vdisk，并共享这个目录。

┌──[root@vms81.liruilongs.github.io]-[~] └─$cat /etc/exports /liruilong *(rw,sync,no_root_squash) /tmp *(rw,sync,no_root_squash) ┌──[root@vms81.liruilongs.github.io]-[~] └─$echo "/vdisk *(rw,sync,no_root_squash)" >>/etc/exports ┌──[root@vms81.liruilongs.github.io]-[~] └─$exportfs -avr exporting *:/vdisk exportfs: Failed to stat /vdisk: No such file or directory exporting *:/tmp exporting *:/liruilong ┌──[root@vms81.liruilongs.github.io]-[/] └─$mkdir vdisks

因为 kubernetes 里，nfs 没有内置分配器，所以需要下载相关插件来创建 nfs 外部分配器。

插件包下载地址：

https://github.com/kubernetes-incubator/external-storage.git

rbac.yaml 部署 rbac 权限。命名空间更换

apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: liruilong-volume-create --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: liruilong-volume-create roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: liruilong-volume-create rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: liruilong-volume-create subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: liruilong-volume-create roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io

因为 nfs 分配器不是自带的，所以这里需要先把 nfs 分配器创建出来。

配置文件参数设置，1.20之后的版本都需要：- --feature-gates=RemoveSelfLink=false

┌──[root@vms81.liruilongs.github.io]-[/etc/kubernetes/manifests] └─$pwd /etc/kubernetes/manifests ┌──[root@vms81.liruilongs.github.io]-[/etc/kubernetes/manifests] └─$head -n 20 kube-apiserver.yaml apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.26.81:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=192.168.26.81 - --feature-gates=RemoveSelfLink=false - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt ┌──[root@vms81.liruilongs.github.io]-[/etc/kubernetes/manifests] └─$

deployment.yaml

因为当前是在命名空间 liruilong-volume-create里的，所以要把 namespace 的值改为 liruilong-volume-create

image 后面的镜像需要提前在所有节点上 pull下来，并修改镜像下载策略

env字段里，PROVISIONER_NAME用于指定分配器的名字，这里是 fuseim.pri/ifs，NFS_SERVER 和 NFS_PATH分别指定这个分配器所使用的存储信息。

在 volumes 里的 server 和 path 里指定共享服务器和目录

apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: liruilong-volume-create spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.26.81 - name: NFS_PATH value: /vdisk volumes: - name: nfs-client-root nfs: server: 192.168.26.81 path: /vdisk

部署 nfs 分配器,查看 pod 的运行情况

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$kubectl apply -f deployment.yaml deployment.apps/nfs-client-provisioner created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-65b5569d76-cz6hh 1/1 Running 0 73s ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$

创建了 nfs 分配器之后，下面开始创建一个使用这个分配器的 storageClass。

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$kubectl get sc No resources found ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$kubectl apply -f class.yaml storageclass.storage.k8s.io/managed-nfs-storage created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE managed-nfs-storage fuseim.pri/ifs Delete Immediate false 3s

class.yaml

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false"

这里 provisioner 的值 fuseim.pri/ifs 是由 deployment.yaml文件里指定的分配器的名字，这

个 yaml 文件的意思是创建一个名字是managed-nfs-storage的 storageClass，使用名字为fuseim.pri/ifs的分配器。

下面开始创建 pvc

pvc_nfs.yaml

kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvc-nfs spec: accessModes: - ReadWriteMany resources: requests: storage: 20Mi storageClassName: "managed-nfs-storage"

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f ./pvc_nfs.yaml persistentvolumeclaim/pvc-nfs created

查看创建信息

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-65b5569d76-7k6gm 1/1 Running 0 35s ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE managed-nfs-storage fuseim.pri/ifs Delete Immediate false 30s ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-nfs Bound pvc-b12e988a-8b55-4d48-87cf-998500df16f8 20Mi RWX managed-nfs-storage 28s ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-b12e988a-8b55-4d48-87cf-998500df16f8 20Mi RWX Delete Bound liruilong-volume-create/pvc-nfs managed-nfs-storage 126m ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create/nfsdy] └─$

使用声明的PVC

pod_storageclass.yaml

apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: podvolumepvc name: podvolumepvc spec: volumes: - name: volumes1 persistentVolumeClaim: claimName: pvc-nfs containers: - image: nginx name: podvolumehostpath resources: {} volumeMounts: - mountPath: /liruilong name: volumes1 imagePullPolicy: IfNotPresent dnsPolicy: ClusterFirst restartPolicy: Always status: {}

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl apply -f pod_storageclass.yaml pod/podvolumepvc created ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl get pods NAME READY STATUS RESTARTS AGE nfs-client-provisioner-65b5569d76-7k6gm 1/1 Running 0 140m podvolumepvc 1/1 Running 0 7s ┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-volume-create] └─$kubectl describe pods podvolumepvc | grep -A 4 Volumes: Volumes: volumes1: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc-nfs ReadOnly: false

其他的数据卷类型

gcePersistentDisk

使用这种类型的Volume表示使用谷歌公有云提供的永久磁盘(PersistentDisk, PD)存放Volume的数据,它与emptyDir不同, PD上的内容会被永久存,当Pod被删除时, PD只是被卸载(Unmount),但不会被删除。需要注意是,你需要先创建一个永久磁盘(PD),才能使用gcePersistentDisk.

awsElasticBlockStore

与GCE类似,该类型的Volume使用亚马逊公有云提供的EBS Volume存储数据,需要先创建一个EBS Volume才能使用awsElasticBlockStore.

Kubernetes

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。