Kubernetes 集群分布式存储插件 Rook Ceph部署

前言

Rook 介绍

简介

2、Rook 架构

前言

Rook 介绍

简介

2、Rook 架构

Rook 部署

前期规划

准备工作

获取 YAML

部署 Rook Operator

部署 cluster

部署 Toolbox

测试 Rook

设置 dashboard

部署 Node SVC

确认验证

Ceph 块存储应用

创建StorageClass

创建PVC

消费块设备

测试持久性

遇到问题

dashboard 点击概述500 Internal Server Error

前言

我们经常会说：容器和 Pod 是短暂的。其含义是它们的生命周期可能很短，会被频繁地销毁和创建。容器销毁时，保存在容器内部文件系统中的数据都会被清除。 为了持久化保存容器的数据，可以使用存储插件在容器里挂载一个基于网络或者其他机制的远程数据卷，使得在容器里创建的文件，实际上是保存在远程存储服务器上，或者以分布式的方式保存在多个节点上，而与当前宿主机没有绑定关系。这样，无论在哪个节点上启动新的容器，都可以请求挂载指定的持久化存储卷。

由于 Kubernetes 本身的松耦合设计，绝大多数存储项目，比如 Ceph、GlusterFS、NFS 等，都可以为 Kubernetes 提供持久化存储能力。在这次的部署实践中，选择一个很重要生产级的存储插件项目：Rook。

Rook 介绍

简介

Rook 项目是一个基于 Ceph 的 Kubernetes 存储插件（后期也在加入对更多存储的支持）。不过，不同于对 Ceph 的简单封装，Rook 在自己的实现中加入了水平扩展、迁移、灾难备份、监控等大量的企业级功能，使得这个项目变成了一个高度可扩展的分布式存储解决方案，提供对象、文件和块存储。

Rook 目前支持 Ceph、NFS、Minio Object Store、Edegefs、Cassandra、CockroachDB 存储的搭建。

Rook 机制：

Rook 提供了卷插件，来扩展了 K8S 的存储系统，使用 Kubelet 代理程序 Pod 可以挂载 Rook 管理的块设备和文件系统。

Rook Operator 负责启动并监控整个底层存储系统，例如 Ceph Pod、Ceph OSD 等，同时它还管理 CRD、对象存储、文件系统。

Rook Agent 代理部署在 K8S 每个节点上以 Pod 容器运行，每个代理 Pod 都配置一个 Flexvolume 驱动，该驱动主要用来跟 K8S 的卷控制框架集成起来，每个节点上的相关的操作，例如添加存储设备、挂载、格式化、删除存储等操作，都有该代理来完成。

更多参考如下官网：

https://rook.io

https://ceph.com/

2、Rook 架构

Rook 部署

前期规划

准备工作

为了配置 Ceph 存储集群，至少需要以下本地存储选项之一：

原始设备（无分区或格式化的文件系统）

原始分区（无格式文件系统）

可通过 block 模式从存储类别获得 PV

可以使用以下命令确认分区或设备是格式化的文件系统：

$ lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT vda ├─vda1 xfs e16ad84e-8cef-4cb1-b19b-9105c57f97b1 /boot ├─vda2 LVM2_member Vg3nyB-iW9Q-4xp0-LEIO-gzHc-2eax-D1razB │ └─centos-root xfs 0bb4bfa4-b315-43ca-a789-2b43e726c10c / ├─vda3 LVM2_member VZMibm-DJ8e-apig-YhR3-a1dF-wHYQ-8pjKan │ └─centos-root xfs 0bb4bfa4-b315-43ca-a789-2b43e726c10c / └─vda4

如果该 FSTYPE 字段不为空，则在相应设备的顶部有一个文件系统。在这种情况下，可以将 vda4 用于 Ceph。

获取 YAML

git clone --single-branch --branch master https://github.com/rook/rook.git

部署 Rook Operator

本实验使用k8s-node1、k8s-node2、k8s-node3 三个节点，因此需要如下修改：

kubectl label nodes {k8s-node1,k8s-node2,k8s-node3} ceph-osd=enabled kubectl label nodes {k8s-node1,k8s-node2,k8s-node3} ceph-mon=enabled kubectl label nodes k8s-node1 ceph-mgr=enabled

注意：当前版 本 rook 中 mgr 只能支持一个节点运行。

执行脚本:

cd rook/cluster/examples/kubernetes/ceph kubectl create -f common.yaml kubectl create -f operator.yaml

注意：如上创建了相应的基础服务（如 serviceaccounts），同时 rook-ceph-operator 会在每个节点创建 rook-ceph-agent 和 rook-discover。

部署 cluster

配置cluster.yaml

vi cluster.yaml

修改完如下：

################################################################################################################# # Define the settings for the rook-ceph cluster with common settings for a production cluster. # All nodes with available raw devices will be used for the Ceph cluster. At least three nodes are required # in this example. See the documentation for more details on storage settings available. # For example, to create the cluster: # kubectl create -f common.yaml # kubectl create -f operator.yaml # kubectl create -f cluster.yaml ################################################################################################################# apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: # The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw). # v13 is mimic, v14 is nautilus, and v15 is octopus. # RECOMMENDATION: In production, use a specific version tag instead of the general v14 flag, which pulls the latest release and could result in different # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/. # If you want to be more precise, you can always use a timestamp tag such ceph/ceph:v14.2.5-20190917 # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities image: ceph/ceph:v15.2.3 # Whether to allow unsupported versions of Ceph. Currently mimic and nautilus are supported, with the recommendation to upgrade to nautilus. # Octopus is the version allowed when this is set to true. # Do not set to true in production. allowUnsupported: false # The path on the host where configuration files will be persisted. Must be specified. # Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster. # In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment. dataDirHostPath: /var/lib/rook # Whether or not upgrade should continue even if a check fails # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise # Use at your OWN risk # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/master/ceph-upgrade.html#ceph-version-upgrades skipUpgradeChecks: false # Whether or not continue if PGs are not clean during an upgrade continueUpgradeAfterChecksEvenIfNotHealthy: false # set the amount of mons to be started mon: count: 3 allowMultiplePerNode: false mgr: modules: # Several modules should not need to be included in this list. The "dashboard" and "monitoring" modules # are already enabled by other settings in the cluster CR and the "rook" module is always enabled. - name: pg_autoscaler enabled: true # enable the ceph dashboard for viewing cluster status dashboard: enabled: true # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy) # urlPrefix: /ceph-dashboard # serve the dashboard at the given port. # port: 8443 # serve the dashboard using SSL ssl: true # enable prometheus alerting for cluster monitoring: # requires Prometheus to be pre-installed enabled: false # namespace to deploy prometheusRule in. If empty, namespace of the cluster will be used. # Recommended: # If you have a single rook-ceph cluster, set the rulesNamespace to the same namespace as the cluster or keep it empty. # If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus # deployed) to set rulesNamespace for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions. rulesNamespace: rook-ceph network: # enable host networking #provider: host # EXPERIMENTAL: enable the Multus network provider #provider: multus #selectors: # The selector keys are required to be `public` and `cluster`. # Based on the configuration, the operator will do the following: # 1. if only the `public` selector key is specified both public_network and cluster_network Ceph settings will listen on that interface # 2. if both `public` and `cluster` selector keys are specified the first one will point to 'public_network' flag and the second one to 'cluster_network' # # In order to work, each selector value must match a NetworkAttachmentDefinition object in Multus # #public: public-conf --> NetworkAttachmentDefinition object name in Multus #cluster: cluster-conf --> NetworkAttachmentDefinition object name in Multus # enable the crash collector for ceph daemon crash collection crashCollector: disable: false cleanupPolicy: # cleanup should only be added to the cluster when the cluster is about to be deleted. # After any field of the cleanup policy is set, Rook will stop configuring the cluster as if the cluster is about # to be destroyed in order to prevent these settings from being deployed unintentionally. # To signify that automatic deletion is desired, use the value "yes-really-destroy-data". Only this and an empty # string are valid values for this field. confirmation: "" # To control where various services will be scheduled by kubernetes, use the placement configuration sections below. # The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and # tolerate taints with a key of 'storage-node'. placement: mon: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: ceph-mon operator: In values: - enabled podAffinity: podAntiAffinity: topologySpreadConstraints: tolerations: - key: ceph-mon operator: Exists osd: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: ceph-osd operator: In values: - enabled podAffinity: podAntiAffinity: topologySpreadConstraints: tolerations: - key: ceph-osd operator: Exists mgr: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: ceph-mgr operator: In values: - enabled podAffinity: podAntiAffinity: topologySpreadConstraints: tolerations: - key: ceph-mgr operator: Exists # cleanup: annotations: # all: # mon: # osd: # cleanup: # If no mgr annotations are set, prometheus scrape annotations will be set by default. # mgr: resources: # The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory # mgr: # limits: # cpu: "500m" # memory: "1024Mi" # requests: # cpu: "500m" # memory: "1024Mi" # The above example requests/limits can also be added to the mon and osd components # mon: # osd: # prepareosd: # crashcollector: # cleanup: # The option to automatically remove OSDs that are out and are safe to destroy. removeOSDsIfOutAndSafeToRemove: false # priorityClassNames: # all: rook-ceph-default-priority-class # mon: rook-ceph-mon-priority-class # osd: rook-ceph-osd-priority-class # mgr: rook-ceph-mgr-priority-class storage: # cluster level storage configuration and selection useAllNodes: false #关闭使用所有Node useAllDevices: false #关闭使用所有设备 deviceFilter: vda4 config: # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore. # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB # journalSizeMB: "1024" # uncomment if the disks are 20 GB or smaller # osdsPerDevice: "1" # this value can be overridden at the node or device level # encryptedDevice: "true" # the default value for this option is "false" # Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named # nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label. nodes: - name: "k8s-node1" #指定存储节点主机 devices: - name: "vda4" #指定磁盘为sdb config: storeType: bluestore - name: "k8s-node2" devices: - name: "vda4" config: storeType: bluestore - name: "k8s-node3" devices: - name: "vda4" config: storeType: bluestore # The section for configuring management of daemon disruptions during upgrade or fencing. disruptionManagement: # If true, the operator will create and manage PodDisruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically # via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph/ceph-managed-disruptionbudgets.md). The operator will # block eviction of OSDs by default and unblock them safely when drains are detected. managePodBudgets: false # A duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the # default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes. osdMaintenanceTimeout: 30 # If true, the operator will create and manage MachineDisruptionBudgets to ensure OSDs are only fenced when the cluster is healthy. # Only available on OpenShift. manageMachineDisruptionBudgets: false # Namespace in which to watch for the MachineDisruptionBudgets. machineDisruptionBudgetNamespace: openshift-machine-api

更多 cluster 的 CRD 配置参考：

https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md

https://blog.gmem.cc/rook-based-k8s-storage-solution

执行cluster.yaml

kubectl create -f cluster.yaml # 查看部署 log $ kubectl logs -f -n rook-ceph rook-ceph-operator-567d7945d6-t9rd4 # 等待一定时间，部分中间态容器可能会波动 [7d@k8s-master ceph]$ kubectl get pods -n rook-ceph -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES csi-cephfsplugin-dr4dq 3/3 Running 0 24h 172.16.106.239 k8s-node2 csi-cephfsplugin-provisioner-6bcb7cdd75-dtzmn 5/5 .......

注意：若部署失败

master 节点执行

kubectl delete -f ./

所有 node 节点执行如下清理操作：

rm -rf /var/lib/rook /dev/mapper/ceph-* dmsetup ls dmsetup remove_all dd if=/dev/zero of=/dev/vda4 bs=512k count=1 wipefs -af /dev/vda4

部署 Toolbox

Toolbox 是一个 Rook 的工具集容器，该容器中的命令可以用来调试、测试 Rook，对 Ceph 临时测试的操作一般在这个容器内执行。

# 启动 rook-ceph-tools pod: $ kubectl create -f toolbox.yaml deployment.apps/rook-ceph-tools created # 等待 rook-ceph-tools 载其容器并进入 running 状态： $ kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" NAME READY STATUS RESTARTS AGE rook-ceph-tools-6d659f5579-knt6x 1/1 Running 0 7s

测试 Rook

$ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. # 查看 Ceph 状态 [root@rook-ceph-tools-6d659f5579-knt6x /]# ceph status cluster: id: 550e2978-26a6-4f3b-b101-10369ab63cf4 health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 2m) mgr: a(active, since 85s) osd: 3 osds: 3 up (since 114s), 3 in (since 114s) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 3.0 GiB used, 147 GiB / 150 GiB avail pgs: 1 active+clean [root@rook-ceph-tools-6d659f5579-knt6x /]# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 k8s-node2 1026M 48.9G 0 0 0 0 exists,up 1 k8s-node1 1026M 48.9G 0 0 0 0 exists,up 2 k8s-node3 1026M 48.9G 0 0 0 0 exists,up [root@rook-ceph-tools-6d659f5579-knt6x /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 150 GiB 147 GiB 6.2 MiB 3.0 GiB 2.00 TOTAL 150 GiB 147 GiB 6.2 MiB 3.0 GiB 2.00 --- POOLS --- POOL ID STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 0 B 0 0 B 0 46 GiB [root@rook-ceph-tools-6d659f5579-knt6x /]# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR device_health_metrics 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B total_objects 0 total_used 3.0 GiB total_avail 147 GiB total_space 150 GiB # 查看 Ceph 所有 keyring [root@rook-ceph-tools-6d659f5579-knt6x /]# ceph auth ls installed auth entries: osd.0 key: AQAjofFe1j9pGhAABnjTXAYZeZdwo2FGHIFv+g== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQAjofFeY0LaHhAAMVLxrH1lqXqyYsZE9yJ5dg== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQAkofFeBjtoDBAAVYW7FursqpbttekW54u2rA== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQDhoPFeqLE4ORAAvBeGwV7p1YY25owP8nS02Q== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQABofFeQclcCxAAEtA9Y4+yF3I6H9RM0/f1DQ== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQABofFeaeBcCxAA9SEnt+RV7neC4uy/xQb5qg== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQABofFe0/NcCxAAqCKwJpzPlav8MuajRk8xmw== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQABofFesAZdCxAAZyWJg+Pa3F0g5Toy4LamPw== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rbd-mirror key: AQABofFejRpdCxAA/9NbTQDJILdSoYJZdol7bQ== caps: [mon] allow profile bootstrap-rbd-mirror client.bootstrap-rgw key: AQABofFeAi5dCxAAKu67ZyM8PRRPcluTXR3YRw== caps: [mon] allow profile bootstrap-rgw client.crash key: AQAcofFeLDr3KBAAw9UowFd26JiQSGjCFyhx8w== caps: [mgr] allow profile crash caps: [mon] allow profile crash client.csi-cephfs-node key: AQAcofFeFRh4DBAA7Z8kgcHGM92vHj6cvGbXXg== caps: [mds] allow rw caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rw tag cephfs *=* client.csi-cephfs-provisioner key: AQAbofFemLuJMRAA4WlGWBjONb1av48rox1q6g== caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rw tag cephfs metadata=* client.csi-rbd-node key: AQAbofFepu7rFhAA+vdit2ipDgVFc/yKUpHHug== caps: [mon] profile rbd caps: [osd] profile rbd client.csi-rbd-provisioner key: AQAaofFe3Yw9OxAAiJzZ6HQne/e9Zob5G311OA== caps: [mgr] allow rw caps: [mon] profile rbd caps: [osd] profile rbd mgr.a key: AQAdofFeh4VZHhAA8VL9gH5jgOxzjTDtEaFWBQ== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * [root@rook-ceph-tools-6d659f5579-knt6x /]# ceph version ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) [root@rook-ceph-tools-6d659f5579-knt6x /]# exit exit

这样，一个基于 Rook 持久化存储集群就以容器的方式运行起来了，而接下来在 Kubernetes 项目上创建的所有 Pod 就能够通过 Persistent Volume（PV）和 Persistent Volume Claim（PVC）的方式，在容器里挂载由 Ceph 提供的数据卷了。而 Rook 项目，则会负责这些数据卷的生命周期管理、灾难备份等运维工作。

设置 dashboard

dashboard 是非常有用的工具，可让你大致了解 Ceph 集群的状态，包括总体运行状况，单仲裁状态，mgr，osd 和其他 Ceph 守护程序的状态，查看池和 PG 状态，显示日志用于守护程序等等。Rook 使启用仪表板变得简单。

部署 Node SVC

修改dashboard-external-https.yaml

$ vi dashboard-external-https.yaml apiVersion: v1 kind: Service metadata: name: rook-ceph-mgr-dashboard-external-https namespace: rook-ceph labels: app: rook-ceph-mgr rook_cluster: rook-ceph spec: ports: - name: dashboard port: 8443 protocol: TCP targetPort: 8443 selector: app: rook-ceph-mgr rook_cluster: rook-ceph sessionAffinity: None type: NodePort

创建 Node SVC

$ kubectl create -f dashboard-external-https.yaml service/rook-ceph-mgr-dashboard-external-https created $ kubectl get svc -n rook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 10.102.32.77 8080/TCP,8081/TCP 17m csi-rbdplugin-metrics ClusterIP 10.101.121.5 8080/TCP,8081/TCP 17m rook-ceph-mgr ClusterIP 10.99.155.138 9283/TCP 16m rook-ceph-mgr-dashboard ClusterIP 10.97.61.135 8443/TCP 16m rook-ceph-mgr-dashboard-external-https NodePort 10.108.210.25 8443:32364/TCP 6s rook-ceph-mon-a ClusterIP 10.102.116.81 6789/TCP,3300/TCP 16m rook-ceph-mon-b ClusterIP 10.101.141.241 6789/TCP,3300/TCP 16m rook-ceph-mon-c ClusterIP 10.101.157.247 6789/TCP,3300/TCP 16m

Rook operator 将启用 ceph-mgr dashboard 模块。将创建一个服务对象以在 Kubernetes 集群中公开该端口。Rook 将启用端口 8443 进行 https 访问。

确认验证

登录 dashboard 需要安全访问。Rook 在运行 Rook Ceph 集群的名称空间中创建一个默认用户，admin 并生成一个称为的秘密rook-ceph-dashboard-admin-password。

要检索生成的密码，可以运行以下命令：

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

Ceph 块存储应用

创建StorageClass

在提供（Provisioning）块存储之前，需要先创建StorageClass 和存储池。K8S 需要这两类资源，才能和Rook 交互，进而分配持久卷（PV）。

cd rook/cluster/examples/kubernetes/ceph/csi/rbd kubectl create -f csi/rbd/storageclass.yaml

解读：如下配置文件中会创建一个名为 replicapool 的存储池，和rook-ceph-block的 storageClass。

apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicapool namespace: rook-ceph spec: failureDomain: host replicated: size: 3 # Disallow setting pool with replica 1, this could lead to data loss without recovery. # Make sure you're *ABSOLUTELY CERTAIN* that is what you want requireSafeReplicaSize: true # gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool # for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size #targetSizeRatio: .5 --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-block provisioner: rook-ceph.rbd.csi.ceph.com parameters: # clusterID is the namespace where the rook cluster is running # If you change this namespace, also change the namespace below where the secret namespaces are defined clusterID: rook-ceph # If you want to use erasure coded pool with RBD, you need to create # two pools. one erasure coded and one replicated. # You need to specify the replicated pool here in the `pool` parameter, it is # used for the metadata of the images. # The erasure coded pool must be set as the `dataPool` parameter below. #dataPool: ec-data-pool pool: replicapool # RBD image format. Defaults to "2". imageFormat: "2" # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature. imageFeatures: layering # The secrets contain Ceph admin credentials. These are generated automatically by the operator # in the same namespace as the cluster. csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # Specify the filesystem type of the volume. If not specified, csi-provisioner # will set default as `ext4`. csi.storage.k8s.io/fstype: ext4 # uncomment the following to use rbd-nbd as mounter on supported nodes # **IMPORTANT**: If you are using rbd-nbd as the mounter, during upgrade you will be hit a ceph-csi # issue that causes the mount to be disconnected. You will need to follow special upgrade steps # to restart your application pods. Therefore, this option is not recommended. #mounter: rbd-nbd allowVolumeExpansion: true reclaimPolicy: Delete

$ kubectl get storageclasses.storage.k8s.io NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 44m

创建PVC

$ kubectl create -f pvc.yaml $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rbd-pvc Bound pvc-b2b7ce1d-7cad-4b7b-afac-dcf6cd597e88 1Gi RWO rook-ceph-block 45m $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-b2b7ce1d-7cad-4b7b-afac-dcf6cd597e88 1Gi RWO Delete Bound default/rbd-pvc rook-ceph-block 45m

解读：如上创建相应的PVC，storageClassName:为基于 rook Ceph 集群的 rook-ceph-block。

pvc.yaml:

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rbd-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: rook-ceph-block

消费块设备

$ kubectl create -f rookpod01.yaml $ kubectl get pods NAME READY STATUS RESTARTS AGE rookpod01 0/1 Completed 0 4m1s

解读：创建如上 Pod，并挂载之前所创建的PVC，等待执行完毕

rookpod01.yaml：

apiVersion: v1 kind: Pod metadata: name: rookpod01 spec: restartPolicy: OnFailure containers: - name: test-container image: busybox volumeMounts: - name: block-pvc mountPath: /var/test command: ['sh', '-c', 'echo "Hello World" > /var/test/data; exit 0'] volumes: - name: block-pvc persistentVolumeClaim: claimName: rbd-pvc readOnly: false

测试持久性

# 删除rookpod01 $ kubectl delete -f rookpod01.yaml pod "rookpod01" deleted # 创建rookpod02 $ kubectl create -f rookpod02.yaml pod/rookpod02 created $ kubectl get pods NAME READY STATUS RESTARTS AGE rookpod02 0/1 Completed 0 59s $ kubectl logs rookpod02 Hello World

解读：创建 rookpod02，并使用所创建的 PVC，测试持久性。

rookpod02.yaml：

apiVersion: v1 kind: Pod metadata: name: rookpod02 spec: restartPolicy: OnFailure containers: - name: test-container image: busybox volumeMounts: - name: block-pvc mountPath: /var/test command: ['sh', '-c', 'cat /var/test/data; exit 0'] volumes: - name: block-pvc persistentVolumeClaim: claimName: rbd-pvc readOnly: false

遇到问题

dashboard 点击概述500 Internal Server Error

解决办法：

创建内置管理员角色的新副本，从该角色中删除 iscsi 权限，然后将此新角色分配给管理员。大概是在上游解决此问题后，可以删除新角色并将管理员角色重新分配给 admin 用户。

ceph dashboard ac-role-create admin-no-iscsi for scope in dashboard-settings log rgw prometheus grafana nfs-ganesha manager hosts rbd-image config-opt rbd-mirroring cephfs user osd pool monitor; do ceph dashboard ac-role-add-scope-perms admin-no-iscsi ${scope} create delete read update; done ceph dashboard ac-user-set-roles admin admin-no-iscsi

[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph dashboard ac-role-create admin-no-iscsi {"name": "admin-no-iscsi", "description": null, "scopes_permissions": {}} [root@rook-ceph-tools-6d659f5579-knt6x /]# for scope in dashboard-settings log rgw prometheus grafana nfs-ganesha manager hosts rbd-image config-opt rbd-mirroring cephfs user osd pool monitor; do > ceph dashboard ac-role-add-scope-perms admin-no-iscsi ${scope} create delete read update; [root@rook-ceph-tools-6d659f5579-knt6x /]# ceph dashboard ac-user-set-roles admin admin-no-iscsi

源码地址：

https://github.com/zuozewei/blog-example/tree/master/Kubernetes/k8s-rook-ceph

Kubernetes 分布式 存储

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。