K8S存储系列之三：K8S Rook Ceph StorageClass

https://rook.io/

什么是Rook

Rook是CNCF已经毕业的云原生存储方案

前置条件

Rook Ceph将每个k8s节点上的块存储联合成一体作为分布式存储，为k8s提供StorageClass，这样的k8s节点由于同时提供了计算、网络、存储，因此也被称作为超融合节点。
为了使用Rook Ceph，k8s集群必须满足的条件有：

• 至少三个node
• 节点提供块存储，满足下列条件之一
  • Raw devices (no partitions or formatted filesystems)
  • Raw partitions (no formatted filesystem)
  • PVs available from a storage class in block mode

我们选择第一个方式提供Raw devices，为每个节点添加一块盘sdb，不对该盘做任何分区操作，也不格式化文件系统：

lsblk -f
NAME                      FSTYPE      LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINT                                                      0   100% /snap/snapd/15904
sda
├─sda1
├─sda2                    ext4              00c05ca1-7dd5-41d1-8ae6-c7deb56cde25    800.9M    11% /boot
└─sda3                    LVM2_member       1NgIKw-CZDb-6t8t-cIMZ-iHYa-jACV-MQIYnR
  └─ubuntu--vg-ubuntu--lv ext4              6686cd41-a6f4-45a7-a564-5581507cd252     10.7G    34% /
sdb

用lsblk看sdb没有FSTYPE

部署Rook Operator

下载资源文件，并部署

git clone --single-branch --branch v1.9.4 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
kubectl get pods -n rook-ceph
rook-ceph-operator-799cd7d684-fzwxb                               1/1     Running     0              18h

部署Rook Ceph Cluster

编辑cluster.yaml，修改spec.storage.deviceFilter，改为sdb。deviceFilter还可以写正则表达式，详见：https://rook.io/docs/rook/v1.9/CRDs/ceph-cluster-crd/?h=devicefilter#storage-selection-settings

……  
  storage: ## cluster level storage configuration and selection
    useAllNodes: true
    useAllDevices: true
    deviceFilter: sdb

接下来部署cluster

cd rook/deploy/examples
kubectl apply -f cluster.yaml

查看pod是否运行。注意下面很多镜像都是从k8s.gcr.io下载的，需要翻墙

kubectl get pods -n rook-ceph
NAME                                                              READY   STATUS      RESTARTS         AGE
csi-cephfsplugin-jhtt7                                            3/3     Running     3 (21h ago)      21h
csi-cephfsplugin-p4chb                                            3/3     Running     4 (26h ago)      4d20h
csi-cephfsplugin-provisioner-7577bb4d59-dmv7m                     6/6     Running     9 (63m ago)      19h
csi-cephfsplugin-provisioner-7577bb4d59-gr8qk                     6/6     Running     13 (4h10m ago)   19h
csi-cephfsplugin-txhqp                                            3/3     Running     2 (25h ago)      4d20h
csi-rbdplugin-5z5nd                                               3/3     Running     3 (21h ago)      21h
csi-rbdplugin-cqnr7                                               3/3     Running     4 (25h ago)      4d20h
csi-rbdplugin-provisioner-847b498845-bxswz                        6/6     Running     9 (63m ago)      19h
csi-rbdplugin-provisioner-847b498845-vfplt                        6/6     Running     14 (4h10m ago)   19h
csi-rbdplugin-s6nlj                                               3/3     Running     2 (26h ago)      4d20h
rook-ceph-crashcollector-k8s-node1.lab.example.com-8b77cffwnjgt   1/1     Running     0                4h11m
rook-ceph-crashcollector-k8s-node2.lab.example.com-9d755bdddmcc   1/1     Running     0                19h
rook-ceph-crashcollector-k8s-node3.lab.example.com-6dbb64b65s5t   1/1     Running     0                4h13m
rook-ceph-mgr-a-78cd6b9ffb-j8r9w                                  2/2     Running     4 (63m ago)      4h13m
rook-ceph-mgr-b-79476687d8-9hfft                                  2/2     Running     43 (26m ago)     19h
rook-ceph-mon-a-c7c674d76-znxzx                                   1/1     Running     48 (26m ago)     19h
rook-ceph-mon-b-5df5487d9f-z7rnb                                  1/1     Running     4 (63m ago)      4h15m
rook-ceph-mon-c-6d8968ff95-h2vqp                                  1/1     Running     8 (30m ago)      19h
rook-ceph-operator-799cd7d684-fzwxb                               1/1     Running     0                19h
rook-ceph-osd-0-5dc6988c7b-l69vx                                  1/1     Running     8 (30m ago)      19h
rook-ceph-osd-1-5777f4dc9d-cl48b                                  1/1     Running     43 (26m ago)     19h
rook-ceph-osd-2-75688c47d-nfbqx                                   1/1     Running     5 (30m ago)      4h15m
rook-ceph-osd-prepare-k8s-node1.lab.example.com-5j4b2             0/1     Completed   0                166m
rook-ceph-osd-prepare-k8s-node2.lab.example.com-sz5b8             0/1     Completed   0                166m
rook-ceph-osd-prepare-k8s-node3.lab.example.com-645cq             0/1     Completed   0                166

If the rook-ceph-mon, rook-ceph-mgr, or rook-ceph-osd pods are not created, please refer to the Ceph common issues for more details and potential solutions.

再部署一个toolbox（里面有ceph命令）用于查看ceph状态：

kubectl apply -f toolbox.yaml
kubectl exec -it rook-ceph-tools-55587cd5f9-mvnhw -n rook-ceph -- bash
[rook@rook-ceph-tools-55587cd5f9-mvnhw /]$ ceph status

If the cluster is not healthy, please refer to the Ceph common issues for more details and potential solutions.

Ceph还提供了一个dashboard：

kubectl expose deployment rook-ceph-mgr-a -n rook-ceph --type=NodePort --name ceph-dashboard
kubectl get svc -n rook-ceph
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
ceph-dashboard             NodePort    10.98.127.28     <none>        6800:30682/TCP,9283:31582/TCP,8443:32441/TCP   21h

kubectl exec -it rook-ceph-tools-55587cd5f9-dp9z2 -n rook-ceph -- bash
[rook@rook-ceph-tools-55587cd5f9-dp9z2 /]$ ceph status
  cluster:
    id:     43a7f46e-58d0-4b09-9c69-d150b716d499
    health: HEALTH_WARN
            3 daemons have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 31m)
    mgr: a(active, since 36m), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 31m), 3 in (since 20h)

  data:
    volumes: 1/1 healthy
    pools:   3 pools, 65 pgs
    objects: 27 objects, 429 KiB
    usage:   86 MiB used, 30 GiB / 30 GiB avail
    pgs:     65 active+clean

  io:
    client:   851 B/s rd, 1 op/s rd, 0 op/s wr

访问https://节点IP:32441
至此Ceph Cluster部署完毕
Ceph可提供三种存储方式：

• Block: Create block storage to be consumed by a pod (RWO)
• Shared Filesystem: Create a filesystem to be shared across multiple pods (RWX)
• Object: Create an object store that is accessible inside or outside the Kubernetes cluster

我们使用shared filesystem提供RWX模式

创建Filesystem

filesystem.yaml

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: replicated
      replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true

kubectl apply -f filesystem.yaml
kubectl get pods -n rook-ceph -l app=rook-ceph-mds
NAME                                    READY   STATUS    RESTARTS   AGE
rook-ceph-mds-myfs-a-9f6f85968-ljr8f    1/1     Running   0          80m
rook-ceph-mds-myfs-b-595fbc5858-jm5wz   1/1     Running   0          79m

提供StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
## Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  ## clusterID is the namespace where the rook cluster is running
  ## If you change this namespace, also change the namespace below where the secret namespaces are defined
  clusterID: rook-ceph

  ## CephFS filesystem name into which the volume shall be created
  fsName: myfs

  ## Ceph pool into which the volume shall be created
  ## Required for provisionVolume: "true"
  pool: myfs-replicated

  ## The secrets contain Ceph admin credentials. These are generated automatically by the operator
  ## in the same namespace as the cluster.
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

reclaimPolicy: Delete

provisioner名称：rook-ceph.cephfs.csi.ceph.com，其中rook-ceph是cluster运行所在namespace

创建PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-fox-dev
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: rook-cephfs

kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cephfs-fox-dev   Bound    pvc-451e4fc3-bda4-445a-b2a6-d5dd40a845d3   5Gi        RWX            rook-cephfs    22h