kubernetes生产实践之redis-cluster


方案一 自定义yaml文件安装redis cluster

背景

在Kubernetes中部署Redis集群面临挑战,因为每个Redis实例都依赖于一个配置文件,该文件可以跟踪其他集群实例及其角色。为此,需要结合使用Kubernetes StatefulSets和PersistentVolumes实现。
redis cluster架构示意图:

创建StatefulSet yaml文件

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
data:
  update.sh: |
    #!/bin/sh
    REDIS_NODES="/data/nodes.conf"
    sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
    exec "$@"
  redis.conf: |+
    bind 0.0.0.0
    cluster-enabled yes
    cluster-require-full-coverage no
    cluster-node-timeout 30000
    cluster-config-file /data/nodes.conf
    cluster-migration-barrier 1
    appendonly yes
    protected-mode no
---
apiVersion: apps.kruise.io/v1beta1
# apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:6.2.1-alpine
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/conf/update.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi
      storageClassName: rbd

注释:

cluster-migration-barrier 那些分配后仍然剩余migration barrier个从节点的主节点才会触发节点分配,而不是分配前有migration barrier个从节点的主节点就会触发节点分配,默认是1,生产环境建议维持默认值
protected-mode no 参数是为了禁止外网访问redis,如果启用了,则只能够通过lookback ip(127.0.0.1)访问Redis,如果外网访问redis,会报出异常。
apiVersion: apps.kruise.io/v1beta1 控制器这里使用kruise提供的Advanced StatefulSet,如果集群没有安装kruise,可以使用 apps/v1

安装redis-cluster

[root@qd01-stop-k8s-master001 redis]# kubectl apply -f install-redis.yaml
configmap/redis-cluster created
statefulset.apps.kruise.io/redis-cluster created

[root@qd01-stop-k8s-master001 redis]# kubectl get po -n op
NAME              READY   STATUS    RESTARTS   AGE
redis-cluster-0   1/1     Running   0          3m26s
redis-cluster-1   1/1     Running   0          3m14s
redis-cluster-2   1/1     Running   0          2m54s
redis-cluster-3   1/1     Running   0          2m23s
redis-cluster-4   1/1     Running   0          2m14s
redis-cluster-5   1/1     Running   0          114s

创建redis-cluster service

---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster
  namespace: op
spec:
  type: ClusterIP
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
[root@qd01-stop-k8s-master001 redis]# kubectl apply -f redis-svc.yml
service/redis-cluster created
[root@qd01-stop-k8s-master001 redis]# kubectl get svc -n op
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
redis-cluster   ClusterIP   10.97.197.224   <none>        6379/TCP,16379/TCP   9s

测试能后连通
[root@qd01-stop-k8s-master001 redis]# telnet  10.97.197.224 6379
Trying 10.97.197.224...
Connected to 10.97.197.224.
Escape character is '^]'.

初始化redis-cluster

执行如下命令,获取到pod IP,然后使用redis-cli --cluster创建集群

[root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-0 -- redis-cli --cluster create --cluster-replicas 1 $(kubectl -n op get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 {end}')
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 100.88.43.67:6379 to 100.64.147.152:6379
Adding replica 100.113.170.5:6379 to 100.98.174.217:6379
Adding replica 100.64.147.153:6379 to 100.80.158.227:6379
M: b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379
   slots:[0-5460] (5461 slots) master
M: 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379
   slots:[5461-10922] (5462 slots) master
M: 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379
   slots:[10923-16383] (5461 slots) master
S: b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379
   replicates 5389ace495b68eeac85370d6783648dff68f2fb6
S: 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379
   replicates b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
S: e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379
   replicates 09543217c903350e963fc4fdf4acb73f8a1b7f8b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 100.64.147.152:6379)
M: b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379
   slots: (0 slots) slave
   replicates b47b27a3dbddf3fc1370cbe14ae753f4fce20b04
M: 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379
   slots: (0 slots) slave
   replicates 09543217c903350e963fc4fdf4acb73f8a1b7f8b
S: b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379
   slots: (0 slots) slave
   replicates 5389ace495b68eeac85370d6783648dff68f2fb6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

验证集群信息

[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-0 -- redis-cli cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:178
cluster_stats_messages_pong_sent:181
cluster_stats_messages_sent:359
cluster_stats_messages_ping_received:176
cluster_stats_messages_pong_received:178
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:359

[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-0 -- redis-cli cluster nodes
0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379@16379 slave b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 0 1615348311156 1 connected
09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379@16379 master - 0 1615348314162 2 connected 5461-10922
b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379@16379 myself,master - 0 1615348312000 1 connected 0-5460
5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379@16379 master - 0 1615348312000 3 connected 10923-16383
e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379@16379 slave 09543217c903350e963fc4fdf4acb73f8a1b7f8b 0 1615348313160 2 connected
b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379@16379 slave 5389ace495b68eeac85370d6783648dff68f2fb6 0 1615348312158 3 connected

从输出可以看到,集群总共6个节点,三主三从

方案二 使用kubeDB安装redis

安装kubeDB

1、安装KubeDB
获取AppsCode License https://license-issuer.appscode.com/
下载KubeDB charts https://github.com/appscode/charts/tree/master/stable/kubedb-community

[root@qd01-stop-k8s-master001 kubedb-community]# unzip kubedb-community-v0.16.2.tgz
[root@qd01-stop-k8s-master001 kubedb-community]# cd kubedb-community
[root@qd01-stop-k8s-master001 kubedb-community]# ls -al
total 96
drwxr-xr-x 4 root root   158 Mar 10 15:26 .
drwxr-xr-x 3 root root    66 Mar 10 15:24 ..
-rw-r--r-- 1 root root   351 Feb 16 09:55 Chart.yaml
drwxr-xr-x 2 root root    28 Mar 10 15:24 ci
-rw-r--r-- 1 root root   493 Feb 16 09:55 doc.yaml
-rw-r--r-- 1 root root   353 Feb 16 09:55 .helmignore
-rw-r--r-- 1 root root 24422 Feb 16 09:55 README.md
drwxr-xr-x 2 root root  4096 Mar 10 15:24 templates
-rw-r--r-- 1 root root 47437 Feb 16 09:55 values.openapiv3_schema.yaml
-rw-r--r-- 1 root root  5230 Feb 16 09:55 values.yaml

修改values.yaml,把License文件放到kubedb-community目录下
2、使用helm安装

[root@qd01-stop-k8s-master001 kubedb-community]# helm install kubedb-community --namespace kube-system --set-file license=./kubedb-community-license.txt -f values.yaml  .
NAME: kubedb-community
LAST DEPLOYED: Wed Mar 10 15:38:59 2021
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that KubeDB has started, run:
  kubectl get deployment --namespace kube-system -l "app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
Now install/upgrade appscode/kubedb-catalog chart.
To install, run:
  helm install kubedb-catalog appscode/kubedb-catalog --version v0.16.2 --namespace kube-system
To upgrade, run:
  helm upgrade kubedb-catalog appscode/kubedb-catalog --version v0.16.2 --namespace kube-system

执行如下命令查看是否安装完成
[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get deployment --namespace kube-system -l "app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
kubedb-community   1/1     1            1           38s

等待crds注册成功
[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get crds -l app.kubernetes.io/name=kubedb -w
NAME                                       CREATED AT
elasticsearches.kubedb.com                 2021-03-10T07:39:42Z
elasticsearchversions.catalog.kubedb.com   2021-03-10T07:39:45Z
etcds.kubedb.com                           2021-03-10T07:39:42Z
etcdversions.catalog.kubedb.com            2021-03-10T07:39:45Z
memcacheds.kubedb.com                      2021-03-10T07:39:43Z
memcachedversions.catalog.kubedb.com       2021-03-10T07:39:45Z
mongodbs.kubedb.com                        2021-03-10T07:39:43Z
mongodbversions.catalog.kubedb.com         2021-03-10T07:39:45Z
mysqls.kubedb.com                          2021-03-10T07:39:43Z
mysqlversions.catalog.kubedb.com           2021-03-10T07:39:46Z
perconaxtradbs.kubedb.com                  2021-03-10T07:39:43Z
perconaxtradbversions.catalog.kubedb.com   2021-03-10T07:39:46Z
pgbouncers.kubedb.com                      2021-03-10T07:39:44Z
pgbouncerversions.catalog.kubedb.com       2021-03-10T07:39:46Z
postgreses.kubedb.com                      2021-03-10T07:39:44Z
postgresversions.catalog.kubedb.com        2021-03-10T07:39:46Z
proxysqls.kubedb.com                       2021-03-10T07:39:44Z
proxysqlversions.catalog.kubedb.com        2021-03-10T07:39:46Z
redises.kubedb.com                         2021-03-10T07:39:45Z
redisversions.catalog.kubedb.com           2021-03-10T07:39:46Z

3、安装KubeDB Catalog
同样,先下载https://github.com/appscode/charts/tree/master/stable/kubedb-catalog

[root@qd01-stop-k8s-master001 kubedb-catalog]# tar -zxf kubedb-catalog-v0.16.2.tgz
[root@qd01-stop-k8s-master001 kubedb-catalog]# cd kubedb-catalog
[root@qd01-stop-k8s-master001 kubedb-catalog]# ls -al
total 24
drwxr-xr-x  3 root root  148 Mar 10 15:48 .
drwxr-xr-x  3 root root   28 Mar 10 15:48 ..
-rw-r--r--  1 root root  321 Jan 26 20:08 Chart.yaml
-rw-r--r--  1 root root  467 Jan 26 20:08 doc.yaml
-rw-r--r--  1 root root  353 Jan 26 20:08 .helmignore
-rw-r--r--  1 root root 3195 Jan 26 20:08 README.md
drwxr-xr-x 12 root root  188 Mar 10 15:48 templates
-rw-r--r--  1 root root  744 Jan 26 20:08 values.openapiv3_schema.yaml
-rw-r--r--  1 root root 1070 Jan 26 20:08 values.yaml

[root@qd01-stop-k8s-master001 kubedb-catalog]# helm install kubedb-catalog --namespace kube-system -f values.yaml  .
NAME: kubedb-catalog
LAST DEPLOYED: Wed Mar 10 15:50:50 2021
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

使用kubedb安装redis

1、先看下官方给的redis生命周期示意图

kubedb安装redis支持如下特性

Features	Availability
Clustering	
Instant Backup	
Scheduled Backup	
Persistent Volume	
Initialize using Snapshot	
Initialize using Script	
Custom Configuration	
Using Custom docker image	
Builtin Prometheus Discovery	
Using Prometheus operator	

2、查看支持的版本

[root@qd01-stop-k8s-master001 kubedb-catalog]# kubectl get redisversions
NAME       VERSION   DB_IMAGE                DEPRECATED   AGE
4.0.11     4.0.11    kubedb/redis:4.0.11                  15m
4.0.6-v2   4.0.6     kubedb/redis:4.0.6-v2                15m
5.0.3-v1   5.0.3     kubedb/redis:5.0.3-v1                15m
6.0.6      6.0.6     kubedb/redis:6.0.6                   15m

3、编辑yaml安装文件
可以参照https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml
这里选择安装6.0.6这个版本,我的集群storageClassName: "rbd",请根据实际修改
如果想自定义redis.conf,请参考https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml

apiVersion: kubedb.com/v1alpha2
kind: Redis
metadata:
  name: redis-cluster
  namespace: op
spec:
  version: 6.0.6
  mode: Cluster
  cluster:
    master: 3
    replicas: 1
  storageType: Durable
  storage:
    resources:
      requests:
        storage: 1Gi
    storageClassName: "rbd"
    accessModes:
      - ReadWriteOnce

执行安装

[root@qd01-stop-k8s-master001 kubedb-community]# kubectl apply -f redis-cluster.yaml
redis.kubedb.com/redis-cluster created

安装完成,可以如下查看
[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get rd,po -n op
NAME                             VERSION   STATUS         AGE
redis.kubedb.com/redis-cluster   6.0.6     Provisioning   6m55s

NAME                         READY   STATUS    RESTARTS   AGE
pod/redis-cluster-shard0-0   1/1     Running   0          6m54s
pod/redis-cluster-shard0-1   1/1     Running   0          6m18s
pod/redis-cluster-shard1-0   1/1     Running   0          5m38s
pod/redis-cluster-shard1-1   1/1     Running   0          5m1s
pod/redis-cluster-shard2-0   1/1     Running   0          4m30s
pod/redis-cluster-shard2-1   1/1     Running   0          4m8s

[root@qd01-stop-k8s-master001 redis]# kubectl get svc -n op
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
redis-cluster        ClusterIP   10.97.197.224   <none>        6379/TCP   5h16m
redis-cluster-pods   ClusterIP   None            <none>        6379/TCP   17m

4、验证集群


[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get pods -n  op  -o jsonpath='{range.items[*]}{.metadata.name} ---------- {.status.podIP}:6379{"\t\n"}{end}' | grep redis
redis-cluster-shard0-0 ---------- 100.64.147.156:6379
redis-cluster-shard0-1 ---------- 100.98.174.218:6379
redis-cluster-shard1-0 ---------- 100.126.252.204:6379
redis-cluster-shard1-1 ---------- 100.113.170.6:6379
redis-cluster-shard2-0 ---------- 100.107.55.69:6379
redis-cluster-shard2-1 ---------- 100.78.230.4:6379

[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-shard0-0  -- redis-cli cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_ping_sent:864
cluster_stats_messages_pong_sent:882
cluster_stats_messages_sent:1746
cluster_stats_messages_ping_received:879
cluster_stats_messages_pong_received:864
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:1746

[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-shard0-0  -- redis-cli cluster nodes
1895cb4b9c31b848666c61000e502f55a29a8255 100.64.147.155:6379@16379 master - 0 1615365162008 2 connected 5461-10922
30bdbf2ca37001774498a9b935afbc1cd2ce389c 100.126.252.203:6379@16379 slave 2c06092fafa99e0158e39e6237a04fed25be3550 0 1615365163000 1 connected
9b2cfbd5c1b417121d410141b6da9512ad29ce3c 100.78.230.3:6379@16379 slave e83446c368839c5fdccf5f70e3b1004eb67cb651 0 1615365163512 3 connected
2c06092fafa99e0158e39e6237a04fed25be3550 100.82.197.130:6379@16379 myself,master - 0 1615365162000 1 connected 0-5460
1379d2b20f26ab13d53068d276ec5d988b7a0273 100.64.122.197:6379@16379 slave 1895cb4b9c31b848666c61000e502f55a29a8255 0 1615365163000 2 connected
e83446c368839c5fdccf5f70e3b1004eb67cb651 100.107.55.68:6379@16379 master - 0 1615365164014 3 connected 10923-16383

本站声明:网站内容来源于网络,如有侵权,请联系我们,我们将及时处理。

  • 分享:
评论
还没有评论
    发表评论 说点什么