Creating Redis Cluster

Jan 14, 2020 · 5 min read · redis ·

Share on:

後端服務需要使用 Redis Cluster, 此篇紀錄在 Centos 7 安裝 Redis Cluster 的過程.

Cluster教學 https://redis.io/topics/cluster-tutorial

注意: binary file放置位置,會造成SELinux異常

在 Build Redis 時,其中有一個參數 PREFIX 是編譯完成後 binary 的位置,如下. 使用 Centos7 時,有變更這個參數,導致 redis-server 無法在 /var/log/redis 底下建立檔案,碰到 avc denied open,若不異動則正常, 原因是SELinux引起的.

1[Justin.Lee@dev-cache redis-5.0.7]$ cat src/Makefile
2...
3PREFIX?=/usr/local
4INSTALL_BIN=$(PREFIX)/bin
5INSTALL=install
6...

注意: 建置 Redis Cluster 群集時, 密碼配置時要注意

建立 Redis Cluster 時有配置密碼, 第一次建立沒問題, 但是若做restart時, 報錯誤說NOAUTH Authentication required.,原因是因為設定檔內只有配置 requirepass 參數未配置 masterauth 參數導致的,補上排除.

 1[Justin.Lee@dev-cache redis]# cat /etc/redis/redis_7000.conf
 2...
 3requirepass {{ redis_config_requirepass }}
 4
 5# If the master is password protected (using the "requirepass" configuration
 6# directive below) it is possible to tell the replica to authenticate before
 7# starting the replication synchronization process, otherwise the master will
 8# refuse the replica request.
 9
10masterauth {{ redis_config_requirepass }}
11...

error log

 1[Justin.Lee@dev-cache redis]# cat /var/log/redis/redis_7004.log
 210920:S 20 Jan 2020 02:29:51.238 # MASTER aborted replication with an error: NOAUTH Authentication required.
 310920:S 20 Jan 2020 02:29:52.240 * Connecting to MASTER 192.168.20.188:7002
 410920:S 20 Jan 2020 02:29:52.240 * MASTER <-> REPLICA sync started
 510920:S 20 Jan 2020 02:29:52.240 * Non blocking connect for SYNC fired the event.
 610920:S 20 Jan 2020 02:29:52.240 * Master replied to PING, replication can continue...
 710920:S 20 Jan 2020 02:29:52.240 * (Non critical) Master does not understand REPLCONF listening-port: -NOAUTH Authentication required.
 810920:S 20 Jan 2020 02:29:52.240 * (Non critical) Master does not understand REPLCONF capa: -NOAUTH Authentication required.
 910920:S 20 Jan 2020 02:29:52.240 * Partial resynchronization not possible (no cached master)
1010920:S 20 Jan 2020 02:29:52.240 # Unexpected reply to PSYNC from master: -NOAUTH Authentication required.

確認主機配置

firewall

 1[Justin.Lee@dev-db2 ~]$ sudo firewall-cmd --list-all
 2public (active)
 3  target: default
 4  icmp-block-inversion: no
 5  interfaces: ens192
 6  sources:
 7  services: ssh dhcpv6-client
 8  ports:
 9  protocols:
10  masquerade: no
11  forward-ports:
12  source-ports:
13  icmp-blocks:
14  rich rules:

ip address

 1[Justin.Lee@dev-db2 ~]$ ip a
 21: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 3    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 4    inet 127.0.0.1/8 scope host lo
 5       valid_lft forever preferred_lft forever
 6    inet6 ::1/128 scope host
 7       valid_lft forever preferred_lft forever
 82: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 9    link/ether 00:0c:29:90:78:61 brd ff:ff:ff:ff:ff:ff
10    inet 192.168.20.189/24 brd 192.168.20.255 scope global noprefixroute ens192
11       valid_lft forever preferred_lft forever
12    inet6 fe80::13b6:9f1a:cc9f:b263/64 scope link noprefixroute
13       valid_lft forever preferred_lft forever

before starting

Redis Cluster TCP ports

Every Redis Cluster node requires two TCP connections open. The normal Redis TCP port used to serve clients, for example 6379, plus the port obtained by adding 10000 to the data port, so 16379 in the example.

Redis Cluster data sharding

Redis Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call an hash slot.

There are 16384 hash slots in Redis Cluster, and to compute what is the hash slot of a given key, we simply take the CRC16 of the key modulo 16384.

install Redis 5.0.7

安裝 remi-release-7.rpm

1[Justin.Lee@dev-db2 ~]$ yum.repos.d]$ sudo yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm

安裝Redis 5.0.7-1.el7.remi

1[Justin.Lee@dev-db2 ~]$ sudo yum list redis --enablerepo=remi --showduplicates
2[Justin.Lee@dev-db2 ~]$ sudo yum install redis-5.0.7-1.el7.remi --enablerepo=remi

Redis service disable

1[Justin.Lee@dev-db2 ~]$ sudo systemctl disable redis

setting Redis configuration

配置其中一個 redis.conf, 其中7000單一個 Redis 的 port 號. 其他的 redis.conf 就依序更改 7000 的數字.

 1# Redis 5.0.7 configuration file example.
 2bind 0.0.0.0
 3protected-mode no
 4port 7000
 5daemonize yes
 6pidfile /var/run/redis_7000.pid
 7logfile /var/log/redis/redis_7000.log
 8dir /var/lib/redis
 9appendonly yes
10cluster-enabled yes
11cluster-config-file nodes-6379.conf
12cluster-node-timeout 15000
13cluster-require-full-coverage no

確認 port 後, 必須開啟防火牆, 例如 port 7000, 就必須要開起 7000/tcp 跟 17000/tcp.

1[Justin.Lee@dev-cache ~]$ sudo firewall-cmd --add-port=7000/tcp --permanent
2[Justin.Lee@dev-cache ~]$ sudo firewall-cmd --add-port=17000/tcp --permanent
3[Justin.Lee@dev-cache ~]$ sudo firewall-cmd --reload
4[Justin.Lee@dev-cache ~]$ sudo firewall-cmd --list-all

Creating the cluster

最少需要 3 個 master, 參數--cluster-replicas為一個 master 需要幾個 slave, 如下依此類推.

每群Redis Cluster

--cluster-replicas 0, 1 master 0 slave
--cluster-replicas 1, 1 master 1 slave
--cluster-replicas 2, 1 master 2 slave

 1[Justin.Lee@dev-db2 ~]$ redis-cli --cluster help
 2[Justin.Lee@dev-db2 ~]$ redis-cli --cluster create 192.168.20.189:7000 192.168.20.189:7001 192.168.20.189:7002 192.168.20.189:7003 192.168.20.189:7004 192.168.20.189:7005 --cluster-replicas 1 -a password
 3>>> Performing hash slots allocation on 6 nodes...
 4Master[0] -> Slots 0 - 5460
 5Master[1] -> Slots 5461 - 10922
 6Master[2] -> Slots 10923 - 16383
 7Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
 8Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
 9Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
10>>> Trying to optimize slaves allocation for anti-affinity
11[WARNING] Some slaves are in the same host as their master
12M: 99ff53a04d977692016e7df517533226ff3bd218 127.0.0.1:7000
13   slots:[0-5460] (5461 slots) master
14M: 4f9af42e6c6e2239e46cf9cd14af3e8c688638d5 127.0.0.1:7001
15   slots:[5461-10922] (5462 slots) master
16M: 01279d2e588ac7b25da2df232412153b26950184 127.0.0.1:7002
17   slots:[10923-16383] (5461 slots) master
18S: c12c3b7aed21133f078a3262ec201ec9106a6ae0 127.0.0.1:7003
19   replicates 99ff53a04d977692016e7df517533226ff3bd218
20S: ea108846aba5cd010feebe8482f29babcbc51ee0 127.0.0.1:7004
21   replicates 4f9af42e6c6e2239e46cf9cd14af3e8c688638d5
22S: 7ea79aec309af8048453c6059d3cd35da57468df 127.0.0.1:7005
23   replicates 01279d2e588ac7b25da2df232412153b26950184

驗證

連至其中一台Redis後可以開始做查詢,例如下面的命令.

1[Justin.Lee@dev-db2 ~]$ redis-cli -c -p 7000 -h  192.168.20.189 -a password
2192.168.20.189:7000> cluster info
3192.168.20.189:7000> cluster nodes

查看log,發現有些warning,依照建議去調整主機配置.

1[Justin.Lee@dev-db2 redis]$ cat redis_7000.log
22030:C 16 Jan 2020 01:27:43.297 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=2030, just started
3...
42031:M 16 Jan 2020 01:27:43.301 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
52031:M 16 Jan 2020 01:27:43.301 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
62031:M 16 Jan 2020 01:27:43.301 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
7...