No F*cking Idea

Common answer to everything

Failover Redis Setup With Sentinel

| Comments

Long time nothing new, recently i started my own company LambdaCu.be and I was massively busy. If you want to hire me ping me at kuba@lambdacu.be =)

I have in pipeline a lot texts about lua scripting in redis and using it to build some tools but can’t find time to finish this stuff ;/.

Auto failover

Every database wants to have auto failover mechanism. This is a great marketing pitch! haha :) Main thing about is that one of your server can go down and you still are operating as normal and when he will go up again everything is fine.. unless your routing server will go down :D ofc.

2.4.16 / 2.6

Since Redis 2.4 and 2.6 there was this idea of adding it. Antirez wrote a draft spec for it and implemented it as experimental feature. It is really well described here http://redis.io/topics/sentinel so i will just write a short note how did i setup it and how does it feel.

Setup

While preparing to this demo i did everything on master 0ee3f05518e081640c1c6f9ae52c3a414f0feaceso what i did was simply start “master” and “replica servers” with this configs (ofc turn daemonize to yes in production lol)

Standard Master setup with default script on port 6379 and replica with

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

daemonize no
timeout 0
loglevel notice
logfile stdout
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./replica_dir
slave-serve-stale-data yes
slave-read-only yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

pidfile /var/run/redis-replica-7789.pid
port 7789
# replication config
slaveof 127.0.0.1 6379

slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

So i had master and slave running :) that was cool next thing i did was! configure and turn on the sentinel!

1
2
3
4
5
6
7
8
9
10
11

sentinel monitor mymaster 127.0.0.1 6379 1
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 900000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1

sentinel monitor resque 127.0.0.1 7789 1
sentinel down-after-milliseconds resque 10000
sentinel failover-timeout resque 900000
sentinel can-failover resque yes
sentinel parallel-syncs resque 5

And i turned him on! with

1

redis-server sentinel-my.conf --sentinel

And stuff started to work :D

Carnage!!!

So i started easy

1
2
3
4

redis 127.0.0.1:6379> get "hello"
"hello"
redis 127.0.0.1:6379> set "hello" "lulz"
OK

Works! I killed master and connected on 26379 to sentinel master did query

1

 sentinel masters

and got

1
2
3
4
5
6
7

1)  1) "name"
    2) "resque"
    3) "ip"
    4) "127.0.0.1"
    5) "port"
    6) "7789"
    ...

Cool works great :D The only thing that worried me was that when i turned on master after failover (it took 8 sec) he did not pickup he is slave and he did not start replicating data.

when you do this…

You will see beefy

1
2
3
4
5

[36373] 26 Sep 21:15:03.441 # Error condition on socket for SYNC: Connection refused
[36373] 26 Sep 21:15:04.521 * Connecting to MASTER...
[36373] 26 Sep 21:15:04.521 * MASTER <-> SLAVE sync started
[36373] 26 Sep 21:15:04.521 # Error condition on socket for SYNC: Connection refused
[36373] 26 Sep 21:15:05.128 * MASTER MODE enabled (user request)

On the initial slave :) things just went from bad to good :D

Summary

This is cool new feature that you can have master-slave and auto failover server the only thing that driver have to do is if you get error while connecting / querying is to ask sentinel for new master connect and retry :) It is very basic but…

I like it!

THIS IS EXPERIMENTAL FEATURE and much more info about it you can finde here http://redis.io/topics/sentinel. Especially about pub/sub way of watching stuff / events while they occur.

Comments