Answer a question

I am trying to setup one master one slave and one sentinel on docker, for that I wrote this docker compose file.

version: '3'
services:
  redis-master:
    container_name: "redis-master"
    image: redis
    ports:
      - "6379:6379"
    command: "redis-server /etc/redis.conf"
    volumes:
      - "./data/master:/data/"
      - "./master.conf:/etc/redis.conf"

  redis-slave:
    container_name: "redis-slave"
    image: redis
    ports:
      - "6380:6379"
    command: "redis-server /etc/redis.conf"
    volumes:
      - "./data/slave:/data/"
      - "./slave.conf:/etc/redis.conf"
    depends_on:
      - redis-master

  redis-sentinel:
    container_name: 'redis-sentinel'
    image: redis
    ports:
      - "26379:26379"
    command: >
      bash -c "chmod 777 /etc/sentinel.conf
      && redis-server /etc/sentinel.conf --sentinel"
    volumes:
      - "./sentinel.conf:/etc/sentinel.conf"
    depends_on:
      - redis-master
      - redis-slave

But when I try to build it using sudo docker-compose up --build --force all the services are runnning fine except the redis-sentinel. I got this error in logs

redis-sentinel    | 1:X 16 Dec 2021 19:15:21.486 # +sdown master mymaster 172.23.0.2 6379
redis-sentinel    | 1:X 16 Dec 2021 19:15:21.486 # +odown master mymaster 172.23.0.2 6379 #quorum 1/1
redis-sentinel    | 1:X 16 Dec 2021 19:15:21.486 # +new-epoch 8
redis-sentinel    | 1:X 16 Dec 2021 19:15:21.487 # +try-failover master mymaster 172.23.0.2 6379
redis-sentinel    | 1:X 16 Dec 2021 19:15:22.955 # Could not rename tmp config file (Device or resource busy)
redis-sentinel    | 1:X 16 Dec 2021 19:15:22.955 # WARNING: Sentinel was not able to save the new configuration on disk!!!: Device or resource busy

I understand this is some file permission and I have to make sentinel.conf executable but I am not able to think of any possible solutions in docker.

Answers

First of all, the sentinel command should be only like this (we will sort out the permissions later):

redis-server /etc/sentinel.conf --sentinel

In order to resolve that warning, you have to do the following (for all the redis nodes - master, slave, sentinel):

In the folder where the docker-compose.yml resides, create a "config" folder, with the following structure:

/config
  /redis-master
     redis.conf
  /redis-slave
     redis.conf
  /redis-sentinel
     redis.conf

Inside config/redis-master folder you copy your current master.conf file as redis.conf (for the sake of simplicity).

Inside config/redis-slave you copy your current slave.conf file as redis.conf

Inside config/redis-sentinel you will copy your current sentinel.conf as redis.conf

Then, execute this command, in order to give full rights to all the content of your "config" folder:

chmod -R 0777 config/

Now, change your service definitions in the docker-compose, like this: (notice my changes in the "command" and "volumes" sections)

  redis-master:
    container_name: "redis-master"
    image: redis
    ports:
      - "6379:6379"
    command: "redis-server /etc/redis-config/redis.conf"
    volumes:
      - "./data/master:/data/"
      - "./config/redis-master:/etc/redis-config"

  redis-slave:
    container_name: "redis-slave"
    image: redis
    ports:
      - "6380:6379"
    command: "redis-server /etc/redis-config/redis.conf"
    volumes:
      - "./data/slave:/data/"
      - "./config/redis-slave:/etc/redis-config"
    depends_on:
      - redis-master

  redis-sentinel:
    container_name: 'redis-sentinel'
    image: redis
    ports:
      - "26379:26379"
    command: "redis-server /etc/redis-config/redis.conf --sentinel"
    volumes:
      - "./config/redis-sentinel:/etc/redis-config"
    depends_on:
      - redis-master
      - redis-slave

Conclusions:

1. In order for redis to have rights to modify the configurations, you have mount the entire configuration folder, not the file itself (and the folder contents should be writable).
2. Back-up all the original master.conf, slave.conf, sentinel.conf files. Be aware that redis will alter your local configuration files (redis.conf), because eventually you will execute failovers, so the slave config will be turned into a master one and the opposite for the master config. It's up to you how you avoid committing the changes to source control. For example, you can keep the originals in a separate folder, and copy them to the mounted folder by a deploy script, before running "docker-compose up" command.
3. All the changes that redis will apply to these config files will be appended at the end of the config and will be preceded by this comment:

# Generated by CONFIG REWRITE

This solution is tested by me and it works. I came out with this solution after reading @yossigo 's answer regarding a similar warning:

https://github.com/redis/redis/issues/8172