I have an nginx web server deployed with docker swarm, and I want to be able to deploy it also with kubernetes.

Right now I'm having trouble inserting the nginx configuration files into the container.

I'll first post here what I already do in docker swarm, and then what I tried in kubernetes.

Dockerfile:

FROM    "nginx:1.19.1-alpine"   AS nginx
[...]
RUN                                                             \
        rm -fv  /etc/nginx/nginx.conf                           && \
        ln -svT /usr/local/etc/nginx/nginx.conf                 \
                /etc/nginx/nginx.conf                           && \
        rm -frv /etc/nginx/conf.d                               && \
        ln -svT /usr/local/etc/nginx/conf.d                     \
                /etc/nginx/conf.d
[...]

Basically I set up the image so that I can place my custom nginx config files into /usr/local/etc/ instead of /etc/

Docker swarm:

docker-compose.yaml:

configs:
    nginx_conf:
        file: /run/configs/www/etc/nginx/nginx.conf
    nginx_conf_security-parameters:
        file: /run/configs/www/etc/nginx/conf.d/security-parameters.conf
    nginx_conf_server:
        file: /run/configs/www/etc/nginx/conf.d/server.conf

networks:
    alejandro-colomar:

services:
    www:
        configs:
        -
            mode: 0440
            source: nginx_conf
            target: /usr/local/etc/nginx/nginx.conf
        -
            mode: 0440
            source: nginx_conf_security-parameters
            target: /usr/local/etc/nginx/conf.d/security-parameters.conf
        -
            mode: 0440
            source: nginx_conf_server
            target: /usr/local/etc/nginx/conf.d/server.conf
        deploy:
            placement:
                constraints:
                -   node.role == worker
            replicas: 1
            restart_policy:
                condition: any
        image: "alejandrocolomar/www:0.16-a6-kube"
        networks:
        -
            "alejandro-colomar"
        ports:
        -   "32001:8080"

version: "3.8"

Here I take the nginx custom config files, which I first put in /run/configs/ with a script so that they are in ram, and introduce them in the container as configs in the right place (/usr/local/etc/nginx/ and its subdir conf.d/).

I'd like to do the same in kubernetes, and I read that I should use a projected volume for that (if it's doable with normal volumes or in some other way, without having to use any dirty workarounds, I'm also open to that), so I tried the following (after seeing some examples which weren't very clear to me):

config.sh:

kubectl create configmap "nginx-conf-cm"                        \
        --from-file "/run/configs/www/etc/nginx/nginx.conf"
kubectl create configmap "nginx-conf-security-parameters-cm"    \
        --from-file "/run/configs/www/etc/nginx/conf.d/security-parameters.conf"
kubectl create configmap "nginx-conf-server-cm"                 \
        --from-file "/run/configs/www/etc/nginx/conf.d/server.conf"

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: www-deploy
spec:
    replicas: 1
    selector:
        matchLabels:
            service: www-svc
    template:
        metadata:
            labels:
                service: www-svc
        spec:
            containers:
            -
                image: "alejandrocolomar/www:0.16-a6-kube"
                name: www-container
                volumeMounts:
                -
                    mountPath: /usr/local/etc/nginx/
                    name: nginx-volume
                    readOnly: true
            volumes:
            -
                name: nginx-volume
                projected:
                    sources:
                    -
                        configMap:
                            name: nginx-conf-cm
                            path: "nginx.conf"
                    -
                        configMap:
                            name: nginx-conf-security-parameters-cm
                            path: "conf.d/security-parameters.conf"
                    -
                        configMap:
                            name: nginx-conf-server-cm
                            path: "conf.d/server.conf"

service.yaml: (This one I put it here only for completeness)

apiVersion: v1
kind: Service
metadata:
    name: www
spec:
    ports:
    -
        nodePort: 32001
        port: 8080
    selector:
        service: www-svc
    type: NodePort

The deployment failed, of course, but it wasn't very wrong I think. When I entered in the container to debug it, the problem was that the three files were placed into /usr/local/etc/nginx/, and the subdir conf.d/ was not created:

/usr/local/etc/nginx/nginx.conf
/usr/local/etc/nginx/security-parameters.conf
/usr/local/etc/nginx/server.conf

How should I fix that (presumably in deployment.yaml) so that I have the following files in the container?:

/usr/local/etc/nginx/nginx.conf
/usr/local/etc/nginx/conf.d/security-parameters.conf
/usr/local/etc/nginx/conf.d/server.conf

There are multiple ways to handle this. One solution would be to create 3 separate volumes from these individual config maps and mount each one into its appropriate destination file/folder.

volumes:
        - name: nginx-cm
          configMap:
              name: nginx-conf-cm
        - name: nginx-sec
          configMap:
              name: nginx-conf-security-parameters-cm
        - name: nginx-serv
          configMap:
              name: nginx-conf-server-cm

        ...
        containers:
            -
                image: "alejandrocolomar/www:0.16-a6-kube"
                name: www-container
                volumeMounts:
                -   mountPath: /usr/local/etc/nginx/nginx.conf
                    name: nginx-cm
                    subPath: nginx.conf
                    readOnly: true
                -   mountPath: /usr/local/etc/nginx/conf.d/security-parameters.conf
                    name: nginx-sec
                    subPath: security-parameters.conf
                    readOnly: true
                -   mountPath: /usr/local/etc/nginx/conf.d/server.conf
                    name: nginx-serv
                    subPath: server.conf
                    readOnly: true

using mountPath and subPath in volumeMounts allows you to pick specific file from a given config map (doesn't matter much since you have on file per cm) and mount it as a file (not overriding the other content in the existing folder).

To explain a bit the code above:

-   mountPath: /usr/local/etc/nginx/nginx.conf
    name: nginx-cm
    subPath: nginx.conf

tells kubernetes to use volume with the name of nignx-cm (defined in a volume section). Pick file nginx.conf (via subpath) that can be found in the associated config map and expose it in the container (mountPath) at the location /usr/local/etc/nginx/nginx.conf.

I have not run the code so there might be typos

PS: note that it might be better to create a custom configuration file inside of ../etc/nginx/conf.d/ instead of overriding ../etc/nginx/nginx.conf. That way you wouldn't need to worry about destroying the original files ../etc/nginx/ and you could mount the whole volume instead of using subpath (to avoid issues with config updates)