Answer a question

I have a Makefile, that runs a docker-compose, which has a container that executes a python script. I want to be able to pass a variable in the command-line to the Makefile and print it within the python script (testing.py).

My directory looks like:

main_folder:
  -docker-compose.yaml
  -Makefile
  -testing.py

I have tried with the following configuration. The Makefile is:

.PHONY: run run-prod stop stop-prod rm

run:
    WORKING_DAG=$(working_dag) docker-compose -f docker-compose.yml up -d --remove-orphans --build --force-recreate

The docker-compose is:

version: "3.7"
services:
  prepare_files:
    image: apache/airflow:1.10.14
    environment:
      WORKING_DAG: ${working_dag}
      PYTHONUNBUFFERED: 1
    entrypoint: /bin/bash
    command: -c "python3 testing.py $$WORKING_DAG"

And the file testing.py is:

import sys

print(sys.argv[0], flush=True)

When I run in the command line:

 make working_dag=testing run

It doesn't fail but it does not print anything neither. How could I make it? Thanks

Answers

I believe that the variable WORKING_DAG is getting assigned correctly through the command-line and Makefile is passing it correctly to the docker-compose. I verified it by running the container to not be destroyed and then after logging into the container, I checked the value of WORKING_DAG:

To not destroy the container once the docker execution is completed, I modified the docker-compose.yml, as follows:

version: "3.7"
services:
  prepare_files:
    image: apache/airflow:1.10.14
    environment:
      WORKING_DAG: ${working_dag}
      PYTHONUNBUFFERED: 1
    entrypoint: /bin/bash
    command: -c "python3 testing.py $$WORKING_DAG"
    command: -c "tail -f /dev/null"

airflow@d8dcb07c926a:/opt/airflow$ echo $WORKING_DAG
testing

The issue that docker does not display Python's std.out when deploying with docker-compose was already commented in Github, here, and it still not resolved. Making it work when using docker-compose, is only possible if we transfer/mount the file into the container, or if we use Dockerfile instead.

When using a Dockerfile, you only have to run the corresponding script as follows,

CMD ["python", "-u", "testing.py", "$WORKING_DAG"]

To mount the script into the container, please look at @DazWilkin's answer, here.