Answer a question

I am trying to build images for my app. However, when I run "docker-compose up" command, it builds some of the containers twice. I couldn't figure the reason of it. I think the tags cause this kind of situation, but I couldn't figure where 'latest' tag come from.

Here it is my docker-compose.yml:

version: '3.2'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTIC_PASSWORD: changeme
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
    networks:
      - elk

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./kibana/config/kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - elk
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: 'bitnami/kafka:latest'
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9093:9093"
    networks:
      - elk
    environment:
      KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
      ALLOW_PLAINTEXT_LISTENER: 'yes'
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CFG_LISTENERS: CLIENT://:9092,EXTERNAL://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS: CLIENT://kafka:9092,EXTERNAL://localhost:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: CLIENT
    links:
      - logstash

  app:
    container_name: "ml-pipeline"
    build: .
    ports:
    - "7000:7000"
    - "5001:5001"
    depends_on:
      - kafka
      - elasticsearch
      - logstash
    networks:
      - elk
    links:
      - kafka

networks:
  elk:
    driver: bridge

volumes:
  elasticsearch:

And output of this is:

As you can see there are duplicate images. How can I solve it ?

Answers

All 3 ELK containers have a build context with a Dockerfile that by default only consists of a FROM line. In the Dockerfiles you could add additional plugins.

part of your docker-compose.yml:

    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION

logstash/Dockerfile:

ARG ELK_VERSION

# https://github.com/elastic/logstash-docker
FROM docker.elastic.co/logstash/logstash:${ELK_VERSION}

# Add your logstash plugins setup here
# Example: RUN logstash-plugin install logstash-filter-json

docker-compose pulls the Image docker.elastic.co/logstash/logstash:${ELK_VERSION} and builds it's own version version twitter-stream-dl-docker_XXX. Since the build doesn't do anything it simply tags the old image with the new tag so they have the same Image ID.

In case you're wondering your folder's name is twitter-stream-dl-docker so the images have that tag (or you used docker-compose -p twitter-stream-dl-docker).

I hope that clears things up, but feel free to ask anything that's ambigious.