标注环境CVAT在Ubuntu18.04下搭建

CVAT Ubuntu 18.04 环境搭建CVAT Ubuntu 18.04 标注环境搭建环境准备Ubuntu环境搭建换源路径Anaconda 下载安装CVAT 下载基础环境配置Ubuntu环境配置★ CVAT Dockerfile修改★ CVAT docker-compose.yml修改耐心等待与使用可能遇到的问题创建超级用户创建普通用户可以探索的使用方法CVAT Ubuntu 18.04 .

AgtNN

3772人浏览 · 2019-11-07 19:05:17

AgtNN · 2019-11-07 19:05:17 发布

CVAT Ubuntu 18.04 环境搭建

CVAT Ubuntu 18.04 标注环境搭建
！对了！在chrome下使用这个CVAT

CVAT Ubuntu 18.04 标注环境搭建

记录一次在Ubuntu 18.04下搭建CVAT 环境的过程，将会在本月内再次搭建一次

环境准备

Ubuntu环境搭建
Anaconda 下载
CVAT 下载
换源路径

Ubuntu环境搭建

传送门链接: link.
U盘启动制作工具: link.

换源路径

推荐Ubuntu清华源 link.

# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse

# 预发布软件源，不建议启用
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse

推荐Anaconda清华源 link.

conda config --set show_channel_urls yes
nano ~/.condarc

# 在.condarc中贴入如下源信息

channels:
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

# 在.condarc追加如下超时时间

remote_connect_timeout_secs: 40.0
remote_read_timeout_secs: 100.0

Anaconda 下载安装

下载对应的版本，路径来源（清华源）
Anaconda3-2019.10-Linux-x86_64.sh .
Anaconda3-2019.10-Windows-x86_64.exe .

CVAT 下载

git clone https://github.com/opencv/cvat

基础环境配置

Ubuntu Docker 环境配置

# 摘自 https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md

sudo apt-get update
sudo apt-get install -y \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg-agent \
  software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io


# docker进程使用Unix Socket而不是TCP端口。
# 而默认情况下，Unix socket属于root用户，需要root权限才能访问。
# add user in docker group
sudo groupadd docker     #添加docker用户组
sudo gpasswd -a $USER docker     #将登陆用户加入到docker用户组中
newgrp docker     #更新用户组
docker ps    #测试docker命令是否可以使用sudo正常使用

# install docker-compose
pip install docker-compose -i https://pypi.tuna.tsinghua.edu.cn/simple

★ CVAT Dockerfile修改

原生的Dockerfile会自动去官方源去做更新，因为国内网络问题，所以要换源
步骤如下

在CVAT目录下，放入source.list

source内容是清华源，对应Ubuntu版本
修改如下位置的Dockerfile
变动后结果如下

FROM ubuntu:16.04

ARG http_proxy
ARG https_proxy
ARG no_proxy
ARG socks_proxy

ENV TERM=xterm \
    http_proxy=${http_proxy}   \
    https_proxy=${https_proxy} \
    no_proxy=${no_proxy} \
    socks_proxy=${socks_proxy}

ENV LANG='C.UTF-8'  \
    LC_ALL='C.UTF-8'

ARG USER
ARG DJANGO_CONFIGURATION
ENV DJANGO_CONFIGURATION=${DJANGO_CONFIGURATION}

# Install necessary apt packages
RUN apt-get update && \
    apt-get install -yq \
        python-software-properties \
        software-properties-common \
        wget && \
    add-apt-repository ppa:mc3man/xerus-media -y && \
    add-apt-repository ppa:mc3man/gstffmpeg-keep -y && \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq \
        apache2 \
        apache2-dev \
        libapache2-mod-xsendfile \
        supervisor \
        ffmpeg \
        gstreamer0.10-ffmpeg \
        libldap2-dev \
        libsasl2-dev \
        python3-dev \
        python3-pip \
        unzip \
        unrar \
        p7zip-full \
        vim && \
    add-apt-repository --remove ppa:mc3man/gstffmpeg-keep -y && \
    add-apt-repository --remove ppa:mc3man/xerus-media -y && \
    rm -rf /var/lib/apt/lists/*

# Add a non-root user
ENV USER=${USER}
ENV HOME /home/${USER}
WORKDIR ${HOME}
RUN adduser --shell /bin/bash --disabled-password --gecos "" ${USER}

COPY components /tmp/components

# OpenVINO toolkit support
ARG OPENVINO_TOOLKIT
ENV OPENVINO_TOOLKIT=${OPENVINO_TOOLKIT}
RUN if [ "$OPENVINO_TOOLKIT" = "yes" ]; then \
        /tmp/components/openvino/install.sh; \
    fi

# CUDA support
ARG CUDA_SUPPORT
ENV CUDA_SUPPORT=${CUDA_SUPPORT}
RUN if [ "$CUDA_SUPPORT" = "yes" ]; then \
        /tmp/components/cuda/install.sh; \
    fi

# Tensorflow annotation support
ARG TF_ANNOTATION
ENV TF_ANNOTATION=${TF_ANNOTATION}
ENV TF_ANNOTATION_MODEL_PATH=${HOME}/rcnn/inference_graph
RUN if [ "$TF_ANNOTATION" = "yes" ]; then \
        bash -i /tmp/components/tf_annotation/install.sh; \
    fi

ARG WITH_TESTS
RUN if [ "$WITH_TESTS" = "yes" ]; then \
        wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
        echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | tee /etc/apt/sources.list.d/google-chrome.list && \
        wget -qO- https://deb.nodesource.com/setup_9.x | bash - && \
        apt-get update && \
        DEBIAN_FRONTEND=noninteractive apt-get install -yq \
            google-chrome-stable \
            nodejs && \
        rm -rf /var/lib/apt/lists/*; \
        mkdir tests && cd tests && npm install \
            eslint \
            eslint-detailed-reporter \
            karma \
            karma-chrome-launcher \
            karma-coveralls \
            karma-coverage \
            karma-junit-reporter \
            karma-qunit \
            qunit; \
        echo "export PATH=~/tests/node_modules/.bin:${PATH}" >> ~/.bashrc; \
    fi

# Install and initialize CVAT, copy all necessary files
COPY cvat/requirements/ /tmp/requirements/
COPY supervisord.conf mod_wsgi.conf wait-for-it.sh manage.py ${HOME}/
RUN pip3 install --no-cache-dir -r /tmp/requirements/${DJANGO_CONFIGURATION}.txt  -i https://pypi.tuna.tsinghua.edu.cn/simple

# Install git application dependencies
RUN apt-get update
RUN apt-get install apt-transport-https
ADD sources.list /etc/apt/
RUN apt-get update && \
    apt-get install -y ssh netcat-openbsd git curl zip  && \
#    wget -qO /dev/stdout https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
    apt-get install -y git-lfs && \
    git lfs install && \
    rm -rf /var/lib/apt/lists/* && \
    if [ -z ${socks_proxy} ]; then \
        echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30\"" >> ${HOME}/.bashrc; \
    else \
        echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30 -o ProxyCommand='nc -X 5 -x ${socks_proxy} %h %p'\"" >> ${HOME}/.bashrc; \
    fi

# Download model for re-identification app
ENV REID_MODEL_DIR=${HOME}/reid
RUN if [ "$OPENVINO_TOOLKIT" = "yes" ]; then \
        mkdir ${HOME}/reid && \
        wget https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/person-reidentification-retail-0079/FP32/person-reidentification-retail-0079.xml -O reid/reid.xml && \
        wget https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/person-reidentification-retail-0079/FP32/person-reidentification-retail-0079.bin -O reid/reid.bin; \
    fi

# TODO: CHANGE URL
ARG WITH_DEXTR
ENV WITH_DEXTR=${WITH_DEXTR}
ENV DEXTR_MODEL_DIR=${HOME}/dextr
RUN if [ "$WITH_DEXTR" = "yes" ]; then \
        mkdir ${DEXTR_MODEL_DIR} -p && \
        wget https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip -O ${DEXTR_MODEL_DIR}/dextr.zip && \
        unzip ${DEXTR_MODEL_DIR}/dextr.zip -d ${DEXTR_MODEL_DIR} && rm ${DEXTR_MODEL_DIR}/dextr.zip; \
    fi

COPY ssh ${HOME}/.ssh
COPY cvat/ ${HOME}/cvat
COPY cvatjs/ ${HOME}/cvatjs
COPY tests ${HOME}/tests
# Binary option is necessary to correctly apply the patch on Windows platform.
# https://unix.stackexchange.com/questions/239364/how-to-fix-hunk-1-failed-at-1-different-line-endings-message
RUN patch --binary -p1 < ${HOME}/cvat/apps/engine/static/engine/js/3rdparty.patch
RUN chown -R ${USER}:${USER} .

# RUN all commands below as 'django' user
USER ${USER}

RUN mkdir data share media keys logs /tmp/supervisord
RUN python3 manage.py collectstatic

EXPOSE 8080 8443
ENTRYPOINT ["/usr/bin/supervisord"]

★ CVAT docker-compose.yml修改

变动如下

# cvat ports 下增加
    ports:
      - "0.0.0.0:9000:8080"
# 使外网可以访问 例如192.168.1.100:9000

# cvat environment 下增加
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: "*"
# 使外网可以访问

变动后结果如下

#
# Copyright (C) 2018 Intel Corporation
#
# SPDX-License-Identifier: MIT
#
version: "2.3"

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:10.3-alpine
    networks:
      default:
        aliases:
          - db
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
    volumes:
      - cvat_db:/var/lib/postgresql/data

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0.5-alpine
    networks:
      default:
        aliases:
          - redis
    restart: always

  cvat:
    container_name: cvat
    image: cvat
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    ports:
      - "0.0.0.0:9000:8080"
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
        TF_ANNOTATION: "no"
        USER: "django"
        DJANGO_CONFIGURATION: "production"
        WITH_TESTS: "no"
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: "*"
    volumes:
      - /cvat/data:/home/ai/cvatdata/data
      - /cvat/keys:/home/ai/cvatdata/keys
      - /cvat/logs:/home/ai/cvatdata/logs
      - /cvat/Users:/home/ai/cvatdata/models

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_models:

耐心等待与使用

docker-compose build

安装过程中，不可避免有些遗漏的地方也是用官方源去下载的，不过数量较少，所以也没有完全去把所有的源都换了。
在网络限制在400kb/s时，docker-compose build一次大概45min

可能遇到的问题

创建超级用户失败

docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'

错误信息是：LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...

处理办法

# 1. 进入到docker镜像中
docker exec -it cvat /bin/bash
# 2. python3 ~/manage.py createsuperuser
# 再运行一次，看看具体的错误信息，一般情况下，是要进行一次migrate
python3 ~/manage.py migrate
# 3. 然后再次运行创建超级用户的命令
python3 ~/manage.py createsuperuser

上述操作之后，依旧没法访问设定的9000端口

处理办法1

# 1. 还是进入到docker镜像中
docker exec -it cvat /bin/bash
# 2. 启动django
python3 manage.py runserver 0.0.0.0:8080
然后按住ctrl + p + q 退出当前docker exec 环境

20191210 增加因PC突然断电，然后无法启动的处理办法

# 不要用 -d
docker-compose up
# 此时注意看打印的日志，可能会多如下日志
cvat          | 2019-12-10 10:23:33,918 DEBG 'runserver' stderr output:
cvat          | [Tue Dec 10 10:23:33.918640 2019] [core:warn] [pid 18:tid 140294961167296] AH00098: pid file /tmp/mod_wsgi-localhost:8080:1000/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
# 怎么办？
# 0 照常启动 不后台
docker-compose up
# 1 还是进入到docker里
docker exec -it cvat /bin/bash
# 2 删掉或备份 mod_wsgi_localhost:8080:1000
cd /tmp
rm -rf mod_wsgi-localhost\:8080\:1000/
# 3 停止docker-compose
ctrl c
# 4 重启
docker-compose up -d

启动后页面展示不全，日志展示

[2020-05-27 12:53:59,190] WARNING django.request: Not Found: /static/CACHE/js/239c24a082e8.js
[2020-05-27 12:53:59,195] WARNING django.request: Not Found: /static/CACHE/js/d91026595cdb.js
[2020-05-27 12:53:59,197] WARNING django.server: "GET /static/CACHE/js/239c24a082e8.js HTTP/1.1" 404 77
[2020-05-27 12:53:59,197] WARNING django.server: "GET /static/CACHE/js/d91026595cdb.js HTTP/1.1" 404 77

原因未知。。。
解决办法：
回到docker-compose.yml所在的目录，docker-compose restart

创建超级用户

方法有两种

直接用bash

docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'

进入到镜像内运行

# 1. 进入到docker镜像中
docker exec -it cvat /bin/bash
# 2. 运行命令
python3 ~/manage.py createsuperuser

创建普通用户

打开admin网页

http://192.168.1.100:9000/admin/

点击添加用户按钮

可以探索的使用方法

与git集成
自动创建标注任务
自动标注

！对了！在chrome下使用这个CVAT

向您推荐>>Eolink开发者社区

权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区

更多推荐

ELK实现containerd的容器日志采集展示【基于logging的全栈监测】

企业级ELK Stack构建介绍

云原生

深入理解 Mocha 测试框架：从零实现一个 Mocha

前言什么是自动化测试自动化测试在很多团队中都是Devops环节中很难执行起来的一个环节，主要原因在于测试代码的编写工作很难抽象，99%的场景都需要和业务强绑定，而且写测试代码的编写工作量往往比编写实际业务代码的工作量更多。在一些很多业务场景中投入产出比很低，适合写自动化测试的应该是那些中长期业务以及一些诸如组件一样的基础库。自动化测试是个比较大的概念，其中分类也比较多，比如单元测试，端对端测试，集

云原生

(20200916 Solved)docker-compose up创建容器自动退出

问题描述如题，创建容器后自动退出了。并且docker start container无效解决方案原因是缺失了控制终端的配置，需要在docker-compose.yml中增加tty:true ，有时候这样也不行，需要再增加一个command:/bin/bash，命令不一定是这个，需要是一个不会退出的命令，然后用-d后台启动容器。Referencesdocker-compose启动容器后自动退出...

云原生

所有评论(0)

查看更多评论

AgtNN

@cahesi

已为社区贡献1条内容