Kubernetes实录系列记录文档完整目录参考: Kubernetes实录-目录

相关记录链接地址 ：

• 集群部署配置(1) CentOS7配置docker环境
• 集群部署配置(2) 配置企业级镜像仓库Harbor

harbor是部署容器私有镜像仓库的事实标准组件，当前版本已经发布到2.0.0(刚刚出炉，还有很多bug要修复)，其中1版本的最新小版本是1.10.3。
本篇记录是我实际配置harbor 1.10.3企业级镜像仓库服务的步骤以及遇到的坑(挫折)，例如harbor使用离线方式配置下载harbor二进制文件遇到墙等。

主机名称	ip地址	操作系统	角色	软件版本	备注
opsharbor-shqs-1	10.99.73.38	CentOS 7.6	docker hub	docker-ce 19.03.11 docker-compose 1.26.0 harbor 1.10.3	测试域名: csrepo.ejuops.com

备注：准备一个域名使用，可以是一个自定义域名不进行解析，直接配置hosts文件，临时验证使用.

一、准备工作

1. 配置系统，以及docker环境

初始化系统以及安装配置docker-ce,参考第1篇记录: CentOS7配置docker环境

2. 配置docker-compose

使用docker-compose版本为1.26.0

curl -L https://github.com/docker/compose/releases/download/1.26.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

3. 验证

# docker -v
	Docker version 19.03.11, build 42e35e61f3

# systemctl status docker
	Active: active (running)

# docker-compose -v
	docker-compose version 1.26.0, build d4451659

二、生成自签证书

因为我这边是测试以及只是在机房内网使用，因此没有申请证书而是采用配置自签证书的方式。
域名：csrepo.ejuops.com

1. 自签证书生成

mkdir certs
cd certs

openssl genrsa -des3 -passout pass:x -out csrepo.ejuops.com.pass.key 2048

openssl rsa -passin pass:x -in csrepo.ejuops.com.pass.key -out csrepo.ejuops.com.key

# 指定-subj参数不用进入交互模式配置
openssl req -new -subj "/C=CN/ST=ShangHai/L=ShangHai/O=eju/OU=ops/CN=csrepo.ejuops.com" -key csrepo.ejuops.com.key -out csrepo.ejuops.com.csr

# 配置证书有效期10年
openssl x509 -req -sha256 -days 3650 -in csrepo.ejuops.com.csr -signkey csrepo.ejuops.com.key -out csrepo.ejuops.com.crt

2. 可以查询证书的一些信息

# 1. 查询证书有效期
# openssl x509 -in csrepo.ejuops.com.crt -noout -dates
notBefore=Jun 19 03:08:01 2020 GMT
notAfter=Jun 17 03:08:01 2030 GMT

# 2. 也可以查询更细的信息
# openssl x509 -in csrepo.ejuops.com.crt -noout -text
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            c6:cb:48:7f:8b:b8:bb:55
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=ShangHai, L=ShangHai, O=eju, OU=ops, CN=csrepo.ejuops.com
        Validity
            Not Before: Jun 19 03:08:01 2020 GMT
            Not After : Jun 17 03:08:01 2030 GMT
        Subject: C=CN, ST=ShangHai, L=ShangHai, O=eju, OU=ops, CN=csrepo.ejuops.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    xxxxxxxx[这里省略]
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         YYYYYYYY[这里省略]

到这里证书准备完成，留在这里备用。

三、安装配置harbor

版本：1.10.3
安装模式：离线安装(offline)

1. 下载harbor二进制文件包

备注：国内github.com下载时断时续的，最好开启断点续传。

# 根据实际网络带宽使用情况限速下载这里限速500KB/s，断点续传
wget -c --limit-rate=500K https://github.com/goharbor/harbor/releases/download/v1.10.3/harbor-offline-installer-v1.10.3.tgz

2. 安装harbor准备

备注：本文档所有的操作都在root下进行的。安装目录为/opt/app/harbor

• 解压到安装目录

~]\# mkdir /opt/app
~]\# tar zxvf harbor-offline-installer-v1.10.3.tgz -C /opt/app/

• 将准备好的证书放到安装目录

mv /root/certs  /opt/app/harbor/

• 目录信息

~]\# cd /opt/app/harbor
~]\# ls -l
drwxr-xr-x 2 root root       127 Jun  19 10:40 certs
-rw-r--r-- 1 root root      3398 Jun  1 19:14 common.sh
-rw-r--r-- 1 root root 670562154 Jun  1 19:15 harbor.v1.10.3.tar.gz
-rw-r--r-- 1 root root      5882 Jun  1 19:14 harbor.yml
-rwxr-xr-x 1 root root      2284 Jun  1 19:14 install.sh
-rw-r--r-- 1 root root     11347 Jun  1 19:14 LICENSE
-rwxr-xr-x 1 root root      1749 Jun  1 19:14 prepare

• 修改配置文件

# vi harbor.yml
# 修改如下信息，其他采用默认
hostname: csrepo.ejuops.com
http:
  port: 80
https:
  port: 443
  certificate: /opt/app/harbor/certs/csrepo.ejuops.com.crt
  private_key: /opt/app/harbor/certs/csrepo.ejuops.com.key
harbor_admin_password: xxxxxxxxx
database:
  password: yyyyyyyyy
data_volume: /data
log:
  level: info
  location: /data/logs/harbor

• 更新信息，并下载镜像

# ./prepare 
prepare base dir is set to /opt/app/harbor
Unable to find image 'goharbor/prepare:v1.10.3' locally
v1.10.3: Pulling from goharbor/prepare
a10779388d4e: Pull complete 
12b8569f46a6: Pull complete 
298230e0a21f: Pull complete 
ea3847dc8bc4: Pull complete 
1b52894b31e5: Pull complete 
ae5b50c2a4c9: Pull complete 
8b279542d577: Pull complete 
Digest: sha256:c757f65f753a73618c7bea3b6e5d0ab281bb60b91ae8d85b20f6d8657d703861
Status: Downloaded newer image for goharbor/prepare:v1.10.3
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

3. 安装harbor

• 安装harbor

~]\# cd /opt/app/harbor
~]\# ./install.sh
[Step 0]: checking if docker is installed ...
[Step 1]: checking docker-compose is installed ...
[Step 2]: loading Harbor images ...
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating redis         ... done
Creating registryctl   ... done
Creating registry      ... done
Creating harbor-db     ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

• 查看一些信息

# docker images
	REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
	goharbor/chartmuseum-photon     v1.10.3             f4cd7df55b4a        2 weeks ago         178MB
	goharbor/harbor-migrator        v1.10.3             524036d275a6        2 weeks ago         357MB
	goharbor/redis-photon           v1.10.3             6562f3c58b71        2 weeks ago         119MB
	goharbor/clair-adapter-photon   v1.10.3             75badac8362a        2 weeks ago         61.2MB
	goharbor/clair-photon           v1.10.3             3c0b186089bd        2 weeks ago         171MB
	goharbor/notary-server-photon   v1.10.3             d0f003ba5021        2 weeks ago         143MB
	goharbor/notary-signer-photon   v1.10.3             304f122f4b13        2 weeks ago         140MB
	goharbor/harbor-registryctl     v1.10.3             2ee885ed2f2d        2 weeks ago         102MB
	goharbor/registry-photon        v1.10.3             f470058ab661        2 weeks ago         84.5MB
	goharbor/nginx-photon           v1.10.3             8c88b468cdb0        2 weeks ago         43.6MB
	goharbor/harbor-log             v1.10.3             b0c9e48d015e        2 weeks ago         82MB
	goharbor/harbor-jobservice      v1.10.3             044b1a98e9ac        2 weeks ago         143MB
	goharbor/harbor-core            v1.10.3             6521cd80151b        2 weeks ago         129MB
	goharbor/harbor-portal          v1.10.3             7d107a7d6eb8        2 weeks ago         51.7MB
	goharbor/harbor-db              v1.10.3             7d10c16f9b2e        2 weeks ago         157MB
	goharbor/prepare                v1.10.3             be46b57a9105        2 weeks ago         163MB

# docker ps |less -S
	CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                   PORTS                        
	06030163f29e        goharbor/harbor-jobservice:v1.10.3    "/harbor/harbor_jobs…"   4 minutes ago       Up 4 minutes (healthy)                                
	74e2240ed6ba        goharbor/nginx-photon:v1.10.3         "nginx -g 'daemon of…"   4 minutes ago       Up 4 minutes (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0
	08a128187c24        goharbor/harbor-core:v1.10.3          "/harbor/harbor_core"    4 minutes ago       Up 4 minutes (healthy)                                
	58143706b0a1        goharbor/harbor-db:v1.10.3            "/docker-entrypoint.…"   4 minutes ago       Up 4 minutes (healthy)   5432/tcp                     
	4ba493895bef        goharbor/registry-photon:v1.10.3      "/home/harbor/entryp…"   4 minutes ago       Up 4 minutes (healthy)   5000/tcp                     
	2b9ac9d7a907        goharbor/redis-photon:v1.10.3         "redis-server /etc/r…"   4 minutes ago       Up 4 minutes (healthy)   6379/tcp                     
	7665b6b28b3e        goharbor/harbor-registryctl:v1.10.3   "/home/harbor/start.…"   4 minutes ago       Up 4 minutes (healthy)                                
	123ed5421dbe        goharbor/harbor-portal:v1.10.3        "nginx -g 'daemon of…"   4 minutes ago       Up 4 minutes (healthy)   8080/tcp                     
	0a1317dc03c5        goharbor/harbor-log:v1.10.3           "/bin/sh -c /usr/loc…"   4 minutes ago       Up 4 minutes (healthy)   127.0.0.1:1514->10514/tcp  


# netstat -pltn
	Active Internet connections (only servers)
	Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
	tcp        0      0 127.0.0.1:1514          0.0.0.0:*               LISTEN      4451/docker-proxy          
	tcp6       0      0 :::80                   :::*                    LISTEN      5200/docker-proxy     
	tcp6       0      0 :::443                  :::*                    LISTEN      5184/docker-proxy

四、使用harbor

1. 浏览器访问harbor UI

https://csrepo.ejuops.com
账户：admin
密码：xxxxxx[实际密码]

可以看到浏览器地址栏显示是不安全的，这是因为使用的是自签名证书导致的

2. 配置docker环境使用该harbor

docker使用自签证书的私有仓库需要配置自签证书根证书到docker环境，使用openssl配置自签证书的方式有所不同，这里使用的方式参考 配置企业级镜像仓库Harbor 生成自签证书相关内容。
docker环境配置使用自签证书的方式有多种。这里列出2中方式

私有仓库地址	自签证书
https://csrepo.ejuops.com	csrepo.ejuops.com.crt

certs]# ll
-rw-r--r-- 1 root root 1212 Jun 19 10:40 csrepo.ejuops.com.crt       #这个就是我们需要的证书文件
-rw-r--r-- 1 root root 1009 Jun 19 10:40 csrepo.ejuops.com.csr
-rw-r--r-- 1 root root 1675 Jun 19 10:40 csrepo.ejuops.com.key
-rw-r--r-- 1 root root 1743 Jun 19 10:40 csrepo.ejuops.com.pass.key

方法一. 配置系统全局使用该自签证书

# 安装ca-certificates
yum -y install ca-certificates

# 证书文件放入 /usr/share/pki/ca-trust-source/anchors
cp csrepo.ejuops.com.crt   /usr/share/pki/ca-trust-source/anchors/

# 更细系统支持证书列表
update-ca-trust

# 查看是否更细追加成功
tail -n23 /etc/ssl/certs/ca-bundle.trust.crt

# csrepo.ejuops.com                   #看到这一行说明更新成功了
-----BEGIN TRUSTED CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  [这里是省略的意思]
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END TRUSTED CERTIFICATE-----

#如果docker守护进程已经启动，需要重启下docker守护进程
systemctl restart docker

# 验证
docker pull hello-world:latest
docker tag hello-world:latest csrepo.ejuops.com/library/hello-world:latest
docker login -u'admin' -p'xxxxx' csrepo.ejuops.com
docker push csrepo.ejuops.com/library/hello-world:latest
docker pull csrepo.ejuops.com/library/hello-world:latest
	latest: Pulling from library/hello-world
	Digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
	Status: Image is up to date for csrepo.ejuops.com/library/hello-world:latest
	csrepo.ejuops.com/library/hello-world:latest

以上说明配置成功了。

方法二. 配置docker服务自己使用自签证书

这里测试的私有仓库域名是： csrepo.ejuops.com

# 配置
DOCKER_REGISTRY=csrepo.ejuops.com 
mkdir  -p /etc/docker/certs.d/${DOCKER_REGISTRY}
cp ${DOCKER_REGISTRY}.crt /etc/docker/certs.d/${DOCKER_REGISTRY}/ca.crt

#验证，这种方式docker守护进程不需要重启
docker pull csrepo.ejuops.com/library/hello-world:latest
	latest: Pulling from library/hello-world
	Digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
	Status: Image is up to date for csrepo.ejuops.com/library/hello-world:latest
	csrepo.ejuops.com/library/hello-world:latest