1、搭建私有仓库

生成证书

mkdir /certs

openssl req -newkey rsa:4096 -nodes -sha256 -keyout /certs/domain.key -x509 -days 365 -out /certs/domain.crt

随便填，到Common Name的时候输入预设的域名：re.bcdgptv.com.cn

如果除了预设的域名以外，还想用其它的方式访问私有仓库比如192.168.137.1/nginx:1.7.9

这个时候需要修改/etc/pki/tls/openssl.cnf,在此节点添加如下信息，再生成证书，必须在生成证书前添加

[ v3_ca ]
subjectAltName = IP:192.168.137.1

启动镜像，本列中工作目录为/

docker run -d --restart=always --name re.bcdgptv.com.cn -v /certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -p 443:443 registry:2

注意：K8S的所有NODE做好与re.bcdgptv.com.cn域名与IP的映射，IP就是运行仓库镜像所在的主机IP

docker hub下载一个镜像，再上传到私有镜像服务器测试

docker pull redis

[root@k8s1 certs]# docker images
REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE
redis                                                            latest              a4fe14ff1981        7 days ago          95MB

标记镜像

docker tag a4fe14ff1981 re.bcdgptv.com.cn/redis:v1

此时推送证书仍然失败：

[root@k8s1 /]# docker push re.bcdgptv.com.cn/redis:v1
The push refers to repository [re.bcdgptv.com.cn/redis]
Get https://re.bcdgptv.com.cn/v2/: x509: certificate signed by unknown authority

如果为这个提示的话，拷贝证书至指定的目录完成信任：

[root@k8s1 /]# mkdir -p /etc/docker/certs.d/re.bcdgptv.com.cn

[root@k8s1 /]# cp /certs/domain.crt /etc/docker/certs.d/re.bcdgptv.com.cn

此时可以正常上传镜像至私有仓库：

[root@k8s1 /]# docker push re.bcdgptv.com.cn/redis:v1                         
The push refers to repository [re.bcdgptv.com.cn/redis]
a5e32065f40a: Pushed 
367796b84542: Pushed 
b8305db6b467: Pushed 
03eafa792876: Pushed 
f99f83132c0a: Pushed 
6270adb5794c: Pushed 
v1: digest: sha256:82ac0e8f4f2cb5db18714b726febea6de9666ad9d9ad6f62f433f073bc3048f0 size: 1572

 

2、K8S从私有仓库拉取镜像完成部署

创建secret：

[root@k8s3 ~]# kubectl create secret docker-registry registry-key --docker-server=re.bcdgptv.com.cn --docker-username=test1 --docker-password=yourpassword --docker-email=bcdgptv@21cn.com
secret "registry-key" created

此处的用户名密码与DOCKER私有仓库建立的用户密码对应一致，registry-key与yaml调用的imagePullSecrets名称一致

部署POD

[root@k8s1 75yml]# cat redistest.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testapi
spec:
  selector:
    matchLabels:
      app: testapi
  replicas: 2
  template:
    metadata:
      labels:
        app: testapi
    spec:
      imagePullSecrets:
      - name: registry-key
      containers:
      - name: testapi
        image: re.bcdgptv.com.cn/redis:v1
        ports: 
        - containerPort: 6379

 

[root@k8s1 75yml]# kubectl create -f redistest.yml 
deployment.apps "testapi" created

拉取成功，两个POD都起来了

[root@k8s1 75yml]# kubectl get all
NAME                           READY     STATUS    RESTARTS   AGE
pod/testapi-7df6d4747b-hr82l   1/1       Running   0          49m
pod/testapi-7df6d4747b-xhrqd   1/1       Running   0          34m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.254.0.1       <none>        443/TCP   41d
service/ngweb        ClusterIP   10.254.145.174   <none>        80/TCP    17d

NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/testapi   2         2         2            2           49m

NAME                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/testapi-7df6d4747b   2         2         2         49m

详细信息：

[root@k8s1 75yml]# kubectl describe pod testapi-7df6d4747b-hr82l
Name:           testapi-7df6d4747b-hr82l
Namespace:      default
Node:           k8s1/192.168.137.71
Start Time:     Wed, 15 May 2019 19:55:21 +0800
Labels:         app=testapi
                pod-template-hash=3892803036
Annotations:    <none>
Status:         Running
IP:             172.30.8.3
Controlled By:  ReplicaSet/testapi-7df6d4747b
Containers:
  testapi:
    Container ID:   docker://5fb745675db23f0cb347084a1c056640672ad553568aabf5fae8cd3921128fc4
    Image:          re.bcdgptv.com.cn/redis:v1
    Image ID:       docker-pullable://redis@sha256:2dfa6432744659268d001d16c39f7be52ee73ef7e1001ff80643f0f7bdee117e
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 15 May 2019 19:56:49 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7xs9j (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  default-token-7xs9j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7xs9j
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From               Message
  ----    ------                 ----  ----               -------
  Normal  Scheduled              51m   default-scheduler  Successfully assigned testapi-7df6d4747b-hr82l to k8s1
  Normal  SuccessfulMountVolume  51m   kubelet, k8s1      MountVolume.SetUp succeeded for volume "default-token-7xs9j"
  Normal  Pulled                 49m   kubelet, k8s1      Container image "re.bcdgptv.com.cn/redis:v1" already present on machine
  Normal  Created                49m   kubelet, k8s1      Created container
  Normal  Started                49m   kubelet, k8s1      Started container
[root@k8s1 75yml]# kubectl describe pod testapi-7df6d4747b-xhrqd
Name:           testapi-7df6d4747b-xhrqd
Namespace:      default
Node:           k8s3/192.168.137.73
Start Time:     Wed, 15 May 2019 20:10:23 +0800
Labels:         app=testapi
                pod-template-hash=3892803036
Annotations:    <none>
Status:         Running
IP:             172.30.34.2
Controlled By:  ReplicaSet/testapi-7df6d4747b
Containers:
  testapi:
    Container ID:   docker://dbaa955cb26a5bf8fba02898b04389c288adbdca24da7c0b846f459413b9b0a0
    Image:          re.bcdgptv.com.cn/redis:v1
    Image ID:       docker-pullable://re.bcdgptv.com.cn/redis@sha256:82ac0e8f4f2cb5db18714b726febea6de9666ad9d9ad6f62f433f073bc3048f0
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 15 May 2019 20:14:22 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7xs9j (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  default-token-7xs9j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7xs9j
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                From               Message
  ----     ------                 ----               ----               -------
  Normal   Scheduled              36m                default-scheduler  Successfully assigned testapi-7df6d4747b-xhrqd to k8s3
  Normal   SuccessfulMountVolume  36m                kubelet, k8s3      MountVolume.SetUp succeeded for volume "default-token-7xs9j"
  Normal   BackOff                34m (x6 over 35m)  kubelet, k8s3      Back-off pulling image "re.bcdgptv.com.cn/redis:v1"
  Warning  Failed                 34m (x6 over 35m)  kubelet, k8s3      Error: ImagePullBackOff
  Normal   Pulling                34m (x4 over 35m)  kubelet, k8s3      pulling image "re.bcdgptv.com.cn/redis:v1"
  Warning  Failed                 34m (x4 over 35m)  kubelet, k8s3      Failed to pull image "re.bcdgptv.com.cn/redis:v1": rpc error: code = Unknown desc = Error response from daemon: Get https://re.bcdgptv.com.cn/v2/: dial tcp: lookup re.bcdgptv.com.cn on 180.76.76.76:53: no such host
  Warning  Failed                 34m (x4 over 35m)  kubelet, k8s3      Error: ErrImagePull

 

 

如果DOCKER私有仓库设置为用户名密码登陆,可以在前面配置的基础上再作如下调整,实验当中使用了test.bcdgptv.com.cn这个测试域名,与上面略有不同

生成证书,拷贝至客户端的步骤一样

其实就是增加生成密码文件,并在容器启动的时候指定路径的步骤

[root@v73 ~]# docker run --entrypoint htpasswd registry -Bbn bcdgptv 123456 > /certs/htpasswd

[root@v73 ~]# cat /certs/htpasswd 
bcdgptv:$2y$05$VCTEhLRj3ykkFaNXxHW9HOGPz5yg0Kdf9ytwFHSKycsBwoEfPVwbK

启动容器,注意相关的参数

[root@v73 ~]# docker run -d --restart=always --name test.bcdgptv.com.cn -v /certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/certs/htpasswd -p 443:443 registry
eb6cea2e98ce35cfd999003ffc9d7f2719a7b26acf56d339329189ac736ee90e

登陆私有仓库

[root@v73 ~]# docker login test.bcdgptv.com.cn                            
Username: bcdgptv
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

此时上传下载镜像必须要登陆认证以后才能完成不然提示权限拒绝

[root@v73 ~]# docker logout test.bcdgptv.com.cn  
Removing login credentials for test.bcdgptv.com.cn

[root@v73 ~]# docker tag f32a97de94e1 test.bcdgptv.com.cn/registry

[root@v73 ~]# docker push test.bcdgptv.com.cn/registry
The push refers to repository [test.bcdgptv.com.cn/registry]
73d61bf022fd: Preparing 
5bbc5831d696: Preparing 
d5974ddb5a45: Preparing 
f641ef7a37ad: Preparing 
d9ff549177a9: Preparing 
no basic auth credentials
[root@v73 ~]# docker login test.bcdgptv.com.cn                          
Username: bcdgptv
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@v73 ~]# docker push test.bcdgptv.com.cn/registry
The push refers to repository [test.bcdgptv.com.cn/registry]
73d61bf022fd: Pushed 
5bbc5831d696: Pushed 
d5974ddb5a45: Pushed 
f641ef7a37ad: Pushed 
d9ff549177a9: Pushed 
latest: digest: sha256:b1165286043f2745f45ea637873d61939bff6d9a59f76539d6228abf79f87774 size: 1363

 

[root@v73 ~]# docker pull test.bcdgptv.com.cn/registry
Using default tag: latest
Error response from daemon: Get https://test.bcdgptv.com.cn/v2/registry/manifests/latest: no basic auth credentials
[root@v73 ~]# docker login test.bcdgptv.com.cn                          
Username: bcdgptv
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@v73 ~]# docker pull test.bcdgptv.com.cn/registry
Using default tag: latest
latest: Pulling from registry
Digest: sha256:b1165286043f2745f45ea637873d61939bff6d9a59f76539d6228abf79f87774
Status: Image is up to date for test.bcdgptv.com.cn/registry:latest