gitlab on k8s

GitLab部署

helm 安装 postgresql

helmchar: github

  • 安装日志

    [root@master2 ~]#helm install gitlib-db -n gitlab /opt/helm/postgresql/
    NAME: gitlib-db
    LAST DEPLOYED: Mon Apr 24 09:05:58 2023
    NAMESPACE: gitlab
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    CHART NAME: postgresql
    CHART VERSION: 12.1.9
    APP VERSION: 15.1.0
    
    ** Please be patient while the chart is being deployed **
    
    PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:
    
        gitlib-db-postgresql.gitlab.svc.cluster.local - Read/Write connection
    
    To get the password for "postgres" run:
    
        export POSTGRES_PASSWORD=$(kubectl get secret --namespace gitlab gitlib-db-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)
    
    To connect to your database run the following command:
    
        kubectl run gitlib-db-postgresql-client --rm --tty -i --restart='Never' --namespace gitlab --image 10.50.10.185/postgresql/bitnami/postgresql:15.1.0-debian-11-r20 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
          --command -- psql --host gitlib-db-postgresql -U postgres -d postgres -p 5432
    
        > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"
    
    To connect to your database from outside the cluster execute the following commands:
    
        kubectl port-forward --namespace gitlab svc/gitlib-db-postgresql 5432:5432 &
        PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432
    
  • 创建gitlab初始化数据库

在这里插入图片描述

helm 安装 redis 集群

chart: github

  • 安装日志

    [root@master2 ~]#helm install gitlib-redis -n gitlab /opt/helm/redis/
    NAME: gitlib-redis
    LAST DEPLOYED: Mon Apr 24 09:08:19 2023
    NAMESPACE: gitlab
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    CHART NAME: redis
    CHART VERSION: 17.4.3
    APP VERSION: 7.0.8
    
    ** Please be patient while the chart is being deployed **
    
    Redis® can be accessed via port 6379 on the following DNS name from within your cluster:
    
        gitlib-redis.gitlab.svc.cluster.local for read only operations
    
    For read/write operations, first access the Redis® Sentinel cluster, which is available in port 26379 using the same domain name above.
    
    To connect to your Redis® server:
    
    1. Run a Redis® pod that you can use as a client:
    
       kubectl run --namespace gitlab redis-client --restart='Never'  --image 10.50.10.185/redis/bitnami/redis:7.0.8-debian-11-r0 --command -- sleep infinity
    
       Use the following command to attach to the pod:
    
       kubectl exec --tty -i redis-client \
       --namespace gitlab -- bash
    
    2. Connect using the Redis® CLI:
       redis-cli -h gitlib-redis -p 6379 # Read only operations
       redis-cli -h gitlib-redis -p 26379 # Sentinel access
    
    To connect to your database from outside the cluster execute the following commands:
    
        kubectl port-forward --namespace gitlab svc/gitlib-redis 6379:6379 &
        redis-cli -h 127.0.0.1 -p 6379
    

gitlab manifest 部署

参考

官方的chart超级复杂,组件超级多。Gitlab 主要涉及到3个应用:Redis、Postgresql、Gitlab 核心程序,实际上我们只要将这3个应用分别启动起来,然后加上对应的配置就可以很方便的安装 Gitlab 了,我们这里选择使用的镜像不是官方的,而是 Gitlab 容器化中使用非常多的一个第三方镜像:sameersbn/gitlab

  • 登录

    http://chot-gitlab.prod.com:32100/ 账密: root/xxx

  • 使用外部redis 和pg

  • 使用sameersbn的gitlab镜像

  • 使用nginx ingress 向外暴露服务 kubernetes.io/ingress.class: nginx

  • gitlab-prod.yaml

    • 注意需要将pg和redis分别开启30300和30302 nodePort
    • 这一步可优化为域名连接避免开太多的nodeport
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: gitlab-data-pvc
      namespace: gitlab
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 200Gi
      storageClassName: nfs-storage-179sc
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: gitlab
      namespace: gitlab
      labels:
        name: gitlab
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: gitlab
      template:
        metadata:
          name: gitlab
          labels:
            name: gitlab
        spec:
          #nodeName: 192.168.102.22  # *
          containers:
          - name: gitlab
            image: 10.50.10.185/gitlab/sameersbn/gitlab:15.8.0-1
            imagePullPolicy: IfNotPresent
            env:
            - name: TZ
              value: Asia/Shanghai
            - name: GITLAB_TIMEZONE
              value: Beijing
            - name: GITLAB_SECRETS_DB_KEY_BASE
              value: long-and-random-alpha-numeric-string  # *
            - name: GITLAB_SECRETS_SECRET_KEY_BASE
              value: long-and-random-alpha-numeric-string  # *
            - name: GITLAB_SECRETS_OTP_KEY_BASE
              value: long-and-random-alpha-numeric-string  # *
            - name: GITLAB_ROOT_PASSWORD
              value: admin123  # *
            - name: GITLAB_ROOT_EMAIL
              value: ninesun@126.com  # *
            - name: GITLAB_HOST
              value: chot-gitlab.prod.com  # *
            - name: GITLAB_PORT
              value: "30400"
            - name: GITLAB_SSH_HOST
              value: k8s-22.host.com  # *
            - name: GITLAB_SSH_PORT
              value: "30401"
            - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
              value: "true"
            - name: GITLAB_NOTIFY_PUSHER
              value: "false"
            - name: GITLAB_BACKUP_SCHEDULE
              value: daily
            - name: GITLAB_BACKUP_TIME
              value: 01:00
            - name: DB_TYPE
              value: postgres
            - name: DB_HOST
              value: gitlib-db-postgresql-hl #headless svc name: gitlib-db-postgresql-hl
            - name: DB_PORT
              value: "5432"
            - name: DB_USER
              value: postgres
            - name: DB_PASS
              value: "postgres"  # *
            - name: DB_NAME
              value: gitlab_production # 这一步如果需要重新创建数据库,就需要单独建立。否则就把gitlab 的数据存储在默认数据库postgres
            - name: REDIS_HOST
              value: gitlib-redis-headless # headless svc name: gitlib-redis-headless
            - name: REDIS_PORT
              value: "6379" # 默认端口是6379
            ports:
            - name: http
              containerPort: 80
            - name: ssh
              containerPort: 22
            volumeMounts:
            - mountPath: /home/git/data
              name: data
            livenessProbe:
              httpGet:
                path: /
                port: 80
              initialDelaySeconds: 180
              timeoutSeconds: 5
            readinessProbe:
              httpGet:
                path: /
                port: 80
              initialDelaySeconds: 25
              timeoutSeconds: 1
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: gitlab-data-pvc
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: gitlab
      namespace: gitlab
      labels:
        name: gitlab
    spec:
      ports:
        - name: http
          port: 80
          targetPort: http
          nodePort: 30400
        - name: ssh
          port: 22
          targetPort: ssh
          nodePort: 30401
      type: NodePort
      selector:
        name: gitlab
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: gitlab
      namespace: gitlab
      annotations:
        kubernetes.io/ingress.class: nginx
    spec:
      rules:
        - host: chot-gitlab.prod.com
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: gitlab
                    port:
                      number: 80
    
    • gitlab需要启动多久?

      启动大约需要4mins,期间可能会导致readlines probe检查失败。可以将initialDelaySeconds 调大一些

在这里插入图片描述

gitlab如何上传项目?

  • 第一步: 添加需要推送项目的服务器的公钥

    key的内容就是id_rsa.pub的内容

    [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NlPzyq3P-1683179705925)(https://s3-us-west-2.amazonaws.com/secure.notion-static.com/19224c7f-f2c7-4569-ace6-28b0a579233b/Untitled.png)]

  • 第二步: git 上传项目

    # 添加remote url
    git remote add origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git
    
    # 查看remote url
    git remote -v
    origin  https://github.com/myysophia/gitlab-ci-k8s-demo.git (fetch)
    origin  https://github.com/myysophia/gitlab-ci-k8s-demo.git (push)
    origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git (fetch)
    origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git (push)
    
    # 推送代码
    git push -u origin1 --all
    

gitlab监控metrics

这部分可以接入外部的grafana监控面板进行监控

http://chot-gitlab.prod.com:30400/-/metrics?token=zuqjYZFKMof22VkTRLek

gitlab runner helm 部署

  • helm 部署

    chart github:

  • 全局runner和项目runner

    官方建议自建的实例使用指定的runner,也就是每个项目一个单独的runner

    两者如何选择?

  • 全局runner token

  • 项目runner token

helm部署问题

域名解析问题

  • 参考链接

https://todoit.tech/k8s/gitlab-runner/

https://blog.csdn.net/boling_cavalry/article/details/106991576

  • 报错处理: Incorrect Usage: flag provided but not defined: -template-config

版本问题

GitLab 社区版 15.8.0 需要使用对应的runner镜像(**https://docs.gitlab.com/runner/)****

  • docker push 10.50.10.185/gitlab/registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.8.0

  • 如何查看gitlab版本: http://chot-gitlab.prod.com:32100/help

  • config.toml 配置 如何覆盖config.template.toml

    非root用户→ 容器中~/.gitlab-runner/config.yaml

    concurrent = 10
    check_interval = 30
    log_level = "info"
    shutdown_timeout = 0
    
    [session_server]
      session_timeout = 1800
    
    [[runners]]
      name = "chot-gitlab-runner-minio-gitlab-runner-55564b6469-k2d6h"
      url = "http://chot-gitlab.prod.com:32100/"
      id = 8
      token = "vbR7MMTPKSL7dyPALsUN"
      token_obtained_at = 2023-02-17T03:24:19Z
      token_expires_at = 0001-01-01T00:00:00Z
      executor = "kubernetes"
      [runners.custom_build_dir]
      [runners.cache]
        Type = "s3"
        Shared = true
        MaxUploadedArchiveSize = 0
        [runners.cache.s3]
          ServerAddress = "chot-minio-web.prod.com:32100"
          AccessKey = "IwA5ttRQsZlKkkQV"
          SecretKey = "C07BrPYktE997bMcWUdcHyXQPVPr3mSJ"
          BucketName = "gitlab"
        [runners.cache.gcs]
        [runners.cache.azure]
      [runners.kubernetes]
        host = ""
        bearer_token_overwrite_allowed = false
        image = "10.50.10.185/gitlab/ubuntu:16.04"
        namespace = "gitlab"
        namespace_overwrite_allowed = ""
        pull_policy = ["if-not-present"]
        node_selector_overwrite_allowed = ""
        helper_image = "10.50.10.185/gitlab/ubuntu:16.04"
        pod_labels_overwrite_allowed = ""
        service_account_overwrite_allowed = ""
        pod_annotations_overwrite_allowed = ""
        [runners.kubernetes.affinity]
        [runners.kubernetes.pod_security_context]
        [runners.kubernetes.init_permissions_container_security_context]
          [runners.kubernetes.init_permissions_container_security_context.capabilities]
        [runners.kubernetes.build_container_security_context]
          [runners.kubernetes.build_container_security_context.capabilities]
        [runners.kubernetes.helper_container_security_context]
          [runners.kubernetes.helper_container_security_context.capabilities]
        [runners.kubernetes.service_container_security_context]
          [runners.kubernetes.service_container_security_context.capabilities]
        [runners.kubernetes.volumes]
        [runners.kubernetes.dns_config]
        [runners.kubernetes.container_lifecycle]
    
  • 跑通一个gitlab cicd

    https://github.com/yangshun2005/gitlab-cicd

  • gitlab exector 报错处理

    • 使用k8s exector 无法拉取镜像

      Running with gitlab-runner 15.8.0 (12335144)
        on chot-gitlab-runner-minio-gitlab-runner-6fbf87f59b-j6bhq S7qPFnrs, system ID: r_vB5NUhtcRQ1R
      Preparing the "kubernetes" executor
      00:00
      Using Kubernetes namespace: gitlab
      Using Kubernetes executor with image ubuntu:16.04 ...
      Using attach strategy to execute scripts...
      Preparing environment
      00:03
      Waiting for pod gitlab/runner-s7qpfnrs-project-1-concurrent-0bc7zl to be running, status is Pending
      WARNING: Failed to pull image with policy "IfNotPresent": image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: no such host
      ERROR: Job failed: prepare environment: waiting for pod running: pulling image "ubuntu:16.04": image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: no such host. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
      

      修改runner values的文件

      image:
          registry: 10.50.10.185/gitlab
          image: ubuntu
          tag: 16.04
      
      Waiting for pod gitlab/runner-gm-nhepv-project-1-concurrent-057xgd to be running, status is Pending
      ERROR: Job failed: prepare environment: waiting for pod running: image pull failed: Failed to apply default image tag "map[image:ubuntu registry:10.50.10.185/gitlab tag:16.04]": couldn't parse image reference "map[image:ubuntu registry:10.50.10.185/gitlab tag:16.04]": invalid reference format. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
      

      exector不支持这种image的方式

      修改为

      image: 10.50.10.185/gitlab/ubuntu:16.04

      镜像拉取成功后会启动两个容器一个helper 一个 build镜像

      如果runner分配给exector的工作1小时没有完成,这些容器就退出了。

      error log

      • helper 和 build镜像启动后,容器启动报错
      /bin/bash: line 1: gitlab-runner-build: command not found
      

      gitlab 流水线报错如下:

      OCI runtime exec failed: exec failed: unable to start container process: exec: "gitlab-runner-helper": executable file not found in $PATH: unknown
      
      # 从pod yaml文件中找出 其中helper 容器中执行这段脚本
      if [ -x /usr/local/bin/bash ]; then
      exec /usr/local/bin/bash 
      elif [ -x /usr/bin/bash ]; then
      exec /usr/bin/bash 
      elif [ -x /bin/bash ]; then
      exec /bin/bash 
      elif [ -x /usr/local/bin/sh ]; then
      exec /usr/local/bin/sh 
      elif [ -x /usr/bin/sh ]; then
      exec /usr/bin/sh 
      elif [ -x /bin/sh ]; then
      exec /bin/sh 
      elif [ -x /busybox/sh ]; then
      exec /busybox/sh 
      else
      echo shell not found
      exit 1
      fi
      

      查阅官方文档后发现这个helper 镜像还是个专有镜像,下载gitlab runner对应版本的helper, 这个helper镜像就是那个具体干活的,gitlab 的 .gitlab-ci.yml 根据配置的时间间隔把活给gitlab runner。

      gitlab runner则是让helper images 去处理。所以刚开始的时候helper的image 用ubuntu是不对的。

      具体理解参考Override the helper image部分:

      Advanced configuration | GitLab

      bitnami/gitlab-runner-helper:15.8.0

      Running with gitlab-runner 15.8.0 (12335144)
        on chot-gitlab-runner-minio-gitlab-runner-75f87cfdbf-d89z6 fhyNaFUz, system ID: r_MDgwGq2YmKGA
      Preparing the "kubernetes" executor
      00:00
      Using Kubernetes namespace: gitlab
      Using Kubernetes executor with image 10.50.10.185/gitlab/ubuntu:16.04 ...
      Using attach strategy to execute scripts...
      Preparing environment
      Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
      Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
      	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
      	ContainersNotReady: "containers with unready status: [build helper]"
      	ContainersNotReady: "containers with unready status: [build helper]"
      Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
      	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
      	ContainersNotReady: "containers with unready status: [build helper]"
      	ContainersNotReady: "containers with unready status: [build helper]"
      Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending
      	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
      	ContainersNotReady: "containers with unready status: [build helper]"
      	ContainersNotReady: "containers with unready status: [build helper]"
      /bin/bash: line 1: gitlab-runner-build: command not found
      

      实在没办法,提个issue吧

      April 21, 2023

      官方更新了gitlab-runner-helper镜像版本

k8s中gitlab exector架构图

一图胜千言。

在这里插入图片描述

The Kubernetes executor for GitLab Runner | GitLab

what is Gitlab Runner Helper?

helm部署gitlab runner的时候有一个help image。

Gitlab Runner Helper 是与 Gitlab Runner 一起使用的辅助容器。 Gitlab Runner 允许运行 CI/CD 作业并将结果发送回 Gitlab。

gitlab如何连接上k8s? KAS Kubernetes agent server

gitlab CI

gitlab CI template

lib/gitlab/ci/templates · master · GitLab.org / GitLab · GitLab

gitlab CI Demo

上来不应该直接尝试on k8s,为啥不从最简单的开始呢?

使用 docker部署 gitlab-runner ,注册一个exector 为shell的runner

  • 先来个简单的.gitlab-ci.yaml

    
    before_script:
      - echo "Before script section122333"
      - echo "For example you might run an update here or install a build dependency"
      - echo "Or perhaps you might print out some debugging details"
    
    after_script:
      - echo "After script section"
      - echo "For example you might do some cleanup here"
    
    build1:
      stage: build 
      script:
        - echo "Do your build heresd"
    
    test1:
      stage: test
      script:
        - echo "Do a test here"
        - echo "For example run a test suite"
    
    test2:
      stage: test
      script:
        - echo "Do another parallel test here"
        - echo "For example run a lint test"
    
    deploy1:
      stage: deploy
      script:
        - echo "Do your deploy here"
      environment: production
    

第一步: 1. 建立 Docker Volume

目前runner部署在10.50.10.36 理论上哪台有docker环境的都可以的.

$ docker volume create gitlab-runner-config

第二步: 使用创建的卷启动GitLab Runner容器:

-env TZ=CST

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v gitlab-runner-config:/etc/gitlab-runner \
		--add-host=chot-gitlab.prod.com:10.50.10.33 \
    10.50.10.185/gitlab/gitlab/gitlab-runner:v15.8.0

第三步: register 一个runner,exector 选择shell

root@27172e22cf90:/# gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=54 revision=12335144 version=15.8.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
http://chot-gitlab.prod.com:30400/
Enter the registration token:
GR1348941BbkUVr8B1UumMfNx4LrL
Enter a description for the runner:
[27172e22cf90]:
Enter tags for the runner (comma-separated):

Enter optional maintenance note for the runner:

WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
Registering runner... succeeded                     runner=GR1348941BbkUVr8B
Enter an executor: docker-ssh, virtualbox, docker-ssh+machine, instance, ssh, docker+machine, kubernetes, custom, docker, parallels, shell:
docker
Enter the default Docker image (for example, ruby:2.7):
10.50.10.185/gitlab/ubuntu:16.04
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

或者一步到位:

gitlab-runner register --url http://chot-gitlab.prod.com:30400/ --registration-tokenGR1348941BbkUVr8B1UumMfNx4LrL --executor shell

第四步 查看CI结果

  • 不同exector的runner配置

    gitlab-runner register时可以指定不同的exector. 可以打不同的tag,让不同的项目或编译条件使用不同的exector.

    gitlab-runner list
    Runtime platform                                    arch=amd64 os=linux pid=27160 revision=12335144 version=15.8.0
    Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
    meta-162                                            Executor=docker Token=_mFCm2xiCnFd8rKaAKFg URL=http://chot-gitlab.prod.com:30400/
    meta-162                                            Executor=shell Token=_yvc4o5ycSACmtyTvBVK URL=http://chot-gitlab.prod.com:30400/
    
  • CI使用minio作为cache

    例如在java项目编译的时候会有很多依赖包需要下载,如果每次都从网络拉取不稳定,这时候如果把依赖包缓存起来,下次构建项目就很快.

    [runners.cache]
            Type = "s3"
            Shared = true
            [runners.cache.s3]
              AccessKey = "IwA5ttRQsZlKkkQV"
              SecretKey = "C07BrPYktE997bMcWUdcHyXQPVPr3mSJ"
              BucketName = "gitlab"
              ServerAddress = "chot-minio-api.prod.com:32100"
    

gitlab 和gitlab-runner的关系

GitLab CI 之 Runner 的 Executor 該如何選擇?

runner和exector的关系

runner并不是实际干活的,runner 可以指定特定的exector干活,在gitlab中有不同的exector,目前有这几种exector:

  • docker-ssh
  • virtualbox
  • docker-ssh+machine
  • instance
  • ssh
  • docker+machine
  • kubernetes
  • custom
  • docker
  • parallels
  • shell

可以结合自己的技术栈进行选择,目前我们常使用的是shell 和 docker、k8s

gitlab环境变量

这块都是gitlab预设定的一些环境变量,更多请打开CI_DEBUG_TRACE mode。

例如下面三个环境变量分别是当前gitlab实例名、绝对路径和相对路径。

$ echo "${CI_PROJECT_NAMESPACE}"
gitlab-instance-f410c318
$ echo "${CI_PROJECT_DIR}"
/home/gitlab-runner/builds/hs9MHCAM/0/gitlab-instance-f410c318/gitlab-ci-k8s-demo
$ echo "${CI_PROJECT_PATH}"
gitlab-instance-f410c318/gitlab-ci-k8s-demo

参考

用 GitLab CI 进行持续集成

Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐