在k8s上面搭建apisix网关

本地安装apisix使用yum安装yum install -y https://github.com/apache/incubator-apisix/releases/download/1.1/apisix-1.1-0.el7.noarch.rpm本地安装dashboard参见：https://github.com/apache/incubator-apisix/tree/v1.1 ...

月夜楓

7851人浏览 · 2020-03-27 16:54:03

月夜楓 · 2020-03-27 16:54:03 发布

本地安装apisix

使用yum安装

yum install -y https://github.com/apache/incubator-apisix/releases/download/1.1/apisix-1.1-0.el7.noarch.rpm

本地安装dashboard

参见：https://github.com/apache/incubator-apisix/tree/v1.1 – dashboard安装部分
或者 https://blog.csdn.net/cyxinda/article/details/105099571–dashboard安装部分

工作目录结构：

如下是最终生成的工作目录

workspace
|-- docker
|   |-- apisix
|   |   |-- apisix.dockerfile  #apisix的dockerfile
|   |   |-- config.yaml       #apisix的配置文件
|   |   |-- dashboard         #dashboard安装目录
|   |-- etcd
|   |   |-- etcd.conf.yml     #etcd的配置文件
|   |   |-- etcd.dockerfile   #etcd的dockerfile
|   |-- uuid                  #测试项目【uuid生成器】,普通的jar项目
|       |-- uuid-service-0.0.1-SNAPSHOT.jar
|       |-- uuid.dockerfile
|   |-- incubator-apisix-docker-master  #制作第一层docker镜像的目录
|-- k8s
    |-- apisix.yaml          #apisix的k8s编排文件
    |-- etcd.yaml            #etcd的k8s编排文件
    |-- uuid.yaml            #测试项目【uuid生成器】的k8s编排文件

制作apisix的docker镜像

下载incubator-apisix-docker到workspace/docker目录中
下载地址: https://github.com/apache/incubator-apisix-docker
build docker镜像：
在本地目录workspace/docker/incubator-apisix-docker-master/ 下，执行如下build命令：
```
docker build -t harbor.aipo.lenovo.com/apisix/apisix:master-alpine -f alpine/Dockerfile alpine
docker push harbor.aipo.lenovo.com/apisix/apisix:master-alpine
```
生成apisix的第一层镜像，并保存到了本地的harbor服务器上面

制作etcd的docker镜像

在目录workspace/docker/etcd/ 中，复制etcd配置文件到当前目录

cp ../incubator-apisix-docker-master/example/etcd_conf/etcd.conf.yml .

创建etcd的dockerfile：etcd.dockerfile，内容如下：

FROM bitnami/etcd:3.3.13-r80
COPY etcd.conf.yml /opt/bitnami/etcd/conf/etcd.conf.yml
EXPOSE 2379 2380

生成docker镜像

docker build -t harbor.aipo.lenovo.com/apisix/etcd:3.3.13 -f etcd.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/etcd:3.3.13

生成etcd镜像，并将镜像保存到harbor上面

获取集群dns信息

在k8s集群的终端，获取集群域名
随便找到一个部署在k8s上面的pod，进入到容器内部：
```
kubectl -n kube-system exec -it pod/metrics-server-749c9798c6-zkg2m -- /bin/sh
```
在容器内部，查看dns配置：
```
[root@k8s-1 bin]# kubectl -n kube-system exec -it pod/metrics-server-749c9798c6-zkg2m -- /bin/sh
/ # cat /etc/resolv.conf
nameserver 10.1.0.10
search kube-system.svc.cluster.local svc.cluster.local cluster.local openstacklocal
options ndots:5
/ #
```
其中kube-system.svc.cluster.local的结构是

<service_name>.<namespace>.svc.<domain>
每个部分的字段意思：

service_name: 服务名称，就是定义 service 的时候取的名字
namespace：service 所在 namespace 的名字
domain：提供的域名后缀，比如默认的 cluster.local
可以参考：https://cizixs.com/2017/04/11/kubernetes-intro-kube-dns/

domain这个字段就是我们要找的：在本集群中的值就是：cluster.local
nameserver 10.1.0.10也可以知道dns的域名服务的ClusterIP为：10.1.0.10

获取集群dns的ClusterIP
check一下步骤一中，找到的ClusterIP是否正确
执行如下命令：

[root@k8s-1 bin]# kubectl -n kube-system get svc
NAME                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
etcd                               ClusterIP   None          <none>        2379/TCP                 133d
kube-controller-manager-headless   ClusterIP   None          <none>        10252/TCP                133d
kube-dns                           ClusterIP   10.1.0.10     <none>        53/UDP,53/TCP,9153/TCP   191d
kube-scheduler-headless            ClusterIP   None          <none>        10251/TCP                133d
kubelet                            ClusterIP   None          <none>        10250/TCP                133d
kubernetes-dashboard               NodePort    10.1.213.19   <none>        443:30003/TCP            157d
metrics-server                     ClusterIP   10.1.185.71   <none>        443/TCP                  133d
tiller-deploy                      ClusterIP   10.1.55.173   <none>        44134/TCP                155d

kube-dns服务的ip就是k8s的DNS服务的ClusterIP，可以验证步骤1找到的ip与此步骤的ip是一致的：10.1.0.10

重新制作镜像

目的有两个：

加入dashboard
便于修改apisix的配置文件

在workspace/docker/apisix/目录下

拷贝incubator-apisix-docker-master中的配置文件到当前目录

cp ../incubator-apisix-docker-master/example/apisix_conf/config.yaml .

拷贝步骤【本地安装dashboard】中安装好的dashboard
```
	cp -r /usr/local/apisix/dashboard .
```

修改apisix的配置文件

apisix:
  node_listen: 9080              # APISIX listening port
  node_ssl_listen: 9443
  enable_heartbeat: true
  enable_admin: true
  enable_debug: false
  enable_ipv6: true
  config_center: etcd             # etcd: use etcd to store the config value
                                  # yaml: fetch the config value from local yaml file `/your_path/conf/apisix.yaml`
  # allow_admin:                  # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
  #   - 127.0.0.0/24              # If we don't set any IP list, then any IP access is allowed by default.
  #   - "::/64"
  # port_admin: 9180              # use a separate port
  router:
    http: 'radixtree_uri'         # radixtree_uri: match route by uri(base on radixtree)
                                  # radixtree_host_uri: match route by host + uri(base on radixtree)
                                  # r3_uri: match route by uri(base on r3)
                                  # r3_host_uri: match route by host + uri(base on r3)

    ssl: 'radixtree_sni'          # r3_sni: match route by SNI(base on r3)
                                  # radixtree_sni: match route by SNI(base on radixtree)
  # stream_proxy:                 # TCP/UDP proxy
  #   tcp:                        # TCP proxy port list
  #     - 9100
  #     - 9101
  #   udp:                        # UDP proxy port list
  #     - 9200
  #     - 9211
  dns_resolver:                   # default DNS resolver, with disable IPv6 and enable local DNS
    - 10.1.0.10

  dns_resolver_valid: 30          # valid time for dns result 30 seconds

  ssl:
    enable: true
    enable_http2: true
    listen_port: 9443
    ssl_protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
    ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "logs/error.log"
  error_log_level: "warn"         # warn,error
  event:
    worker_connections: 10620
  http:
    access_log: "logs/access.log"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Real-IP"     # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 'unix:'

etcd:
  host: "http://etcd-service.saas.svc.cluster.local:12379"   # etcd address
  prefix: "/apisix"               # apisix configurations prefix
  timeout: 1                      # 1 seconds

plugins:                          # plugin list
  - example-plugin
  - limit-req
  - limit-count
  - limit-conn
  - key-auth
  - prometheus
  - node-status
  - jwt-auth
  - zipkin
  - ip-restriction
  - grpc-transcode
  - serverless-pre-function
  - serverless-post-function
  - openid-connect
  - proxy-rewrite

stream_plugins:
  - mqtt-proxy

修改两项：

apisix.dns_resolver的值为10.1.0.10（前面获取到的k8s DNS的ClusterIP）
etcd.host的值为http://etcd-service.saas.svc.cluster.local :12379
1. etcd-service是k8s启动的etcd的service的名称，后文中会创建
2. saas是etcd所属的命名空间
3. cluster.local是前文中获取的k8s集群的域名
4. 12379是etcd的service的port端口

备注1：apisix依赖于nginx，而nginx的域名解析服务是需要独立配置的，nginx域名解析不会自动依赖系统设置的域名解析服务器
备注2：k8s的域名解析服务，只能够解析本域下的命名，即以cluster.local结尾的命名

创建apisix的dockerfile：apisix.dockerfile，内容如下：

FROM harbor.aipo.lenovo.com/apisix/apisix:master-alpine
COPY config.yaml /usr/local/apisix/conf/config.yaml
COPY dashboard /usr/local/apisix/dashboard
EXPOSE 9080 9443

buid镜像

docker build -t harbor.aipo.lenovo.com/apisix/apisix:v1.1 -f apisix.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/apisix:v1.1

创建k8s的编排文件

etcd编排文件

apiVersion: v1
kind: Namespace
metadata:
  name: saas
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd-server
  namespace: saas #命名空间
  labels:
    app: etcd-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: etcd-server
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: etcd-server
    spec:
      containers:
      - name: etcd-server
        image: harbor.aipo.lenovo.com/apisix/etcd:3.3.13
        imagePullPolicy:  Always
        ports:
        - name: http
          containerPort: 2379 #容器对外暴露的端口
        - name: peer 
          containerPort: 2380
---
apiVersion: v1
kind: Service
metadata:
  name: etcd-service
  namespace: saas
spec:
  type: ClusterIP
  selector:
    app: etcd-server
  ports:
  - name: http
    port: 12379  #etcd的service暴露给其他服务的端口
    targetPort: 2379 #指向容器暴露的端口
#通过template.metadata.labels字段为即将新建的Pod附加Label。在上面的例子中，新建了一个名称为nginx的Pod，它拥有一个键值对为app:nginx的Label。
#通过spec.selector字段来指定这个RC管理哪些Pod。在上面的例子中，新建的RC会管理所有拥有app:nginxLabel的Pod。这样的spec.selector在Kubernetes中被称作Label Selector。

namespace为saas

apisix的编排文件：

apiVersion: v1
kind: Namespace
metadata:
  name: saas
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix-server
  namespace: saas
  labels:
    app: apisix-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apisix-server
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: apisix-server
    spec:
      containers:
      - name: apisix-server
        image: harbor.aipo.lenovo.com/apisix/apisix:v1.1
        imagePullPolicy:  Always
        ports:
        - name: http
          containerPort: 9080
        - name: https
          containerPort: 9443
---
apiVersion: v1
kind: Service
metadata:
  name: apisix-service
  namespace: saas
spec:
  type: NodePort #选择NodePort，暴露给k8s外部请求
  selector:
    app: apisix-server
  ports:
  - port: 80
    targetPort: 9080
    nodePort: 31191 #暴露给k8s外部的服务端口

启动服务

启动etcd服务

[root@k8s-1 k8s]# kubectl apply -f etcd.yaml
namespace/saas created
deployment.apps/etcd-server created
service/etcd-service created

[root@k8s-1 k8s]# kubectl get pods -n saas
NAME                           READY   STATUS    RESTARTS   AGE
etcd-server-69d9fbbcd7-dmxr2   1/1     Running   0          19s

可以看到etcd的pod已经启动了

启动apisix服务

[root@k8s-1 k8s]# kubectl apply -f apisix.yaml
namespace/saas unchanged
deployment.apps/apisix-server created
service/apisix-service created

[root@k8s-1 k8s]# kubectl get pods -n saas
NAME                            READY   STATUS    RESTARTS   AGE
apisix-server-8b4cb7ff9-7mst5   1/1     Running   0          3m41s
etcd-server-69d9fbbcd7-dmxr2    1/1     Running   0          4m12s

  可以看到apisix的pod已经启动了

验证

浏览器打开k8s集群的任意一台ip地址，如http://10.110.149.172:31191/apisix/dashboard
可以在页面上面看到

页面上面可以正常添加route和upstream，表明搭建成功！

其他

附调试常用命令行
在调试k8s的调试过程中，用到了好多常用的k8s的命令，贴出来，供大家参考：

cd /root/username/workspace/docker/etcd
docker build -t harbor.aipo.lenovo.com/apisix/etcd:3.3.13 -f etcd.dockerfile .
docker rm -f etcd-server
docker run -it --name etcd-server \
 -p 23791:2379 \
 -p 23801:2380 \
 -d harbor.aipo.lenovo.com/apisix/etcd:3.3.13
docker ps |grep etcd-server
docker logs etcd-server
 -------------------------------------------

kubectl delete -f apisix.yaml
kubectl apply -f etcd.yaml
kubectl get pods -n saas -o wide
kubectl logs pods/etcd-server-746d5b4cf7-th44j -c etcd-server --namespace=saas # --previous 该参数可以查看上一次启动的容器的日志，非常有用
kubectl exec -it etcd-server-746d5b4cf7-k4bhv -c etcd-server -n saas /bin/bash 

cd /root/username/workspace/docker/apisix
docker build --no-cache -t harbor.aipo.lenovo.com/apisix/apisix:v1.1 -f apisix.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/apisix:v1.1

cd /root/caoyong/apisix/k8s
kubectl get pods -n saas -o wide
kubectl get svc -n saas -o wide
kubectl delete svc apisix-service -n saas
kubectl delete pod apisix-server-88fbc7f99-gvnr4  -n saas
kubectl delete -f apisix.yaml
kubectl get endpoints -n saas -o wide
kubectl apply -f apisix.yaml
kubectl get pods -n saas -o wide
kubectl get svc -n saas -o wide
kubectl logs pods/apisix-server-8b4cb7ff9-7mst5 -c apisix-server  --namespace=saas
kubectl exec -it  apisix-server-88fbc7f99-pf8sj -c apisix-server -n saas /bin/bash

由service寻找pod的过程

kubectl get svc -n kube-system
kubectl -n kube-system describe svc/metrics-server	
kubectl -n kube-system get pod -l app=metrics-server	
kubectl -n kube-system describe pod/kubernetes-dashboard-7844b55485-c2dxm	
kubectl -n kube-system exec -it kubernetes-dashboard-7844b55485-c2dxm /bin/sh

附可调试的etcd镜像

需要下载etcd按照源码到本地，参考：https://github.com/etcd-io/etcd/releases/tag/v3.4.5
比如下在版本为v3.4.5的etcd，如下将ETCD_VER替换成v3.4.5即可

GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

/tmp/etcd-download-test/etcd --version
/tmp/etcd-download-test/etcdctl version

创建一个工具比较齐全的docker，附centos.tools.dockerfile：

FROM centos:7.6.1810
LABEL maintainer=caoyong1
WORKDIR /root
COPY *.repo /etc/yum.repos.d/
#指定时区以及安装各种插件
RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && yum install kde-l10n-Chinese -y && yum install glibc-common -y && localedef -c -f UTF-8 -i zh_CN zh_CN.utf8 && mkdir services && yum clean all && yum makecache && yum install -y epel-release && yum clean all && yum makecache && yum -y install iputils && yum -y install net-tools.x86_64 && yum install -y redhat-lsb && yum -y install bridge-utils && yum -y install traceroute && yum -y install vim*
#指定字符集
ENV LANG zh_CN.UTF-8
ENV LANGUAGE zh_CN.UTF-8
ENV LC_ALL zh_CN.UTF-8

附本目录中的镜像源：wget http://mirrors.163.com/.help/CentOS7-Base-163.repo

docker build -t harbor.aipo.lenovo.com/apisix/centos:tools -f centos.tools.dockerfile .
docker push harbor.aipo.lenovo.com/apisix/centos:tools

以上面的镜像为基础，创建工具比较全的etcd镜像源，附etcd3.4.5.dockerfile：

FROM harbor.aipo.lenovo.com/apisix/centos:tools
WORKDIR /root
COPY etcd.conf.yml /opt/bitnami/etcd/conf/etcd.conf.yml
ADD  etcd-v3.4.5-linux-amd64.tar.gz .
WORKDIR /root/etcd-v3.4.5-linux-amd64
EXPOSE 2379 2380
ENV ALLOW_NONE_AUTHENTICATION=yes
ENTRYPOINT [ "./etcd","--config-file=/opt/bitnami/etcd/conf/etcd.conf.yml"

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub