香橙派4和树莓派4B构建K8S集群实践之四：BuildKit与LNMP
namespace为iot-age, nginx 与 php分别部署到不同的pods, 它们共用相同的pvc存储（即wwwroot目录相同，应用部署在里面）, php pods 访问相同的数据库。- 使用buildkit作为image构建工具，通过指定镜像部署定制化的php pods。
目录
3.1 申请一个pvc存储区 (wwwroot-pvc.yaml)
1. 说明
k8s带来的灵活性，使我们在部署上面有了很多花样选择，这里我希望做到以下方案：
Plan:
- namespace为iot-age, nginx 与 php分别部署到不同的pods, 它们共用相同的pvc存储（即wwwroot目录相同，应用部署在里面）, php pods 访问相同的数据库。
- 使用buildkit作为image构建工具，通过指定镜像部署定制化的php pods。
PlanB: namespace为ia2, nginx与php及其应用联合到一个deployment.yaml部署，有自己对应的存储目录, pods 共用相同的数据库。
2. 开始前的准备工作
2.1 docker 验证用户信息设置
因为将使用BuildKit来访问docker仓库（buildkit使用~/.docker/config.json）, 需要验证用户信息，先行进行设置，以免push时出现验证错误：
# 生成base64编码的auth值, 其中user, pw为hub.docker.com的账密
echo -n "{user}:{pw}" | base64
vi ~/.docker/config.json
# example:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "{auth}"
}
}
}
2.2 安装BuildKit
法1：在指定主机上安装使用
# amd64用这个
wget https://github.com/moby/buildkit/releases/download/v0.9.0/buildkit-v0.9.0.linux-amd64.tar.gz
tar xvf buildkit-v0.9.0.linux-amd64.tar.gz -C /usr/local/
# arm64用这个
wget https://github.com/moby/buildkit/releases/download/v0.9.0/buildkit-v0.9.0.linux-arm64.tar.gz
tar xvf buildkit-v0.9.0.linux-arm64.tar.gz -C /usr/local/
# 创建服务配置
cat >/etc/systemd/system/buildkitd.service <<EOF
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
After=containerd.service
[Service]
Type=notify
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl restart buildkitd
systemctl enable buildkitd
systemctl status buildkitd
法2： 创建 buildkitd和buildkit-cli pods，并在buildkit-cli里调用服务构建，需要说明的是: 我尚未能成功构建并推送上去仓库，可参考这篇文章Building Docker Images with BuildKit | Kubernetes Course Labs
# 安装buildkit server
kubectl apply -f https://kubernetes.courselabs.co/labs/buildkit/specs/buildkitd/buildkitd.yaml
kubectl logs -l app=buildkitd
# 删除buildkit
kubectl delete deploy,svc,pod -l kubernetes.courselabs.co=buildkit
kubectl create secret docker-registry registry-creds --docker-server=docker.io --docker-username={user} --docker-password={pw}
kubectl create configmap build-config --from-literal=REGISTRY=docker.io --from-literal=REPOSITORY=bennybi
kubectl apply -f https://kubernetes.courselabs.co/labs/buildkit/specs/buildkit-cli/buildkit-cli.yaml
# 进入buildkit cli pod
kubectl exec -it buildkit-cli -- sh
# 在buildkit-cli pod内调用
buildctl --addr tcp://buildkitd:1234 build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=${REGISTRY}/${REPOSITORY}/simple,push=true3. 安装步骤
3.1 申请一个pvc存储区 (wwwroot-pvc.yaml)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wwwroot
namespace: iot-age
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2G
3.2 Nginx
文件清单: nginx.yaml, nginx-configmap.yaml
iot-age中所有的nginx pods将使用统一的配置，ConfigMap (nginx-configmap.yaml)
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-conf
namespace: iot-age
data:
default.conf: |
server {
listen 80;
listen [::]:80;
server_name localhost;
index index.php index.html;
error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log;
root /var/www/html;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
kubectl apply -f nginx-configmap.yaml
nginx.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nginx
namespace: iot-age
spec:
selector:
matchLabels:
app: nginx
serviceName: "nginx"
replicas: 2
#volumeClaimTemplates:
# - kind: PersistentVolumeClaim
# apiVersion: v1
# metadata:
# name: wwwroot
# spec:
# accessModes:
# - ReadWriteOnce
# resources:
# requests:
# storage: 2Gi
# volumeMode: Filesystem
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- name: wwwroot
mountPath: /var/www/html
- name: nginx-conf
mountPath: /etc/nginx/conf.d/
volumes:
- name: wwwroot
persistentVolumeClaim:
claimName: wwwroot
- name: nginx-conf
configMap:
name: nginx-conf
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: iot-age
spec:
type: ClusterIP
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
ports:
- name: nginx
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
#---
#apiVersion: v1
#kind: Service
#metadata:
# name: nginx
# namespace: iot-age
#spec:
# type: NodePort
# ports:
# - name: nginx
# port: 80
# protocol: TCP
# targetPort: 80
# nodePort: 30010
# selector:
# app: nginxkubectl apply -f nginx.yaml
3.3 PHP-FPM
文件清单: Dockerfile, php.yaml
Dockerfile， 该文件参考了php/Dockerfile at master · khs1994-docker/php · GitHub
# syntax=docker/dockerfile:labs
ARG PHP_VERSION=8.2.5
FROM --platform=$TARGETPLATFORM php:${PHP_VERSION}-fpm-alpine3.17 as php
LABEL maintainer="khs1994-docker/lnmp <khs1994@khs1994.com>"
ARG PHP_EXTENSION_EXTRA
ARG PECL_EXTENSION_EXTRA
ARG APK_EXTRA
ARG APK_DEV_EXTRA
ENV TZ=Asia/Shanghai \
APP_ENV=development
ENV PHP_EXTENSION \
bcmath \
bz2 \
calendar \
enchant \
exif \
gd \
gettext \
gmp \
imap \
intl \
mysqli \
pcntl \
pdo_pgsql \
pdo_mysql \
pgsql \
sockets \
sysvmsg \
sysvsem \
sysvshm \
# tidy \
# xsl \
zip \
shmop \
ffi \
${PHP_EXTENSION_EXTRA:-}
ENV PECL_EXTENSION \
# mongodb \
# https://github.com/mongodb/mongo-php-driver/archive/master.tar.gz \
# igbinary \
https://github.com/igbinary/igbinary/archive/master.tar.gz \
# redis \
https://github.com/phpredis/phpredis/archive/develop.tar.gz \
memcached \
# 安装测试版的扩展，可以在扩展名后加 -beta
# yaml \
# https://github.com/php/pecl-file_formats-yaml/archive/php7.tar.gz \
https://github.com/kjdev/php-ext-zstd/archive/0.11.0.tar.gz \
${PECL_EXTENSION_EXTRA:-}
ARG ALPINE_URL=dl-cdn.alpinelinux.org
RUN sed -i "s/dl-cdn.alpinelinux.org/${ALPINE_URL}/g" /etc/apk/repositories \
&& set -xe \
\
# && apk add --no-cache patch \
# && cd / \
# && { \
# echo "--- a/usr/local/lib/php/build/php.m4"; \
# } | tee 1.diff \
# && patch -p1 < 1.diff \
# && apk del --no-network patch \
# && rm -rf 1.diff \
\
# 不要删除
&& PHP_FPM_RUN_DEPS=" \
bash \
tzdata \
libmemcached-libs \
libpq \
libzip \
zlib \
libpng \
freetype \
libjpeg-turbo \
libxpm \
libwebp \
libbz2 \
libexif \
gmp \
enchant2 \
c-client \
icu-libs \
zstd-libs \
linux-headers \
libavif \
${APK_EXTRA:-} \
" \
# tidyhtml-libs \
# libxslt \
# yaml \
# *-dev 编译之后删除
&& PHP_FPM_BUILD_DEPS=" \
openssl-dev \
libmemcached-dev \
cyrus-sasl-dev \
postgresql-dev \
libzip-dev \
zlib-dev \
libpng-dev \
freetype-dev \
libjpeg-turbo-dev \
libxpm-dev \
libwebp-dev \
libexif-dev \
gmp-dev \
bzip2-dev \
enchant2-dev \
imap-dev \
gettext-dev \
libwebp-dev \
icu-dev \
zstd-dev \
libavif-dev \
${APK_DEV_EXTRA:-} \
" \
# tidyhtml-dev \
# libxslt-dev \
# yaml-dev \
&& apk add --no-cache --virtual .php-fpm-run-deps $PHP_FPM_RUN_DEPS \
&& apk add --no-cache --virtual .php-fpm-build-deps $PHP_FPM_BUILD_DEPS \
&& apk add --no-cache --virtual .phpize-deps $PHPIZE_DEPS \
&& curl -fsSL -o /usr/local/bin/pickle \
https://github.com/khs1994-php/pickle/releases/download/nightly/pickle-debug.phar \
&& chmod +x /usr/local/bin/pickle \
# 安装内置扩展
&& docker-php-source extract \
&& docker-php-ext-configure zip \
--with-zip \
&& docker-php-ext-install zip \
&& strip --strip-all $(php-config --extension-dir)/zip.so \
# && docker-php-ext-configure gd \
&& echo " \
--disable-gd-jis-conv \
--with-freetype \
--with-jpeg \
--with-webp \
--with-xpm \
--with-avif" > /tmp/gd.configure.options \
# && docker-php-ext-install $PHP_EXTENSION \
&& pickle install $PHP_EXTENSION -n --defaults --strip \
&& docker-php-source delete \
# 安装 PECL 扩展
&& echo "--enable-redis-igbinary --enable-redis-zstd" > /tmp/redis.configure.options \
&& echo "--enable-memcached-igbinary" > /tmp/memcached.configure.options \
&& echo "--with-libzstd" > /tmp/zstd.configure.options \
&& pickle install $PECL_EXTENSION -n --defaults \
--strip --cleanup \
# 默认不启用的扩展
&& pickle install \
# https://github.com/tideways/php-xhprof-extension.git
https://github.com/tideways/php-xhprof-extension/archive/master.tar.gz \
-n --defaults --strip --cleanup --no-write \
&& pickle install opcache \
# && docker-php-ext-enable opcache \
&& apk del --no-network .phpize-deps .php-fpm-build-deps \
&& rm -rf /tmp/* \
# 创建日志文件夹
&& mkdir -p /var/log/php-fpm \
&& ln -sf /dev/stdout /var/log/php-fpm/access.log \
&& ln -sf /dev/stderr /var/log/php-fpm/error.log \
&& chmod -R 777 /var/log/php-fpm \
&& rm -rf /usr/local/lib/php/.registry/.channel.pecl.php.net/* \
&& php -m \
&& ls -la $(php-config --extension-dir) \
&& php -d error_reporting=22527 -d display_errors=1 -r 'var_dump(iconv("UTF-8", "UTF-8//IGNORE", "This is the Euro symbol '\''€'\''."));' \
# smoke test
&& php --version
WORKDIR /app
ARG VCS_REF="unknow"
LABEL org.opencontainers.image.revision=$VCS_REF \
org.opencontainers.image.source="https://github.com/khs1994-docker/php"3.3.1 构建并推送镜像
buildctl build \
--frontend=dockerfile.v0 \
--local context=. \
--local dockerfile=. \
--export-cache type=inline \
--output type=image,name=docker.io/bennybi/php-fpm:v1,push=true
php.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: php
namespace: iot-age
spec:
selector:
matchLabels:
app: php
tier: backend
replicas: 2
template:
metadata:
labels:
app: php
tier: backend
spec:
containers:
- name: php
#image: php:8.2-fpm
image: bennybi/php-fpm:v1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9000
volumeMounts:
- name: wwwroot
mountPath: /var/www/html/
volumes:
- name: wwwroot
persistentVolumeClaim:
claimName: wwwroot
---
apiVersion: v1
kind: Service
metadata:
name: php
namespace: iot-age
spec:
type: ClusterIP
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
ports:
- name: php
port: 9000
protocol: TCP
targetPort: 9000
selector:
app: php
kubectl apply -f php.yaml
3.3.2 定义入口(ingress 方式)
编写ingres-nginx.yaml，注意填写对应的 ingress.class是哪个
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ia-web-service1
namespace: iot-age
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/session-cookie-name: stickounet
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
spec:
rules:
- host: "*.k8s-t1.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
ingressClassName: nginx安装效果如图： 
3.4 Maria Galera
用helm工具来安装maria-galera集群是相当方便的，但遇到的重启机器后crash问题解决却要找好久(见遇到的问题集合)。
3.4.1 安装相关命令
# 用在线源安装，如果有现存数据库，须加上现用的用密，否则重装会产生新的密码字典，造成pod内部root登录问题，不能启动!
helm install my-release oci://registry-1.docker.io/bitnamicharts/mariadb-galera \
--set galera.mariabackup.password=aaa\
--set rootUser.password=bbb\
--set db.password=ccc\
--set galera.bootstrap.forceSafeToBootstrap=true \
--set galera.bootstrap.bootstrapFromNode=0
# 下载安装
helm list
helm search repo mariadb-galera
helm fetch bitnami/mariadb-galera --version 8.1.1
tar -zxvf mariadb-galera-8.1.1.tgz
# To uninstall/delete the my-release release:
helm delete my-release
# 临时建立个client pod来访问db
kubectl run my-release-mariadb-galera-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/mariadb-galera:10.11.3-debian-11-r0 --command \
-- mysql -h my-release-mariadb-galera -P 3306 -uroot -p$(kubectl get secret --namespace default my-release-mariadb-galera -o jsonpath="{.data.mariadb-root-password}" | base64 -d) my_database
设置本地端口转发远程pod端口
kubectl port-forward [portName] [LocalPort:RemotePort]3.4.2 特别注意
在关停宿主机或K8s集群服务的时候，要先停掉mariadb-galera相关的pods, 不然重启后会出现一堆莫名错误（靠修改safe_to_bootstrap:1也不保证能顺利启动），执行：
kubectl scale --replicas=0 statefulset/my-release-mariadb-galera 暂停
kubectl scale --replicas=3 statefulset/my-release-mariadb-galera 恢复
如图：
或者，k8s启动完毕后，再重建pods (mariadb-galera-start.sh)
#!/bin/bash
export MARIADB_PASSWORD=$(kubectl get secret --namespace "default" my-release-mariadb-galera -o jsonpath="{.data.mariadb-password}" | base64 -d) && \
export MARIADB_ROOT_PASSWORD=$(kubectl get secret --namespace "default" my-release-mariadb-galera -o jsonpath="{.data.mariadb-root-password}" | base64 -d) && \
export MARIADB_GALERA_MARIABACKUP_PASSWORD=$(kubectl get secret --namespace "default" my-release-mariadb-galera -o jsonpath="{.data.mariadb-galera-mariabackup-password}" | base64 -d)
helm upgrade my-release \
--set galera.mariabackup.password=$MARIADB_GALERA_MARIABACKUP_PASSWORD \
--set rootUser.password=$MARIADB_ROOT_PASSWORD \
--set db.password=$MARIADB_PASSWORD \
--set galera.bootstrap.forceSafeToBootstrap=true \
--set galera.bootstrap.bootstrapFromNode=0 ../mariadb-galera (values.yaml所在目录)
3.4.3 外部连接数据库
法一：临时
# 在192.168.0.106设置转发端口 => 服务端口
kubectl port-forward --namespace default svc/my-release-mariadb-galera 3306:3306 --address 127.0.0.1,192.168.0.106 &法二：screen 方式
screen -S mariadb-galera
kubectl port-forward --namespace default svc/my-release-mariadb-galera 3306:3306 --address 127.0.0.1,192.168.0.106
# 离开会话并让程序断续运行：ctrl a d (按住ctrl不放，分别按 a 和 d)法三: 配置ingress-nginx开放转发3306端口，另外更进一步可同时配置ingress-mariadb用子域名访问，详见香橙派4和树莓派4B构建K8S集群实践之五：端口公开访问配置_bennybi的博客-CSDN博客。
3.3.4 内部跨域访问服务
通过 {SERVICE_NAME}.{NAMESPACE_NAME}.svc.cluster.local 这样的格式，访问目标 namespace 下的服务。
如上述的db服务：my-release-mariadb-galera.default
4. 遇到的问题
buildctl build 并 push上去时，出现 "authorization status: 401: authorization failed"错误
解决办法：确认 ~/.docker/config.json已正确设置auth信息
重启k8s集群后遇到maria-galera不能启动的状况：
[ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .
解决办法：
- 分别去db pods对应的pvc部署目录，找到grastate.dat文件，修改seqno最大的文件(如果都为0，就修改my-release-mariadb-galera-0)，把safe_to_bootstrap设为1，删除该pod后将自动重建。
# GALERA saved state
version: 2.1
uuid: a550ccd5-f31f-11ed-8130-0ecae2f1ac87
seqno: 16
safe_to_bootstrap: 15. 相关命令
# 查看本地就绪镜像
ctr -n buildkit images list 或 ctr -n buildkit i ls
# 检查证书过期时间
kubeadm certs check-expiration
6. 参考
GitHub - moby/buildkit: concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
Building Docker Images with BuildKit | Kubernetes Course Labs
Deploy a PHP Application on Kubernetes Cluster with Ubuntu 18.04
K8s---Pod搭建LNMP_k8s搭建lnmp_我不满意的博客-CSDN博客
k8s之StatefulSet详解_最美dee时光的博客-CSDN博客
Helm Charts to deploy MariaDB Galera in Kubernetes
charts/bitnami/mariadb-galera at main · bitnami/charts · GitHub
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
2
0- 0
已为社区贡献9条内容
所有评论(0)