Kubernetes入门完全指南：从零到企业级实战
·
1. 真实的生产事故场景
场景：某电商公司"双11"大促期间，凌晨2点，核心订单服务发生雪崩式崩溃…
事故回顾：
# 事故现场 - 传统部署方式
$ ssh root@web-server-01
$ systemctl status order-service
● order-service.service - Order Service
Loaded: loaded (/etc/systemd/system/order-service.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Wed 2023-11-11 02:00:00 CST; 5min ago
Process: 12345 ExecStart=/opt/order-service/start.sh (code=killed, signal=KILL)
Main PID: 12345 (code=killed, signal=KILL)
CGroup: /system.slice/order-service.service
└─12345 /usr/bin/java -jar order-service.jar
# 日志显示
$ journalctl -u order-service -n 100
Nov 11 02:00:00 web-server-01 java[12345]: java.lang.OutOfMemoryError: Java heap space
Nov 11 02:00:01 web-server-01 systemd[1]: order-service.service: Main process exited, code=killed, status=9/KILL
Nov 11 02:00:05 web-server-01 systemd[1]: order-service.service: Failed with result 'timeout'.
传统部署的问题：
- 单点故障：服务器宕机，服务完全不可用
- 手动运维：需要SSH到每台服务器检查
- 资源隔离差：一个服务OOM可能影响其他服务
- 扩缩容慢：添加新服务器需要数小时
- 配置漂移：各环境配置不一致
Kubernetes解决方案：
# 在K8s集群中查看相同服务的状态
$ kubectl get pods -n order-service
NAME READY STATUS RESTARTS AGE
order-service-7c8d9f5c8-abc12 1/1 Running 0 7d
order-service-7c8d9f5c8-def34 1/1 Running 0 7d
order-service-7c8d9f5c8-ghi56 1/1 Running 0 7d
# 当一个Pod崩溃时
$ kubectl get pods -n order-service
NAME READY STATUS RESTARTS AGE
order-service-7c8d9f5c8-abc12 0/1 Error 1 7d # 这个Pod挂了
order-service-7c8d9f5c8-def34 1/1 Running 0 7d
order-service-7c8d9f5c8-ghi56 1/1 Running 0 7d
order-service-7c8d9f5c8-jkl78 1/1 Running 0 10s # 自动新启动的Pod
2. 什么是Kubernetes？解决什么问题？
2.1 Kubernetes定义
Kubernetes（常简称为K8s）是一个开源的容器编排平台，用于自动化部署、扩展和管理容器化应用程序。
版本说明：
- 当前稳定版本：1.28（2023年8月发布）
- 长期支持版本：1.26、1.27、1.28
- 开发版本：1.29（开发中）
2.2 核心价值：解决了什么问题？
| 问题类别 | 传统方案痛点 | Kubernetes解决方案 |
|---|---|---|
| 部署管理 | 手动部署，易出错 | 声明式配置，自动部署 |
| 可用性 | 单点故障，恢复慢 | 自动故障转移，自愈能力 |
| 伸缩性 | 手动扩缩容，响应慢 | 自动水平扩缩容（HPA） |
| 资源利用 | 资源浪费，利用率低 | 资源调度优化 |
| 配置管理 | 配置漂移，环境差异 | 统一配置管理 |
| 服务发现 | 硬编码IP，维护困难 | DNS服务发现 |
| 负载均衡 | 硬件负载均衡器昂贵 | 内置负载均衡 |
2.3 典型应用场景
场景1：微服务架构
# 微服务应用部署示例
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service
spec:
replicas: 3
selector:
matchLabels:
app: user-service
template:
metadata:
labels:
app: user-service
spec:
containers:
- name: user-service
image: user-service:v1.2.0
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: database.host
场景2：CI/CD流水线
# GitLab CI/CD + K8s
deploy-to-k8s:
stage: deploy
script:
- kubectl config use-context production
- kubectl apply -f k8s/deployment.yaml
- kubectl rollout status deployment/myapp
only:
- main
场景3：大数据处理
# Spark on K8s
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: spark-etl-job
spec:
mode: cluster
image: spark:3.3.0
mainClass: org.apache.spark.examples.SparkPi
sparkVersion: "3.3.0"
3. Kubernetes核心架构深度解析
3.1 架构总览
┌─────────────────────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 控制平面 (Control Plane) │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────┐ │ │
│ │ │ API Server │ │ Scheduler │ │ Controller │ │ etcd │ │ │
│ │ │ │ │ │ │ Manager │ │ │ │ │
│ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └────┬────┘ │ │
│ │ │ │ │ │ │ │
│ │ ┌──────▼─────────────────▼─────────────────▼───────────────▼────┐ │ │
│ │ │ 集群状态存储与同步 │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 数据平面 (Data Plane/Worker Nodes) │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ 工作节点 1 │ │ 工作节点 2 │ │ 工作节点 3 │ │ │
│ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │
│ │ │ │ kubelet │ │ │ │ kubelet │ │ │ │ kubelet │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │kube-proxy │ │ │ │kube-proxy │ │ │ │kube-proxy │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │ 容器运行时 │ │ │ │ 容器运行时 │ │ │ │ 容器运行时 │ │ │ │
│ │ │ │(containerd) │ │ │ │(containerd) │ │ │ │(containerd) │ │ │ │
│ │ │ │ ┌───────┐ │ │ │ │ ┌───────┐ │ │ │ │ ┌───────┐ │ │ │ │
│ │ │ │ │ Pod 1 │ │ │ │ │ │ Pod 2 │ │ │ │ │ │ Pod 3 │ │ │ │ │
│ │ │ │ └───────┘ │ │ │ │ └───────┘ │ │ │ │ └───────┘ │ │ │ │
│ │ │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 云提供商/基础设施层 │ │
│ │ AWS / Azure / GCP / 本地数据中心 / 边缘计算 │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
3.2 控制平面组件详解
1. kube-apiserver（API服务器）
# API服务器暴露的端口
$ netstat -tlnp | grep kube-apiserver
tcp6 0 0 :::6443 :::* LISTEN 1234/kube-apiserver
# 查看API版本
$ curl -k https://localhost:6443/version
{
"major": "1",
"minor": "28",
"gitVersion": "v1.28.0",
"gitCommit": "a7b5c7f2e5",
"gitTreeState": "clean",
"buildDate": "2023-08-15T00:00:00Z",
"goVersion": "go1.20.5",
"compiler": "gc",
"platform": "linux/amd64"
}
2. etcd（键值存储）
# etcd集群健康检查
$ kubectl get pods -n kube-system -l component=etcd
NAME READY STATUS RESTARTS AGE
etcd-k8s-master 1/1 Running 0 15d
# etcd数据备份
$ etcdctl snapshot save /backup/etcd-snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Snapshot saved at /backup/etcd-snapshot.db
3. kube-scheduler（调度器）
# 调度器配置示例
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
plugins:
preFilter:
enabled:
- name: NodeResourcesFit
disabled:
- name: "*"
filter:
enabled:
- name: NodeResourcesFit
disabled:
- name: "*"
score:
enabled:
- name: NodeResourcesFit
weight: 1
4. kube-controller-manager（控制器管理器）
# 查看运行的控制器
$ kubectl get pods -n kube-system -l component=kube-controller-manager
NAME READY STATUS RESTARTS AGE
kube-controller-manager-k8s-master 1/1 Running 0 15d
# 控制器包括：
# - Node Controller（节点控制器）
# - Replication Controller（副本控制器）
# - Endpoints Controller（端点控制器）
# - Service Account & Token Controllers（服务账户和令牌控制器）
4. Kubernetes 1.28+ 新特性一览
4.1 1.28版本重要特性
特性1：Sidecar容器（稳定版）
apiVersion: v1
kind: Pod
metadata:
name: sidecar-example
spec:
containers:
- name: main-app
image: nginx:1.25
ports:
- containerPort: 80
- name: log-agent
image: fluentd:latest
# 标记为sidecar容器
restartPolicy: Always
# Sidecar容器会在主容器启动前启动，主容器结束后才结束
特性2：混合版本代理（Mixed Version Proxy）
# 允许不同版本的kube-proxy共存
$ kubectl get daemonsets -n kube-system kube-proxy
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-proxy 3 3 3 3 3 <none> 15d
特性3：节点内存交换支持（生产就绪）
apiVersion: v1
kind: Node
metadata:
name: worker-node-1
spec:
# 节点配置支持swap
config:
kubeletConfiguration:
failSwapOn: false
memorySwap:
swapBehavior: LimitedSwap
4.2 1.27版本回顾
特性1：原地资源调整（In-place Resource Resize）
apiVersion: v1
kind: Pod
metadata:
name: resize-example
spec:
containers:
- name: app
image: nginx
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "200Mi"
cpu: "200m"
# 支持原地调整资源，无需重启Pod
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired
- resourceName: memory
restartPolicy: NotRequired
5. 核心概念：必须掌握的10个组件
5.1 Pod：最小的部署单元
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
tier: frontend
spec:
containers:
- name: nginx-container
image: nginx:1.25
ports:
- containerPort: 80
env:
- name: ENVIRONMENT
value: "production"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
restartPolicy: Always
运行验证：
$ kubectl apply -f pod.yaml
pod/myapp-pod created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-pod 1/1 Running 0 10s
$ kubectl describe pod myapp-pod
Name: myapp-pod
Namespace: default
Priority: 0
Service Account: default
Node: node-1/192.168.1.10
Start Time: Mon, 01 Jan 2024 10:00:00 +0800
Labels: app=myapp
tier=frontend
Annotations: <none>
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Containers:
nginx-container:
Container ID: containerd://abc123...
Image: nginx:1.25
Image ID: docker.io/library/nginx@sha256:...
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 01 Jan 2024 10:00:05 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
5.2 Deployment：无状态应用的管家
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # 最大激增Pod数
maxUnavailable: 0 # 最大不可用Pod数
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
periodSeconds: 5
部署验证：
$ kubectl apply -f deployment.yaml
deployment.apps/nginx-deployment created
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 10s
$ kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE
nginx-deployment-7c8d9f5c8-abc12 1/1 Running 0 15s
nginx-deployment-7c8d9f5c8-def34 1/1 Running 0 15s
nginx-deployment-7c8d9f5c8-ghi56 1/1 Running 0 15s
# 查看部署详情
$ kubectl describe deployment nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Mon, 01 Jan 2024 10:00:00 +0800
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.25
Port: 80/TCP
Liveness: http-get http://:80/ delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:80/ delay=5s timeout=1s period=5s #success=1 #failure=3
Environment: <none>
5.3 Service：服务的稳定入口
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
type: ClusterIP # 也可以是 NodePort, LoadBalancer, ExternalName
服务验证：
$ kubectl apply -f service.yaml
service/nginx-service created
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service ClusterIP 10.96.123.123 <none> 80/TCP,443/TCP 10s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 15d
# 从集群内部访问服务
$ kubectl run curl-test --image=curlimages/curl --rm -it -- sh
/ $ curl http://nginx-service.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body>
<h1>Welcome to nginx!</h1>
</body>
</html>
5.4 ConfigMap和Secret：配置管理
# ConfigMap示例
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
# 简单键值对
database.host: "mysql-primary"
database.port: "3306"
app.environment: "production"
# 配置文件
nginx.conf: |
server {
listen 80;
server_name localhost;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
# 多行配置
application.properties: |
server.port=8080
spring.datasource.url=jdbc:mysql://mysql-primary:3306/mydb
logging.level.root=INFO
# Secret示例
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
type: Opaque
stringData:
# 这些数据会自动base64编码
database-password: "MySecurePass123!"
api-key: "ak_sk_1234567890abcdef"
jwt-secret: "my-super-secret-jwt-key"
# TLS Secret
apiVersion: v1
kind: Secret
metadata:
name: tls-secret
type: kubernetes.io/tls
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCi4uLiA= # base64编码的证书
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCi4uLiA= # base64编码的私钥
使用验证：
# 创建ConfigMap
$ kubectl apply -f configmap.yaml
configmap/app-config created
# 创建Secret
$ kubectl apply -f secret.yaml
secret/app-secrets created
# 查看ConfigMap
$ kubectl get configmap app-config
NAME DATA AGE
app-config 3 10s
$ kubectl describe configmap app-config
Name: app-config
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
database.host:
----
mysql-primary
database.port:
----
3306
nginx.conf:
----
server {
listen 80;
server_name localhost;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
BinaryData
====
Events: <none>
# 查看Secret（数据是base64编码的）
$ kubectl get secret app-secrets
NAME TYPE DATA AGE
app-secrets Opaque 3 15s
$ kubectl get secret app-secrets -o yaml
apiVersion: v1
data:
api-key: YWtfc2tfMTIzNDU2Nzg5MGFiY2RlZg==
database-password: TXlTZWN1cmVQYXNzMTIzIQ==
jwt-secret: bXktc3VwZXItc2VjcmV0LWp3dC1rZXk=
kind: Secret
metadata:
name: app-secrets
namespace: default
type: Opaque
5.5 Namespace：逻辑隔离
# 查看所有命名空间
$ kubectl get namespaces
NAME STATUS AGE
default Active 15d
kube-system Active 15d
kube-public Active 15d
kube-node-lease Active 15d
# 创建命名空间
$ kubectl create namespace production
namespace/production created
$ kubectl create namespace development
namespace/development created
# 在特定命名空间中部署应用
$ kubectl apply -f deployment.yaml -n production
deployment.apps/nginx-deployment created
# 查看特定命名空间的资源
$ kubectl get pods -n production
NAME READY STATUS RESTARTS AGE
nginx-deployment-7c8d9f5c8-abc12 1/1 Running 0 10s
5.6 其他重要概念
- StatefulSet：有状态应用（如数据库）
- DaemonSet：每个节点运行一个Pod（如日志收集器）
- Job/CronJob：批处理任务
- HorizontalPodAutoscaler：水平自动扩缩容
- NetworkPolicy：网络策略
- PersistentVolume/PersistentVolumeClaim：持久化存储
- Role/RoleBinding：RBAC权限控制
6. 实战：5分钟搭建你的第一个K8s集群
6.1 方案选择比较
| 工具 | 适用场景 | 资源需求 | 学习曲线 | 生产就绪 |
|---|---|---|---|---|
| minikube | 本地开发测试 | 低（2CPU/2GB） | 简单 | 否 |
| k3s | 边缘/IoT/测试 | 低（512MB内存） | 中等 | 是（轻量级） |
| kubeadm | 生产环境 | 中高（2CPU/4GB每节点） | 较陡 | 是 |
| kind | CI/CD测试 | 中（Docker容器） | 简单 | 否 |
6.2 方案1：使用minikube（开发环境）
步骤1：安装minikube
# Linux
$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ sudo install minikube-linux-amd64 /usr/local/bin/minikube
# MacOS
$ brew install minikube
# Windows (管理员PowerShell)
$ choco install minikube
步骤2：启动集群
# 启动minikube集群
$ minikube start --driver=docker --cpus=2 --memory=4096 --kubernetes-version=v1.28.0
😄 minikube v1.32.0 on Ubuntu 20.04
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=4096MB) ...
🐳 Preparing Kubernetes v1.28.0 on Docker 24.0.7 ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
# 验证集群状态
$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
# 查看节点
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 45s v1.28.0
步骤3：部署第一个应用
# 创建Deployment
$ kubectl create deployment nginx --image=nginx:1.25
deployment.apps/nginx created
# 暴露服务
$ kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
# 查看服务
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2m
nginx NodePort 10.96.123.123 <none> 80:32123/TCP 10s
# 获取访问URL
$ minikube service nginx --url
http://192.168.49.2:32123
# 访问应用
$ curl http://192.168.49.2:32123
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body>
<h1>Welcome to nginx!</h1>
</body>
</html>
步骤4：查看Dashboard
# 启用Dashboard
$ minikube dashboard
🤔 Verifying dashboard health ...
🚀 Launching proxy ...
🤔 Verifying proxy health ...
🎉 Opening http://127.0.0.1:43321/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...
6.3 方案2：使用k3s（生产轻量级）
步骤1：安装k3s
# 一键安装（单节点）
$ curl -sfL https://get.k3s.io | sh -
# 或者指定版本安装
$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.28.0+k3s1 sh -
# 验证安装
$ sudo k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-master Ready control-plane,master 30s v1.28.0+k3s1
步骤2：多节点集群
# 主节点
$ curl -sfL https://get.k3s.io | K3S_TOKEN=mysecret sh -s - server --cluster-init
# 工作节点
$ curl -sfL https://get.k3s.io | K3S_TOKEN=mysecret K3S_URL=https://master-ip:6443 sh -
# 查看所有节点
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-master Ready control-plane,master 2m v1.28.0+k3s1
k3s-node-1 Ready <none> 30s v1.28.0+k3s1
k3s-node-2 Ready <none> 25s v1.28.0+k3s1
6.4 方案3：使用kubeadm（生产环境）
步骤1：准备环境
# 所有节点执行
# 1. 关闭swap
$ sudo swapoff -a
$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
# 2. 安装容器运行时（containerd）
$ sudo apt-get update
$ sudo apt-get install -y containerd
$ sudo mkdir -p /etc/containerd
$ containerd config default | sudo tee /etc/containerd/config.toml
$ sudo systemctl restart containerd
# 3. 安装kubeadm, kubelet, kubectl
$ sudo apt-get update
$ sudo apt-get install -y apt-transport-https ca-certificates curl
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl
步骤2：初始化主节点
$ sudo kubeadm init \
--kubernetes-version=v1.28.0 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-advertise-address=192.168.1.100
# 输出示例
[init] Using Kubernetes version: v1.28.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster. Run:
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.1.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
步骤3：配置网络插件
# 安装Flannel网络插件
$ kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
# 或者安装Calico
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml
步骤4：加入工作节点
# 在工作节点上执行
$ sudo kubeadm join 192.168.1.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
# 在主节点查看所有节点
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane 5m v1.28.0
worker-1 Ready <none> 30s v1.28.0
worker-2 Ready <none> 25s v1.28.0
7. kubectl命令完全指南
7.1 基本命令格式
# 基本语法
kubectl [command] [TYPE] [NAME] [flags]
# 示例
kubectl get pods # 获取所有Pod
kubectl describe pod nginx # 查看Pod详情
kubectl create -f config.yaml # 创建资源
kubectl apply -f config.yaml # 应用配置
kubectl delete pod nginx # 删除Pod
kubectl logs nginx # 查看日志
kubectl exec -it nginx -- sh # 进入容器
7.2 常用命令分类
1. 集群管理命令
# 查看集群信息
$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.1.100:6443
CoreDNS is running at https://192.168.1.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
$ kubectl cluster-info dump # 导出完整集群信息
$ kubectl get componentstatuses # 查看组件状态
$ kubectl get cs # 简写
# 查看版本
$ kubectl version
Client Version: v1.28.0
Server Version: v1.28.0
$ kubectl version --short # 简洁版本
$ kubectl version --output=yaml # YAML格式输出
2. 节点管理命令
# 查看节点
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane 5m v1.28.0
worker-1 Ready <none> 30s v1.28.0
$ kubectl get nodes -o wide # 显示详细信息
$ kubectl get nodes -o yaml # YAML格式
$ kubectl get nodes -o json # JSON格式
# 节点详情
$ kubectl describe node master-node
Name: master-node
Roles: control-plane
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=master-node
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
CreationTimestamp: Mon, 01 Jan 2024 10:00:00 +0800
Taints: node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: master-node
AcquireTime: <unset>
RenewTime: Mon, 01 Jan 2024 10:05:00 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Mon, 01 Jan 2024 10:00:05 +0800 Mon, 01 Jan 2024 10:00:05 +0800 FlannelIsUp Flannel is running on this node
MemoryPressure False Mon, 01 Jan 2024 10:04:55 +0800 Mon, 01 Jan 2024 10:00:00 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 01 Jan 2024 10:04:55 +0800 Mon, 01 Jan 2024 10:00:00 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 01 Jan 2024 10:04:55 +0800 Mon, 01 Jan 2024 10:00:00 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Mon, 01 Jan 2024 10:04:55 +0800 Mon, 01 Jan 2024 10:00:05 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.1.100
Hostname: master-node
Capacity:
cpu: 4
ephemeral-storage: 20484812Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8050176Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 18879241538
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7947776Ki
pods: 110
System Info:
Machine ID: 0123456789abcdef
System UUID: 01234567-89ab-cdef-0123-456789abcdef
Boot ID: abcdef01-2345-6789-abcd-ef0123456789
Kernel Version: 5.4.0-42-generic
OS Image: Ubuntu 20.04.1 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.8
Kubelet Version: v1.28.0
Kube-Proxy Version: v1.28.0
3. Pod管理命令
# 创建Pod
$ kubectl run nginx --image=nginx:1.25 --port=80
$ kubectl create deployment nginx --image=nginx:1.25
# 查看Pod
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7c8d9f5c8-abc12 1/1 Running 0 5m
$ kubectl get pods -o wide # 显示详细信息
$ kubectl get pods -w # 实时监控
$ kubectl get pods --show-labels # 显示标签
$ kubectl get pods -l app=nginx # 按标签过滤
$ kubectl get pods --all-namespaces # 所有命名空间
$ kubectl get pods -A # 简写
# Pod详情
$ kubectl describe pod nginx-7c8d9f5c8-abc12
$ kubectl get pod nginx-7c8d9f5c8-abc12 -o yaml # YAML格式
$ kubectl get pod nginx-7c8d9f5c8-abc12 -o json # JSON格式
# Pod日志
$ kubectl logs nginx-7c8d9f5c8-abc12
$ kubectl logs -f nginx-7c8d9f5c8-abc12 # 实时日志
$ kubectl logs --tail=100 nginx-7c8d9f5c8-abc12 # 最后100行
$ kubectl logs --since=1h nginx-7c8d9f5c8-abc12 # 最近1小时日志
$ kubectl logs -c container-name pod-name # 多容器Pod指定容器
# 进入Pod
$ kubectl exec -it nginx-7c8d9f5c8-abc12 -- sh
$ kubectl exec nginx-7c8d9f5c8-abc12 -- ls -la
$ kubectl exec nginx-7c8d9f5c8-abc12 -- cat /etc/os-release
# 复制文件
$ kubectl cp nginx-7c8d9f5c8-abc12:/etc/nginx/nginx.conf ./nginx.conf
$ kubectl cp ./config.conf nginx-7c8d9f5c8-abc12:/tmp/config.conf
# 删除Pod
$ kubectl delete pod nginx-7c8d9f5c8-abc12
$ kubectl delete pod nginx-7c8d9f5c8-abc12 --force --grace-period=0 # 强制删除
4. Deployment管理命令
# 查看Deployment
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 5m
$ kubectl get deploy # 简写
$ kubectl get deploy -o wide # 详细信息
$ kubectl describe deploy nginx-deployment
# 扩缩容
$ kubectl scale deployment nginx-deployment --replicas=5
deployment.apps/nginx-deployment scaled
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-deployment-7c8d9f5c8-abc12 1/1 Running 0 5m
nginx-deployment-7c8d9f5c8-def34 1/1 Running 0 5m
nginx-deployment-7c8d9f5c8-ghi56 1/1 Running 0 5m
nginx-deployment-7c8d9f5c8-jkl78 1/1 Running 0 10s
nginx-deployment-7c8d9f5c8-mno90 1/1 Running 0 10s
# 更新镜像
$ kubectl set image deployment/nginx-deployment nginx=nginx:1.26
deployment.apps/nginx-deployment image updated
# 回滚
$ kubectl rollout history deployment/nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 <none>
2 <none>
$ kubectl rollout undo deployment/nginx-deployment
deployment.apps/nginx-deployment rolled back
# 查看更新状态
$ kubectl rollout status deployment/nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
deployment "nginx-deployment" successfully rolled out
# 暂停/恢复更新
$ kubectl rollout pause deployment/nginx-deployment
$ kubectl rollout resume deployment/nginx-deployment
5. Service管理命令
# 查看Service
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
nginx ClusterIP 10.96.123.123 <none> 80/TCP 5m
$ kubectl get svc # 简写
$ kubectl describe svc nginx
# 端口转发
$ kubectl port-forward svc/nginx 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
# 浏览器访问 http://localhost:8080
6. ConfigMap/Secret管理
# 创建ConfigMap
$ kubectl create configmap app-config \
--from-literal=database.host=mysql-primary \
--from-literal=database.port=3306
$ kubectl create configmap nginx-config \
--from-file=nginx.conf=./nginx.conf
# 创建Secret
$ kubectl create secret generic db-secret \
--from-literal=username=admin \
--from-literal=password=MyPass123
$ kubectl create secret tls tls-secret \
--cert=tls.crt \
--key=tls.key
# 查看
$ kubectl get configmaps
$ kubectl get secrets
$ kubectl describe configmap app-config
$ kubectl describe secret db-secret
7. 命名空间管理
# 创建命名空间
$ kubectl create namespace production
$ kubectl create namespace development
# 查看所有命名空间
$ kubectl get namespaces
$ kubectl get ns # 简写
# 在指定命名空间操作
$ kubectl get pods -n production
$ kubectl apply -f deployment.yaml -n production
$ kubectl config set-context --current --namespace=production # 设置默认命名空间
8. 标签和选择器
# 添加标签
$ kubectl label pod nginx-abc123 app=nginx version=v1.2.0
pod/nginx-abc123 labeled
# 修改标签
$ kubectl label pod nginx-abc123 version=v1.3.0 --overwrite
pod/nginx-abc123 labeled
# 按标签查询
$ kubectl get pods -l app=nginx
$ kubectl get pods -l 'app in (nginx,web)'
$ kubectl get pods -l 'version!=v1.0.0'
$ kubectl get pods -l 'app=nginx,version=v1.2.0'
# 删除标签
$ kubectl label pod nginx-abc123 version-
pod/nginx-abc123 labeled
9. 事件和调试
# 查看事件
$ kubectl get events
$ kubectl get events --sort-by=.metadata.creationTimestamp
$ kubectl get events -w # 实时监控
$ kubectl get events --field-selector involvedObject.kind=Pod
$ kubectl get events --field-selector involvedObject.name=nginx-abc123
# 查看资源使用
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master-node 120m 3% 1200Mi 15%
worker-1 80m 2% 800Mi 10%
$ kubectl top pods
NAME CPU(cores) MEMORY(bytes)
nginx-deployment-abc12 1m 10Mi
nginx-deployment-def34 2m 12Mi
# 调试
$ kubectl debug -it nginx-abc123 --image=busybox --target=nginx -- sh
$ kubectl logs --previous nginx-abc123 # 查看前一个容器的日志
$ kubectl attach nginx-abc123 -c nginx # 附加到运行中的容器
10. 导出和导入
# 导出资源配置
$ kubectl get deployment nginx-deployment -o yaml > nginx-deployment.yaml
$ kubectl get deployment nginx-deployment -o json > nginx-deployment.json
$ kubectl get all -o yaml > all-resources.yaml
# 从标准输入创建
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: busybox
command: ["sleep", "3600"]
EOF
# 批量操作
$ kubectl apply -f ./k8s/ # 应用目录下所有文件
$ kubectl delete -f ./k8s/ # 删除目录下所有资源
$ kubectl get $(kubectl api-resources --namespaced=true --verbs=list -o name | paste -sd, -) # 获取所有资源
8. YAML配置：从基础到精通
8.1 YAML基础语法
# 注释以#开头
# 键值对使用冒号分隔
key: value
# 列表使用短横线
list:
- item1
- item2
- item3
# 多行字符串
description: |
This is a
multi-line
string.
# 折叠字符串（转换为单行）
description: >
This is a
folded string
that becomes one line.
# 锚点和别名（复用配置）
defaults: &defaults
image: nginx:latest
ports:
- 80
service1:
<<: *defaults
name: service-1
service2:
<<: *defaults
name: service-2
8.2 完整Pod配置示例
# pod-complete.yaml
apiVersion: v1 # API版本，Pod属于v1
kind: Pod # 资源类型
metadata: # 元数据
name: complete-pod # Pod名称，必须唯一
namespace: default # 命名空间，默认default
labels: # 标签，用于选择器
app: nginx # 应用名称
version: "1.25" # 版本标签
environment: production # 环境标签
annotations: # 注解，不会用于选择器
kubernetes.io/description: "Complete Pod example" # 描述
maintainer: "ops-team" # 维护者
created-by: "kubectl apply" # 创建方式
spec: # 规格定义
# 调度相关配置
nodeSelector: # 节点选择器
disktype: ssd # 只调度到有disktype=ssd标签的节点
affinity: # 亲和性配置
nodeAffinity: # 节点亲和性
requiredDuringSchedulingIgnoredDuringExecution: # 硬性要求
nodeSelectorTerms: # 节点选择条件
- matchExpressions: # 匹配表达式
- key: kubernetes.io/os
operator: In
values:
- linux
podAffinity: # Pod亲和性
preferredDuringSchedulingIgnoredDuringExecution: # 软性偏好
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S1
topologyKey: topology.kubernetes.io/zone
podAntiAffinity: # Pod反亲和性
requiredDuringSchedulingIgnoredDuringExecution: # 硬性要求
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx
topologyKey: kubernetes.io/hostname
tolerations: # 容忍
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
# 容器配置
containers: # 容器列表
- name: nginx-container # 容器名称
image: nginx:1.25-alpine # 容器镜像
imagePullPolicy: IfNotPresent # 镜像拉取策略：Always, IfNotPresent, Never
# 环境变量
env: # 环境变量
- name: ENV_VAR_1 # 变量名
value: "value1" # 变量值
- name: ENV_VAR_2
valueFrom: # 从其他资源获取
configMapKeyRef: # 从ConfigMap获取
name: app-config
key: database.host
- name: ENV_VAR_3
valueFrom:
secretKeyRef: # 从Secret获取
name: app-secret
key: password
# 端口配置
ports: # 端口定义
- name: http # 端口名称
containerPort: 80 # 容器端口
protocol: TCP # 协议：TCP, UDP, SCTP
- name: https
containerPort: 443
protocol: TCP
# 资源限制
resources: # 资源请求和限制
requests: # 请求资源（调度依据）
memory: "64Mi" # 内存：64MB
cpu: "250m" # CPU：250毫核（0.25核）
ephemeral-storage: "1Gi" # 临时存储
limits: # 资源限制（运行限制）
memory: "128Mi" # 内存限制
cpu: "500m" # CPU限制
ephemeral-storage: "2Gi" # 临时存储限制
hugepages-2Mi: "100Mi" # 大页内存
# 健康检查
livenessProbe: # 存活探针
httpGet: # HTTP检查
path: /healthz # 检查路径
port: 80 # 检查端口
scheme: HTTP # 协议
httpHeaders: # HTTP头
- name: Custom-Header
value: Awesome
initialDelaySeconds: 30 # 初始延迟
periodSeconds: 10 # 检查间隔
timeoutSeconds: 5 # 超时时间
successThreshold: 1 # 成功阈值
failureThreshold: 3 # 失败阈值
readinessProbe: # 就绪探针
tcpSocket: # TCP检查
port: 80
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
startupProbe: # 启动探针（K8s 1.16+）
exec: # 执行命令检查
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 30
# 生命周期钩子
lifecycle: # 生命周期
postStart: # 容器启动后执行
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
preStop: # 容器停止前执行
httpGet:
path: /shutdown
port: 80
# 安全配置
securityContext: # 安全上下文
runAsUser: 1000 # 以指定用户ID运行
runAsGroup: 3000 # 以指定组ID运行
allowPrivilegeEscalation: false # 禁止权限提升
capabilities: # 能力配置
drop: # 删除的能力
- ALL
add: # 添加的能力
- NET_BIND_SERVICE
readOnlyRootFilesystem: true # 只读根文件系统
runAsNonRoot: true # 以非root用户运行
seccompProfile: # seccomp配置
type: RuntimeDefault
# 挂载点
volumeMounts: # 卷挂载
- name: config-volume # 卷名称
mountPath: /etc/nginx # 挂载路径
readOnly: true # 只读
- name: data-volume
mountPath: /var/log/nginx
- name: secret-volume
mountPath: /etc/secret
readOnly: true
# 命令和参数
command: ["nginx"] # 容器启动命令
args: ["-g", "daemon off;"] # 命令参数
# 工作目录
workingDir: /usr/share/nginx/html
# 标准输入
stdin: true # 开启标准输入
stdinOnce: true # 只开启一次
# 终端
tty: true # 分配TTY
# 初始化容器
initContainers: # 初始化容器（在主容器前运行）
- name: init-db
image: busybox:1.28
command: ['sh', '-c', 'until nslookup mysql-service; do echo waiting for mysql; sleep 2; done;']
resources:
requests:
memory: "16Mi"
cpu: "100m"
limits:
memory: "32Mi"
cpu: "200m"
# 卷定义
volumes: # 卷列表
- name: config-volume # ConfigMap卷
configMap:
name: nginx-config
items: # 指定键
- key: nginx.conf
path: nginx.conf
defaultMode: 0644 # 默认权限
- name: secret-volume # Secret卷
secret:
secretName: nginx-secret
- name: data-volume # 空目录卷
emptyDir: {}
- name: host-path-volume # 主机路径卷
hostPath:
path: /var/log/nginx
type: DirectoryOrCreate
- name: pvc-volume # PVC卷
persistentVolumeClaim:
claimName: nginx-pvc
# 重启策略
restartPolicy: Always # Always, OnFailure, Never
# 服务账户
serviceAccountName: default
# 自动挂载服务账户令牌
automountServiceAccountToken: true
# 节点名称
nodeName: node-1 # 直接指定节点
# 主机网络模式
hostNetwork: false # 使用主机网络
# 主机PID模式
hostPID: false # 使用主机PID命名空间
# 主机IPC模式
hostIPC: false # 使用主机IPC命名空间
# DNS策略
dnsPolicy: ClusterFirst # ClusterFirst, ClusterFirstWithHostNet, Default, None
# DNS配置
dnsConfig:
nameservers:
- 1.2.3.4
searches:
- ns1.svc.cluster-domain.example
- my.dns.search.suffix
options:
- name: ndots
value: "2"
- name: edns0
# 安全配置
securityContext: # Pod安全上下文
fsGroup: 2000 # 文件系统组
runAsGroup: 3000 # 运行时组
runAsUser: 1000 # 运行时用户
supplementalGroups: # 补充组
- 1001
sysctls: # 系统调用
- name: net.core.somaxconn
value: "1024"
seLinuxOptions: # SELinux选项
level: "s0:c123,c456"
windowsOptions: # Windows选项
gmsaCredentialSpecName: my-gmsa
# 优先级
priorityClassName: high-priority
# 拓扑约束
topologySpreadConstraints: # 拓扑分布约束
- maxSkew: 1 # 最大偏斜度
topologyKey: zone # 拓扑键
whenUnsatisfiable: DoNotSchedule # 不满足时处理
labelSelector: # 标签选择器
matchLabels:
app: nginx
# 运行时类
runtimeClassName: gvisor
# 开销
overhead: # 运行时开销
cpu: "100m"
memory: "128Mi"
# 临时容器
ephemeralContainers: # 临时容器（调试用）
- name: debug-container
image: busybox
command: ["sh"]
stdin: true
tty: true
targetContainerName: nginx-container
验证配置：
# 应用配置
$ kubectl apply -f pod-complete.yaml
pod/complete-pod created
# 验证Pod状态
$ kubectl get pod complete-pod
NAME READY STATUS RESTARTS AGE
complete-pod 1/1 Running 0 30s
# 查看详细状态
$ kubectl describe pod complete-pod
Name: complete-pod
Namespace: default
Priority: 0
Service Account: default
Node: node-1/192.168.1.101
Start Time: Mon, 01 Jan 2024 10:00:00 +0800
Labels: app=nginx
environment=production
version=1.25
Annotations: kubernetes.io/description: Complete Pod example
created-by: kubectl apply
maintainer: ops-team
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Containers:
nginx-container:
Container ID: containerd://abc123...
Image: nginx:1.25-alpine
Image ID: docker.io/library/nginx@sha256:...
Ports: 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 01 Jan 2024 10:00:10 +0800
Ready: True
Restart Count: 0
Liveness: http-get http://:80/healthz delay=30s timeout=5s period=10s #success=1 #failure=3
Readiness: tcp-socket :80 delay=5s timeout=1s period=5s #success=1 #failure=1
Startup: exec [cat /tmp/healthy] delay=10s timeout=1s period=5s #success=1 #failure=30
Limits:
cpu: 500m
memory: 128Mi
ephemeral-storage: 2Gi
Requests:
cpu: 250m
memory: 64Mi
ephemeral-storage: 1Gi
Environment:
ENV_VAR_1: value1
ENV_VAR_2: <set to the key 'database.host' of config map 'app-config'>
ENV_VAR_3: <set to the key 'password' in secret 'app-secret'>
Mounts:
/etc/nginx from config-volume (ro)
/etc/secret from secret-volume (ro)
/var/log/nginx from data-volume (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nginx-config
Optional: false
secret-volume:
Type: Secret (a volume populated by a Secret)
SecretName: nginx-secret
Optional: false
data-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: disktype=ssd
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 60s default-scheduler Successfully assigned default/complete-pod to node-1
Normal Pulling 59s kubelet Pulling image "busybox:1.28"
Normal Pulled 50s kubelet Successfully pulled image "busybox:1.28" in 9.123456789s
Normal Created 49s kubelet Created container init-db
Normal Started 49s kubelet Started container init-db
Normal Pulling 40s kubelet Pulling image "nginx:1.25-alpine"
Normal Pulled 30s kubelet Successfully pulled image "nginx:1.25-alpine" in 10.123456789s
Normal Created 29s kubelet Created container nginx-container
Normal Started 29s kubelet Started container nginx-container
8.3 Deployment完整配置
# deployment-complete.yaml
apiVersion: apps/v1 # API版本
kind: Deployment # 资源类型
metadata: # 元数据
name: nginx-deployment # Deployment名称
namespace: production # 命名空间
labels: # 标签
app: nginx
environment: production
annotations: # 注解
deployment.kubernetes.io/revision: "1" # 修订版本
kubernetes.io/change-cause: "Initial deployment" # 变更原因
spec: # 规格
# 副本数
replicas: 3 # 期望的Pod副本数
# 选择器
selector: # 标签选择器
matchLabels: # 匹配标签
app: nginx
environment: production
matchExpressions: # 匹配表达式
- key: version
operator: In
values:
- v1
- v2
# 策略
strategy: # 更新策略
type: RollingUpdate # 滚动更新
rollingUpdate: # 滚动更新配置
maxSurge: 1 # 最大激增Pod数（可以是数字或百分比）
maxUnavailable: 0 # 最大不可用Pod数
# 最小就绪时间
minReadySeconds: 10 # Pod就绪后等待时间（秒）
# 修订历史限制
revisionHistoryLimit: 10 # 保留的历史版本数
# 进度期限
progressDeadlineSeconds: 600 # 部署超时时间（秒）
# 接续Deployment完整配置
```yaml
# 暂停部署
paused: false # 是否暂停部署
# Pod模板
template: # Pod模板
metadata: # Pod元数据
labels: # Pod标签
app: nginx
environment: production
version: "1.25"
annotations: # Pod注解
prometheus.io/scrape: "true"
prometheus.io/port: "80"
spec: # Pod规格
containers:
- name: nginx
image: nginx:1.25-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
periodSeconds: 5
验证部署：
# 应用Deployment配置
$ kubectl apply -f deployment-complete.yaml
deployment.apps/nginx-deployment created
# 查看部署状态
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 10s
$ kubectl describe deployment nginx-deployment
Name: nginx-deployment
Namespace: production
CreationTimestamp: Mon, 01 Jan 2024 10:00:00 +0800
Labels: app=nginx
environment=production
Annotations: deployment.kubernetes.io/revision: 1
kubernetes.io/change-cause: Initial deployment
Selector: app=nginx,environment=production
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 10
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=nginx
environment=production
version=1.25
Annotations: prometheus.io/port: 80
prometheus.io/scrape: true
Containers:
nginx:
Image: nginx:1.25-alpine
Port: 80/TCP
Liveness: http-get http://:80/ delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:80/ delay=5s timeout=1s period=5s #success=1 #failure=3
Environment: <none>
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-7c8d9f5c8 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 10s deployment-controller Scaled up replica set nginx-deployment-7c8d9f5c8 to 3
9. 企业级配置模板
9.1 生产环境Web应用配置
# web-app-production.yaml
---
# 命名空间
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
name: production
environment: prod
managed-by: kubernetes
---
# 服务账户
apiVersion: v1
kind: ServiceAccount
metadata:
name: web-app-sa
namespace: production
labels:
app: web-app
environment: production
automountServiceAccountToken: false
---
# 角色绑定
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: web-app-role
namespace: production
rules:
- apiGroups: [""]
resources: ["pods", "services", "endpoints", "persistentvolumeclaims", "secrets", "configmaps"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: web-app-role-binding
namespace: production
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: web-app-role
subjects:
- kind: ServiceAccount
name: web-app-sa
namespace: production
---
# 配置映射
apiVersion: v1
kind: ConfigMap
metadata:
name: web-app-config
namespace: production
labels:
app: web-app
environment: production
data:
# 应用配置
application.properties: |
server.port=8080
server.servlet.context-path=/
# 数据库配置
spring.datasource.url=jdbc:mysql://mysql-primary:3306/webapp
spring.datasource.username=${DB_USERNAME}
spring.datasource.password=${DB_PASSWORD}
# Redis配置
spring.redis.host=redis-master
spring.redis.port=6379
spring.redis.password=${REDIS_PASSWORD}
# 日志配置
logging.level.root=INFO
logging.level.com.example=DEBUG
logging.file.name=/var/log/web-app/app.log
logging.file.max-size=10MB
logging.file.max-history=30
# 监控配置
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.health.show-details=always
# Nginx配置
nginx.conf: |
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
client_max_body_size 20M;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types
text/plain
text/css
text/xml
text/javascript
application/json
application/javascript
application/xml+rss
application/atom+xml
image/svg+xml;
include /etc/nginx/conf.d/*.conf;
}
---
# 密钥
apiVersion: v1
kind: Secret
metadata:
name: web-app-secrets
namespace: production
labels:
app: web-app
environment: production
type: Opaque
stringData:
# 数据库凭据
db-username: "webapp_user"
db-password: "SecurePass123!"
# Redis密码
redis-password: "RedisPass456!"
# JWT密钥
jwt-secret: "my-super-secret-jwt-key-for-production-2024"
# API密钥
api-key: "prod_ak_sk_1234567890abcdef"
# 加密盐
encryption-salt: "production-salt-2024"
---
# 部署配置
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
namespace: production
labels:
app: web-app
component: backend
environment: production
version: v1.2.0
annotations:
deployment.kubernetes.io/revision: "1"
kubernetes.io/change-cause: "Initial deployment of web-app v1.2.0"
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/actuator/prometheus"
spec:
# 副本策略
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
# 就绪等待
minReadySeconds: 30
# 修订历史
revisionHistoryLimit: 10
# 进度期限
progressDeadlineSeconds: 600
# 选择器
selector:
matchLabels:
app: web-app
component: backend
# Pod模板
template:
metadata:
labels:
app: web-app
component: backend
environment: production
version: v1.2.0
annotations:
# 监控注解
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/actuator/prometheus"
# 链路追踪
sidecar.istio.io/inject: "true"
# 时区配置
timezone: "Asia/Shanghai"
# 安全策略
seccomp.security.alpha.kubernetes.io/pod: "runtime/default"
apparmor.security.beta.kubernetes.io/pod: "runtime/default"
spec:
# 服务账户
serviceAccountName: web-app-sa
automountServiceAccountToken: true
# 节点选择
nodeSelector:
node-type: application
disk-type: ssd
# 亲和性
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-type
operator: In
values:
- application
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- web-app
topologyKey: kubernetes.io/hostname
# 容忍
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300
# 容器定义
containers:
- name: web-app
image: myregistry.example.com/web-app:v1.2.0
imagePullPolicy: IfNotPresent
# 端口
ports:
- name: http
containerPort: 8080
protocol: TCP
- name: jmx
containerPort: 9090
protocol: TCP
# 环境变量
env:
- name: JAVA_OPTS
value: "-Xms512m -Xmx1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"
- name: TZ
value: "Asia/Shanghai"
- name: APP_ENV
value: "production"
# 从ConfigMap和Secret获取环境变量
envFrom:
- configMapRef:
name: web-app-config
- secretRef:
name: web-app-secrets
# 资源限制
resources:
requests:
memory: "512Mi"
cpu: "250m"
ephemeral-storage: "1Gi"
limits:
memory: "1024Mi"
cpu: "1000m"
ephemeral-storage: "2Gi"
hugepages-2Mi: "100Mi"
# 健康检查
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
scheme: HTTP
httpHeaders:
- name: Custom-Health-Check
value: "K8s-Liveness"
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
scheme: HTTP
httpHeaders:
- name: Custom-Health-Check
value: "K8s-Readiness"
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
startupProbe:
httpGet:
path: /actuator/health/startup
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 30
# 生命周期
lifecycle:
postStart:
exec:
command:
- "/bin/sh"
- "-c"
- |
echo "Application started at $(date)" > /tmp/app-start-time
echo "Environment: production" >> /tmp/app-start-time
preStop:
httpGet:
path: /actuator/shutdown
port: 8080
scheme: HTTP
# 安全上下文
securityContext:
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
seccompProfile:
type: RuntimeDefault
# 卷挂载
volumeMounts:
- name: config-volume
mountPath: /app/config
readOnly: true
- name: secret-volume
mountPath: /app/secrets
readOnly: true
- name: logs-volume
mountPath: /var/log/web-app
- name: tmp-volume
mountPath: /tmp
# 命令和参数
command: ["java"]
args:
- "-jar"
- "/app/web-app.jar"
- "--spring.config.location=/app/config/application.properties"
# 工作目录
workingDir: /app
# 标准输入
stdin: false
# 终端
tty: false
# 初始化容器
initContainers:
- name: init-config
image: busybox:1.35
command: ['sh', '-c', 'echo "Waiting for dependencies..." && sleep 5']
resources:
requests:
memory: "16Mi"
cpu: "10m"
limits:
memory: "32Mi"
cpu: "50m"
securityContext:
runAsUser: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
# 临时容器（调试用）
ephemeralContainers:
- name: debug-container
image: busybox:1.35
command: ["sh"]
stdin: true
tty: true
targetContainerName: web-app
# 卷定义
volumes:
- name: config-volume
configMap:
name: web-app-config
items:
- key: application.properties
path: application.properties
defaultMode: 0644
- name: secret-volume
secret:
secretName: web-app-secrets
- name: logs-volume
emptyDir:
sizeLimit: 10Gi
- name: tmp-volume
emptyDir:
sizeLimit: 1Gi
# 优先级
priorityClassName: high-priority
# 拓扑约束
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: web-app
# 安全上下文
securityContext:
fsGroup: 1000
runAsGroup: 1000
runAsUser: 1000
supplementalGroups:
- 1001
sysctls:
- name: net.core.somaxconn
value: "1024"
seLinuxOptions:
level: "s0:c123,c456"
# DNS配置
dnsConfig:
nameservers:
- 8.8.8.8
- 8.8.4.4
searches:
- production.svc.cluster.local
- svc.cluster.local
- cluster.local
options:
- name: ndots
value: "5"
- name: single-request-reopen
---
# 服务配置
apiVersion: v1
kind: Service
metadata:
name: web-app-service
namespace: production
labels:
app: web-app
component: backend
environment: production
annotations:
# 负载均衡器注解
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-internal: "false"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-east-1:123456789012:certificate/abc123"
# 监控注解
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
# 健康检查
alb.ingress.kubernetes.io/healthcheck-path: "/actuator/health"
alb.ingress.kubernetes.io/healthcheck-port: "8080"
spec:
# 服务类型
type: ClusterIP
# 对于生产环境，通常使用LoadBalancer
# type: LoadBalancer
# 选择器
selector:
app: web-app
component: backend
# 端口配置
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
- name: https
port: 443
targetPort: 8080
protocol: TCP
- name: metrics
port: 9090
targetPort: 9090
protocol: TCP
# 会话亲和性
sessionAffinity: None
# 内部流量策略
internalTrafficPolicy: Cluster
# IP族
ipFamilies:
- IPv4
- IPv6
# IP族策略
ipFamilyPolicy: RequireDualStack
# 外部流量策略
externalTrafficPolicy: Cluster
---
# 水平Pod自动扩缩容
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web-app-hpa
namespace: production
labels:
app: web-app
environment: production
spec:
# 目标引用
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-app
# 副本范围
minReplicas: 3
maxReplicas: 10
# 扩缩容行为
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
- type: Pods
value: 1
periodSeconds: 60
selectPolicy: Min
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
- type: Pods
value: 4
periodSeconds: 15
selectPolicy: Max
# 指标
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: 1000
- type: Object
object:
metric:
name: requests-per-second
describedObject:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: web-app-ingress
target:
type: Value
value: 10k
---
# 网络策略
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-app-network-policy
namespace: production
labels:
app: web-app
environment: production
spec:
# 策略目标
podSelector:
matchLabels:
app: web-app
component: backend
# 策略类型
policyTypes:
- Ingress
- Egress
# 入站规则
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
- podSelector:
matchLabels:
app: ingress-nginx
ports:
- protocol: TCP
port: 8080
# 出站规则
egress:
- to:
- podSelector:
matchLabels:
app: mysql
ports:
- protocol: TCP
port: 3306
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- protocol: TCP
port: 443
- protocol: TCP
port: 80
---
# 持久化存储声明
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: web-app-data-pvc
namespace: production
labels:
app: web-app
environment: production
storage-tier: standard
spec:
# 访问模式
accessModes:
- ReadWriteOnce
# 访问模式选项：
# - ReadWriteOnce: 可被单个节点读写
# - ReadOnlyMany: 可被多个节点只读
# - ReadWriteMany: 可被多个节点读写
# - ReadWriteOncePod: 可被单个Pod读写 (K8s 1.22+)
# 存储类
storageClassName: ssd-standard
# 资源请求
resources:
requests:
storage: 100Gi
# 卷模式
volumeMode: Filesystem
# 选择器
selector:
matchLabels:
type: ssd
environment: production
---
# Ingress配置
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-app-ingress
namespace: production
labels:
app: web-app
environment: production
annotations:
# Nginx Ingress注解
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "20m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
# 速率限制
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/limit-burst: "200"
nginx.ingress.kubernetes.io/limit-connections: "1000"
# 跨域
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-max-age: "1728000"
# 认证
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
# 监控
nginx.ingress.kubernetes.io/enable-access-log: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Request-ID: $request_id";
more_set_headers "X-Cache-Status: $upstream_cache_status";
# 缓存
nginx.ingress.kubernetes.io/enable-modsecurity: "true"
nginx.ingress.kubernetes.io/enable-owasp-core-rules: "true"
# WebSocket支持
nginx.ingress.kubernetes.io/websocket-services: "web-app-service"
# 重写
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/use-regex: "true"
# 灰度发布
nginx.ingress.kubernetes.io/canary: "false"
nginx.ingress.kubernetes.io/canary-weight: "0"
spec:
# Ingress类
ingressClassName: nginx
# TLS配置
tls:
- hosts:
- app.example.com
- api.example.com
secretName: tls-secret
# 规则
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web-app-service
port:
number: 80
- path: /api
pathType: Prefix
backend:
service:
name: web-app-service
port:
number: 80
- path: /static
pathType: Prefix
backend:
service:
name: web-app-service
port:
number: 80
- host: api.example.com
http:
paths:
- path: /v1
pathType: Prefix
backend:
service:
name: web-app-service
port:
number: 80
- path: /v2
pathType: Prefix
backend:
service:
name: web-app-service
port:
number: 80
---
# Pod Disruption Budget
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-app-pdb
namespace: production
labels:
app: web-app
environment: production
spec:
# 选择器
selector:
matchLabels:
app: web-app
component: backend
# 最大不可用
maxUnavailable: 1
# 或最小可用
# minAvailable: 2
应用完整配置：
# 创建生产环境命名空间
$ kubectl apply -f web-app-production.yaml
namespace/production created
serviceaccount/web-app-sa created
role.rbac.authorization.k8s.io/web-app-role created
rolebinding.rbac.authorization.k8s.io/web-app-role-binding created
configmap/web-app-config created
secret/web-app-secrets created
deployment.apps/web-app created
service/web-app-service created
horizontalpodautoscaler.autoscaling/web-app-hpa created
networkpolicy.networking.k8s.io/web-app-network-policy created
persistentvolumeclaim/web-app-data-pvc created
ingress.networking.k8s.io/web-app-ingress created
poddisruptionbudget.policy/web-app-pdb created
# 验证所有资源
$ kubectl get all -n production
NAME READY STATUS RESTARTS AGE
pod/web-app-7c8d9f5c8-abc12 1/1 Running 0 30s
pod/web-app-7c8d9f5c8-def34 1/1 Running 0 30s
pod/web-app-7c8d9f5c8-ghi56 1/1 Running 0 30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/web-app-service ClusterIP 10.96.123.123 <none> 80/TCP,443/TCP,9090/TCP 30s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web-app 3/3 3 3 30s
NAME DESIRED CURRENT READY AGE
replicaset.apps/web-app-7c8d9f5c8 3 3 3 30s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/web-app-hpa Deployment/web-app 0%/70% 3 10 3 30s
# 查看网络策略
$ kubectl get networkpolicies -n production
NAME POD-SELECTOR AGE
web-app-network-policy app=web-app,component=backend 30s
# 查看PDB
$ kubectl get pdb -n production
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
web-app-pdb N/A 1 1 30s
# 查看PVC
$ kubectl get pvc -n production
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
web-app-data-pvc Bound pvc-abc123-def4-5678-90ab-cdef01234567 100Gi RWO ssd-standard 30s
# 查看Ingress
$ kubectl get ingress -n production
NAME CLASS HOSTS ADDRESS PORTS AGE
web-app-ingress nginx app.example.com,api.example.com 80, 443 30s
10. 工作原理：深入理解控制平面
10.1 API Server工作流程
# 查看API Server配置
$ ps aux | grep kube-apiserver
root 1234 2.5 5.2 1234567 89012 ? Ssl 10:00 0:15 kube-apiserver \
--advertise-address=192.168.1.100 \
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--client-ca-file=/etc/kubernetes/pki/ca.crt \
--enable-admission-plugins=NodeRestriction \
--enable-bootstrap-token-auth=true \
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
--etcd-servers=https://127.0.0.1:2379 \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
--requestheader-allowed-names=front-proxy-client \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--secure-port=6443 \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-cluster-ip-range=10.96.0.0/12 \
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key
# API Server的认证流程
1. 客户端发送请求
2. 认证阶段：验证客户端身份
3. 鉴权阶段：检查是否有权限
4. 准入控制：修改/验证请求
5. 验证阶段：验证对象规范
6. etcd操作：写入/读取数据
7. 响应客户端
10.2 Controller Manager工作原理
# 查看Controller Manager配置
$ cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --cluster-cidr=10.244.0.0/16
- --cluster-name=kubernetes
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
- --controllers=*,bootstrapsigner,tokencleaner
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --node-cidr-mask-size=24
- --port=0
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --use-service-account-credentials=true
image: registry.k8s.io/kube-controller-manager:v1.28.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10257
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-controller-manager
resources:
requests:
cpu: 200m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 10257
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
name: flexvolume-dir
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/ca-certificates
type: DirectoryOrCreate
name: etc-ca-certificates
- hostPath:
path: /usr/share/ca-certificates
type: DirectoryOrCreate
name: usr-share-ca-certificates
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
type: DirectoryOrCreate
name: flexvolume-dir
Controller Manager包含的控制器：
| 控制器名称 | 作用 | 工作频率 | 关键特性 |
|---|---|---|---|
| Node Controller | 管理节点状态 | 5s | 节点心跳检测 |
| Replication Controller | 维护Pod副本数 | 10s | 已被Deployment替代 |
| Endpoints Controller | 填充Endpoint对象 | 5s | 服务发现 |
| Service Account & Token Controllers | 创建默认账户和API访问令牌 | 1m | 安全认证 |
| ResourceQuota Controller | 确保不超过资源配额 | 5s | 资源限制 |
| Namespace Controller | 删除命名空间中的对象 | 5s | 命名空间生命周期 |
| Service Controller | 创建/更新LoadBalancer | 5s | 负载均衡器 |
| Route Controller | 配置云提供商路由 | 5s | 云集成 |
| PV-Binder Controller | 绑定PVC和PV | 15s | 存储管理 |
| Attach/Detach Controller | 附加/分离卷 | 1m | 存储卷管理 |
| Certificate Controller | 管理证书 | 1h | 证书轮换 |
10.3 Scheduler调度流程
# 查看Scheduler配置
$ cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-scheduler
tier: control-plane
name: kube-scheduler
namespace: kube-system
spec:
containers:
- command:
- kube-scheduler
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
- --port=0
image: registry.k8s.io/kube-scheduler:v1.28.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10259
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-scheduler
resources:
requests:
cpu: 100m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 10259
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/scheduler.conf
name: kubeconfig
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes/scheduler.conf
type: FileOrCreate
name: kubeconfig
调度器调度流程：
1. 过滤阶段 (Filtering/Predicate)
↓
[检查节点资源是否满足Pod需求]
↓
2. 打分阶段 (Scoring/Priority)
↓
[为通过过滤的节点打分]
↓
3. 绑定阶段 (Binding)
↓
[选择最高分节点，绑定Pod]
详细流程：
1. 预选 (Predicates)
- PodFitsResources: 检查节点资源
- PodFitsHostPorts: 检查主机端口
- PodMatchNodeSelector: 节点选择器
- NoVolumeZoneConflict: 卷区域冲突
- NoDiskConflict: 磁盘冲突
- MaxCSIVolumeCount: CSI卷数量
- CheckNodeMemoryPressure: 内存压力
- CheckNodeDiskPressure: 磁盘压力
- CheckNodePIDPressure: PID压力
- CheckNodeCondition: 节点条件
- PodToleratesNodeTaints: 污点容忍
- CheckVolumeBinding: 卷绑定
- CheckNodeUnschedulable: 节点是否可调度
2. 优选 (Priorities)
- SelectorSpreadPriority: 扩展选择器
- InterPodAffinityPriority: Pod亲和性
- LeastRequestedPriority: 最少请求资源
- MostRequestedPriority: 最多请求资源
- BalancedResourceAllocation: 平衡资源分配
- NodePreferAvoidPodsPriority: 节点偏好
- NodeAffinityPriority: 节点亲和性
- TaintTolerationPriority: 污点容忍
- ImageLocalityPriority: 镜像本地性
- ServiceSpreadingPriority: 服务扩展
10.4 etcd工作原理
# 查看etcd状态
$ kubectl exec -n kube-system etcd-master -- etcdctl endpoint status \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
-w table
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://127.0.0.1:2379 | 8e9e05c52164694d | 3.5.0 | 20 MB | true | false | 2 | 9168 | 9168 | |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
# 查看etcd成员
$ kubectl exec -n kube-system etcd-master -- etcdctl member list \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
-w table
+------------------+---------+--------+------------------------+------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+--------+------------------------+------------------------+------------+
| 8e9e05c52164694d | started | master | https://192.168.1.100:2380 | https://192.0.2.1:2379 | false |
+------------------+---------+--------+------------------------+------------------------+------------+
# 查看etcd中存储的Pod数据
$ kubectl exec -n kube-system etcd-master -- etcdctl get /registry/pods/default/nginx --prefix \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
| head -50
11. 调度机制：Pod是如何被分配的？
11.1 节点选择器
apiVersion: v1
kind: Pod
metadata:
name: nginx-node-selector
spec:
nodeSelector:
disktype: ssd
gpu: "true"
containers:
- name: nginx
image: nginx:1.25
11.2 亲和性与反亲和性
apiVersion: v1
kind: Pod
metadata:
name: nginx-affinity
spec:
affinity:
# 节点亲和性
nodeAffinity:
# 必须满足的条件
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- amd64
# 偏好条件
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: disktype
operator: In
values:
- ssd
- weight: 50
preference:
matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- zone-a
# Pod亲和性
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S1
topologyKey: topology.kubernetes.io/zone
# Pod反亲和性
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx
topologyKey: kubernetes.io/hostname
containers:
- name: nginx
image: nginx:1.25
验证调度结果：
# 查看Pod调度到的节点
$ kubectl get pod nginx-affinity -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-affinity 1/1 Running 0 10s 10.244.1.5 node-2 <none> <none>
# 查看节点标签
$ kubectl describe node node-2
Name: node-2
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
disktype=ssd
kubernetes.io/arch=amd64
kubernetes.io/hostname=node-2
kubernetes.io/os=linux
topology.kubernetes.io/zone=zone-a
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
11.3 污点和容忍
# 给节点添加污点
$ kubectl taint nodes node-1 key1=value1:NoSchedule
node/node-1 tainted
$ kubectl taint nodes node-1 key1=value1:NoExecute
node/node-1 tainted
$ kubectl taint nodes node-1 key2=value2:PreferNoSchedule
node/node-1 tainted
# 查看节点污点
$ kubectl describe node node-1 | grep Taints
Taints: key1=value1:NoExecute, key1=value1:NoSchedule, key2=value2:PreferNoSchedule
# 创建带有容忍的Pod
apiVersion: v1
kind: Pod
metadata:
name: nginx-toleration
spec:
tolerations:
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule"
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoExecute"
- key: "key2"
operator: "Exists"
effect: "PreferNoSchedule"
containers:
- name: nginx
image: nginx:1.25
11.4 拓扑分布约束
apiVersion: v1
kind: Pod
metadata:
name: nginx-topology
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: nginx
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: nginx
containers:
- name: nginx
image: nginx:1.25
12. 网络模型：Service和Ingress全解
12.1 Service类型详解
1. ClusterIP（默认）
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: ClusterIP
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
2. NodePort
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: NodePort
selector:
app: MyApp
ports:
- port: 80
targetPort: 80
nodePort: 30007
3. LoadBalancer
apiVersion: v1
kind: Service
metadata:
name: my-service
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
type: LoadBalancer
selector:
app: MyApp
ports:
- port: 80
targetPort: 80
4. ExternalName
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: ExternalName
externalName: my.database.example.com
验证Service：
# 创建Service
$ kubectl apply -f service.yaml
service/my-service created
# 查看Service
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-service LoadBalancer 10.96.123.123 192.168.1.100 80:30007/TCP 10s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
# 查看Service详情
$ kubectl describe service my-service
Name: my-service
Namespace: default
Labels: <none>
Annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector: app=MyApp
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.96.123.123
IPs: 10.96.123.123
LoadBalancer Ingress: 192.168.1.100
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30007/TCP
Endpoints: 10.244.1.2:80,10.244.2.3:80,10.244.3.4:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 10s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 5s service-controller Ensured load balancer
# 查看Endpoint
$ kubectl get endpoints my-service
NAME ENDPOINTS AGE
my-service 10.244.1.2:80,10.244.2.3:80,10.244.3.4:80 15s
12.2 Ingress配置深度解析
# ingress-advanced.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: advanced-ingress
namespace: production
annotations:
# Nginx Ingress Controller注解
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "20m"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
nginx.ingress.kubernetes.io/proxy-buffers: "4 16k"
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
# 认证配置
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
# 跨域配置
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, PATCH, OPTIONS"
nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-max-age: "1728000"
# 速率限制
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/limit-rpm: "6000"
nginx.ingress.kubernetes.io/limit-burst: "200"
nginx.ingress.kubernetes.io/limit-connections: "1000"
nginx.ingress.kubernetes.io/limit-whitelist: "10.0.0.0/8"
# WebSocket支持
nginx.ingress.kubernetes.io/websocket-services: "websocket-service"
nginx.ingress.kubernetes.io/websocket-read-timeout: "3600"
nginx.ingress.kubernetes.io/websocket-send-timeout: "3600"
# 会话保持
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
nginx.ingress.kubernetes.io/session-cookie-path: "/"
nginx.ingress.kubernetes.io/session-cookie-samesite: "Strict"
# 灰度发布
nginx.ingress.kubernetes.io/canary: "false"
nginx.ingress.kubernetes.io/canary-weight: "0"
nginx.ingress.kubernetes.io/canary-by-header: "canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "always"
nginx.ingress.kubernetes.io/canary-by-cookie: "canary"
# 监控
nginx.ingress.kubernetes.io/enable-access-log: "true"
nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Request-ID: $request_id";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-Frame-Options: DENY";
more_set_headers "X-XSS-Protection: 1; mode=block";
more_set_headers "Referrer-Policy: strict-origin-when-cross-origin";
more_set_headers "Content-Security-Policy: default-src 'self'";
# 安全
nginx.ingress.kubernetes.io/enable-modsecurity: "true"
nginx.ingress.kubernetes.io/enable-owasp-core-rules: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleEngine On
SecRequestBodyAccess On
SecRule REQUEST_HEADERS:Content-Type "text/xml" "id:200000,phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type "application/json" "id:200001,phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
# 缓存
nginx.ingress.kubernetes.io/proxy-cache-path: "/tmp/nginx/cache levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m use_temp_path=off"
nginx.ingress.kubernetes.io/proxy-cache-key: "$scheme$request_method$host$request_uri"
nginx.ingress.kubernetes.io/proxy-cache-valid: "200 302 10m"
nginx.ingress.kubernetes.io/proxy-cache-valid: "404 1m"
nginx.ingress.kubernetes.io/proxy-cache-use-stale: "error timeout updating http_500 http_502 http_503 http_504"
# 压缩
nginx.ingress.kubernetes.io/gzip: "on"
nginx.ingress.kubernetes.io/gzip-types: "text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript"
nginx.ingress.kubernetes.io/gzip-comp-level: "6"
nginx.ingress.kubernetes.io/gzip-min-length: "256"
nginx.ingress.kubernetes.io/gzip-proxied: "any"
nginx.ingress.kubernetes.io/gzip-vary: "on"
# 上游配置
nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri"
nginx.ingress.kubernetes.io/load-balance: "ewma"
nginx.ingress.kubernetes.io/upstream-keepalive-connections: "32"
nginx.ingress.kubernetes.io/upstream-keepalive-requests: "100"
nginx.ingress.kubernetes.io/upstream-keepalive-timeout: "60s"
# 自定义错误页面
nginx.ingress.kubernetes.io/custom-http-errors: "404,500,502,503,504"
nginx.ingress.kubernetes.io/default-backend: "default-http-backend"
nginx.ingress.kubernetes.io/custom-headers: "X-Custom-Header: value"
# 地理位置
nginx.ingress.kubernetes.io/geoip-country: "/etc/nginx/geoip/GeoIP.dat"
nginx.ingress.kubernetes.io/geoip-city: "/etc/nginx/geoip/GeoLiteCity.dat"
# 实时配置
nginx.ingress.kubernetes.io/server-snippet: |
location /nginx_status {
stub_status on;
access_log off;
allow 127.0.0.1;
deny all;
}
# SSL配置
nginx.ingress.kubernetes.io/ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
nginx.ingress.kubernetes.io/ssl-protocols: "TLSv1.2 TLSv1.3"
nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
nginx.ingress.kubernetes.io/ssl-session-cache: "true"
nginx.ingress.kubernetes.io/ssl-session-cache-size: "10m"
nginx.ingress.kubernetes.io/ssl-session-timeout: "10m"
# HSTS
nginx.ingress.kubernetes.io/hsts: "true"
nginx.ingress.kubernetes.io/hsts-include-subdomains: "true"
nginx.ingress.kubernetes.io/hsts-max-age: "31536000"
nginx.ingress.kubernetes.io/hsts-preload: "true"
# 重定向
nginx.ingress.kubernetes.io/permanent-redirect: "https://old.example.com$request_uri"
nginx.ingress.kubernetes.io/permanent-redirect-code: "308"
nginx.ingress.kubernetes.io/temporal-redirect: "https://maintenance.example.com"
# 代理协议
nginx.ingress.kubernetes.io/use-proxy-protocol: "true"
nginx.ingress.kubernetes.io/proxy-protocol-header: "X-Forwarded-For"
# 真实IP
nginx.ingress.kubernetes.io/enable-real-ip: "true"
nginx.ingress.kubernetes.io/real-ip-header: "X-Forwarded-For"
nginx.ingress.kubernetes.io/real-ip-recursive: "true"
spec:
# Ingress类
ingressClassName: nginx
# TLS配置
tls:
- hosts:
- app.example.com
- api.example.com
- admin.example.com
secretName: tls-secret-prod
# 规则
rules:
# 主域名规则
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web-app-service
port:
number: 80
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /static
pathType: Prefix
backend:
service:
name: static-service
port:
number: 80
- path: /admin
pathType: Prefix
backend:
service:
name: admin-service
port:
number: 80
# API子域名规则
- host: api.example.com
http:
paths:
- path: /v1
pathType: Prefix
backend:
service:
name: api-v1-service
port:
number: 80
- path: /v2
pathType: Prefix
backend:
service:
name: api-v2-service
port:
number: 80
- path: /v3
pathType: Prefix
backend:
service:
name: api-v3-service
port:
number: 80
# 管理后台规则
- host: admin.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: admin-panel-service
port:
number: 80
- path: /api
pathType: Prefix
backend:
service:
name: admin-api-service
port:
number: 80
# 通配符子域名规则
- host: "*.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wildcard-service
port:
number: 80
# 默认主机规则（无host）
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: default-service
port:
number: 80
- path: /healthz
pathType: Exact
backend:
service:
name: health-check-service
port:
number: 80
- path: /ready
pathType: Exact
backend:
service:
name: ready-check-service
port:
number: 80
# 重写规则
- host: old.example.com
http:
paths:
- path: /old-path
pathType: Prefix
backend:
service:
name: redirect-service
port:
number: 80
# 路径重写规则
- host: rewrite.example.com
http:
paths:
- path: /something(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: rewrite-service
port:
number: 80
验证Ingress：
# 应用Ingress配置
$ kubectl apply -f ingress-advanced.yaml
ingress.networking.k8s.io/advanced-ingress created
# 查看Ingress状态
$ kubectl get ingress -n production
NAME CLASS HOSTS ADDRESS PORTS AGE
advanced-ingress nginx app.example.com,api.example.com,admin.example.com,*.example.com 192.168.1.100 80, 443 10s
# 查看Ingress详情
$ kubectl describe ingress advanced-ingress -n production
Name: advanced-ingress
Labels: <none>
Namespace: production
Address: 192.168.1.100
Ingress Class: nginx
Default backend: <default>
TLS:
tls-secret-prod terminates app.example.com,api.example.com,admin.example.com
Rules:
Host Path Backends
---- ---- --------
app.example.com
/ web-app-service:80 (10.244.1.2:80,10.244.2.3:80,10.244.3.4:80)
/api api-service:80 (10.244.1.3:80,10.244.2.4:80,10.244.3.5:80)
/static static-service:80 (10.244.1.4:80,10.244.2.5:80,10.244.3.6:80)
/admin admin-service:80 (10.244.1.5:80,10.244.2.6:80,10.244.3.7:80)
api.example.com
/v1 api-v1-service:80 (10.244.1.6:80,10.244.2.7:80,10.244.3.8:80)
/v2 api-v2-service:80 (10.244.1.7:80,10.244.2.8:80,10.244.3.9:80)
/v3 api-v3-service:80 (10.244.1.8:80,10.244.2.9:80,10.244.3.10:80)
admin.example.com
/ admin-panel-service:80 (10.244.1.9:80,10.244.2.10:80,10.244.3.11:80)
/api admin-api-service:80 (10.244.1.10:80,10.244.2.11:80,10.244.3.12:80)
*.example.com
/ wildcard-service:80 (10.244.1.11:80,10.244.2.12:80,10.244.3.13:80)
/ default-service:80 (10.244.1.12:80,10.244.2.13:80,10.244.3.14:80)
/healthz health-check-service:80 (10.244.1.13:80,10.244.2.14:80,10.244.3.15:80)
/ready ready-check-service:80 (10.244.1.14:80,10.244.2.15:80,10.244.3.16:80)
old.example.com
/old-path redirect-service:80 (10.244.1.15:80,10.244.2.16:80,10.244.3.17:80)
rewrite.example.com
/something(/|$)(.*) rewrite-service:80 (10.244.1.16:80,10.244.2.17:80,10.244.3.18:80)
Annotations: nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
nginx.ingress.kubernetes.io/auth
# 13. 存储管理：PV/PVC深度剖析
## 13.1 存储基础概念
### 13.1.1 存储卷类型对比
| 存储卷类型 | 用途场景 | 生命周期 | 数据持久性 | 访问模式 | 性能 |
|-----------|---------|---------|-----------|---------|------|
| **emptyDir** | 临时存储，Pod内容器共享 | 同Pod | Pod删除即丢失 | ReadWriteOnce | 高 |
| **hostPath** | 节点本地存储 | 持久 | 依赖节点 | ReadWriteOnce | 高 |
| **ConfigMap** | 配置文件 | 持久 | 独立 | ReadOnlyMany | 中 |
| **Secret** | 敏感配置 | 持久 | 独立 | ReadOnlyMany | 中 |
| **PersistentVolumeClaim** | 持久化存储 | 持久 | 独立 | 多种模式 | 取决于后端 |
| **CSI Volume** | 云存储/外部存储 | 持久 | 独立 | 多种模式 | 取决于后端 |
### 13.2 PersistentVolume (PV) 详解
#### 13.2.1 静态PV配置
```yaml
# static-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: static-pv-1
labels:
type: local
storage-tier: ssd
environment: production
annotations:
pv.kubernetes.io/provisioned-by: kubernetes.io/no-provisioner
volume.beta.kubernetes.io/storage-class: "standard"
spec:
# 容量
capacity:
storage: 10Gi # 存储容量
# 访问模式
accessModes:
- ReadWriteOnce # 可被单个节点读写
# - ReadOnlyMany # 可被多个节点只读
# - ReadWriteMany # 可被多个节点读写
# - ReadWriteOncePod # 单个Pod读写 (K8s 1.22+)
# 持久化回收策略
persistentVolumeReclaimPolicy: Retain
# 可选值：
# - Retain: 保留（手动清理）
# - Recycle: 回收（基本擦除）
# - Delete: 删除（删除后端存储）
# 存储类
storageClassName: standard
# 卷模式
volumeMode: Filesystem
# 可选值：
# - Filesystem: 文件系统
# - Block: 块设备
# 节点亲和性
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node-1
# 本地路径（hostPath类型）
hostPath:
path: /mnt/data/static-pv-1
type: DirectoryOrCreate
# hostPath类型：
# - DirectoryOrCreate: 目录不存在则创建
# - Directory: 目录必须存在
# - FileOrCreate: 文件不存在则创建
# - File: 文件必须存在
# - Socket: UNIX socket
# - CharDevice: 字符设备
# - BlockDevice: 块设备
---
# NFS类型PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-1
labels:
type: nfs
storage-tier: standard
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: nfs
volumeMode: Filesystem
nfs:
server: 192.168.1.200
path: "/exports/data"
readOnly: false
---
# AWS EBS类型PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: aws-ebs-pv-1
labels:
type: aws-ebs
storage-tier: gp3
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: gp3
volumeMode: Filesystem
awsElasticBlockStore:
volumeID: vol-0123456789abcdef0
fsType: ext4
readOnly: false
---
# GCE PD类型PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: gce-pd-pv-1
spec:
capacity:
storage: 200Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: pd-standard
gcePersistentDisk:
pdName: my-data-disk
fsType: ext4
---
# Azure Disk类型PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: azure-disk-pv-1
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: managed-premium
azureDisk:
diskName: myManagedDisk
diskURI: /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Compute/disks/<disk-name>
kind: Managed
cachingMode: ReadOnly
fsType: ext4
---
# Ceph RBD类型PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-rbd-pv-1
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: ceph-rbd
volumeMode: Filesystem
rbd:
monitors:
- 192.168.1.201:6789
- 192.168.1.202:6789
- 192.168.1.203:6789
pool: rbd
image: kubernetes-dynamic-pvc-abc123
user: admin
secretRef:
name: ceph-secret
fsType: ext4
readOnly: false
应用和验证PV：
# 创建PV
$ kubectl apply -f static-pv.yaml
persistentvolume/static-pv-1 created
persistentvolume/nfs-pv-1 created
persistentvolume/aws-ebs-pv-1 created
persistentvolume/gce-pd-pv-1 created
persistentvolume/azure-disk-pv-1 created
persistentvolume/ceph-rbd-pv-1 created
# 查看所有PV
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
static-pv-1 10Gi RWO Retain Available standard 10s
nfs-pv-1 100Gi RWX Retain Available nfs 10s
aws-ebs-pv-1 50Gi RWO Delete Available gp3 10s
gce-pd-pv-1 200Gi RWO Delete Available pd-standard 10s
azure-disk-pv-1 100Gi RWO Delete Available managed-premium 10s
ceph-rbd-pv-1 20Gi RWO Delete Available ceph-rbd 10s
# 查看PV详情
$ kubectl describe pv static-pv-1
Name: static-pv-1
Labels: environment=production
storage-tier=ssd
type=local
Annotations: pv.kubernetes.io/provisioned-by: kubernetes.io/no-provisioner
volume.beta.kubernetes.io/storage-class: standard
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [node-1]
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /mnt/data/static-pv-1
HostPathType: DirectoryOrCreate
Events: <none>
13.3 PersistentVolumeClaim (PVC) 详解
13.3.1 PVC配置示例
# pvc-examples.yaml
---
# 基础PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: basic-pvc
namespace: default
labels:
app: database
storage-tier: standard
annotations:
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
volume.kubernetes.io/selected-node: node-1
spec:
# 访问模式
accessModes:
- ReadWriteOnce
# 存储类
storageClassName: standard
# 资源请求
resources:
requests:
storage: 5Gi
# 卷模式
volumeMode: Filesystem
# 选择器（绑定特定PV）
selector:
matchLabels:
storage-tier: ssd
matchExpressions:
- key: environment
operator: In
values:
- production
---
# 动态PVC（自动创建PV）
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dynamic-pvc
namespace: production
labels:
app: web-app
component: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 20Gi
volumeMode: Filesystem
---
# 只读多节点PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: read-only-pvc
namespace: shared
spec:
accessModes:
- ReadOnlyMany
storageClassName: nfs
resources:
requests:
storage: 100Gi
---
# 读写多节点PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: read-write-many-pvc
namespace: shared
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs
resources:
requests:
storage: 50Gi
---
# 块设备PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: block-pvc
namespace: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: ssd-block
resources:
requests:
storage: 100Gi
volumeMode: Block
---
# 带数据源的PVC（克隆/快照）
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cloned-pvc
namespace: production
spec:
accessModes:
- ReadWriteOnce
storageClassName: ssd
resources:
requests:
storage: 10Gi
dataSource:
kind: PersistentVolumeClaim
name: source-pvc
apiGroup: ""
---
# 带扩展的PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: expandable-pvc
namespace: production
spec:
accessModes:
- ReadWriteOnce
storageClassName: expandable-sc
resources:
requests:
storage: 10Gi
应用和验证PVC：
# 创建PVC
$ kubectl apply -f pvc-examples.yaml
persistentvolumeclaim/basic-pvc created
persistentvolumeclaim/dynamic-pvc created
persistentvolumeclaim/read-only-pvc created
persistentvolumeclaim/read-write-many-pvc created
persistentvolumeclaim/block-pvc created
persistentvolumeclaim/cloned-pvc created
persistentvolumeclaim/expandable-pvc created
# 查看PVC状态
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
basic-pvc Bound static-pv-1 10Gi RWO standard 10s
dynamic-pvc Pending gp3 10s
read-only-pvc Pending nfs 10s
read-write-many-pvc Pending nfs 10s
block-pvc Pending ssd-block 10s
cloned-pvc Pending ssd 10s
expandable-pvc Pending expandable-sc 10s
# 查看PVC详情
$ kubectl describe pvc basic-pvc
Name: basic-pvc
Namespace: default
StorageClass: standard
Status: Bound
Volume: static-pv-1
Labels: app=database
storage-tier=standard
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
volume.kubernetes.io/selected-node: node-1
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 10s persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 9s persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubernetes.io/aws-ebs" or manually created by system administrator
Normal Provisioning 8s ebs.csi.aws.com External provisioner is provisioning volume for claim "default/basic-pvc"
Normal ProvisioningSucceeded 7s ebs.csi.aws.com Successfully provisioned volume pvc-abc123-def4-5678-90ab-cdef01234567
13.4 StorageClass配置
13.4.1 StorageClass示例
# storage-class-examples.yaml
---
# AWS EBS gp3存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
labels:
kubernetes.io/cluster-service: "true"
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
type: gp3
encrypted: "true"
kmsKeyId: alias/aws/ebs
iops: "3000"
throughput: "125"
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
- key: topology.kubernetes.io/zone
values:
- us-west-2a
- us-west-2b
- us-west-2c
---
# AWS EBS io2存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: io2
provisioner: ebs.csi.aws.com
parameters:
type: io2
iopsPerGB: "100"
encrypted: "true"
csi.storage.k8s.io/fstype: xfs
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# 本地存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
---
# NFS存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
archiveOnDelete: "false"
pathPattern: "${.PVC.namespace}/${.PVC.name}"
onDelete: retain
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate
---
# Ceph RBD存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd
provisioner: rbd.csi.ceph.com
parameters:
clusterID: 01234567-89ab-cdef-0123-456789abcdef
pool: kube
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: kube-system
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
# 带快照的存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: snapshot-enabled
provisioner: ebs.csi.aws.com
parameters:
type: gp3
encrypted: "true"
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
- key: topology.ebs.csi.aws.com/zone
values:
- us-west-2a
应用StorageClass：
# 创建StorageClass
$ kubectl apply -f storage-class-examples.yaml
storageclass.storage.k8s.io/gp3 created
storageclass.storage.k8s.io/io2 created
storageclass.storage.k8s.io/local-storage created
storageclass.storage.k8s.io/nfs-client created
storageclass.storage.k8s.io/ceph-rbd created
storageclass.storage.k8s.io/snapshot-enabled created
# 查看StorageClass
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 10s
io2 ebs.csi.aws.com Delete WaitForFirstConsumer true 10s
local-storage kubernetes.io/no-provisioner Retain WaitForFirstConsumer false 10s
nfs-client k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate true 10s
ceph-rbd rbd.csi.ceph.com Delete WaitForFirstConsumer true 10s
snapshot-enabled ebs.csi.aws.com Delete WaitForFirstConsumer true 10s
# 查看StorageClass详情
$ kubectl describe storageclass gp3
Name: gp3
IsDefaultClass: Yes
Annotations: storageclass.kubernetes.io/is-default-class=true
Provisioner: ebs.csi.aws.com
Parameters: csi.storage.k8s.io/fstype=ext4,encrypted=true,iops=3000,kmsKeyId=alias/aws/ebs,throughput=125,type=gp3
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
AllowedTopologies:
Match Label Expressions:
topology.kubernetes.io/zone In [us-west-2a, us-west-2b, us-west-2c]
Events: <none>
13.5 使用PVC的Pod示例
13.5.1 有状态应用示例
# stateful-app-with-pvc.yaml
---
# MySQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: database
labels:
app: mysql
component: database
spec:
# 服务名称
serviceName: mysql
# 副本数
replicas: 3
# 选择器
selector:
matchLabels:
app: mysql
# 更新策略
updateStrategy:
type: RollingUpdate
# Pod模板
template:
metadata:
labels:
app: mysql
component: database
spec:
# 亲和性
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mysql
topologyKey: kubernetes.io/hostname
# 容器
containers:
- name: mysql
image: mysql:8.0
imagePullPolicy: IfNotPresent
# 环境变量
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secrets
key: root-password
- name: MYSQL_DATABASE
value: "appdb"
- name: MYSQL_USER
value: "appuser"
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secrets
key: user-password
# 端口
ports:
- name: mysql
containerPort: 3306
protocol: TCP
# 资源限制
resources:
requests:
memory: "1Gi"
cpu: "500m"
ephemeral-storage: "10Gi"
limits:
memory: "2Gi"
cpu: "2000m"
ephemeral-storage: "20Gi"
# 健康检查
livenessProbe:
exec:
command:
- mysqladmin
- ping
- -h
- localhost
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- mysql
- -h
- localhost
- -u
- root
- -p$(MYSQL_ROOT_PASSWORD)
- -e
- SELECT 1
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
# 卷挂载
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql
- name: mysql-config
mountPath: /etc/mysql/conf.d
readOnly: true
- name: mysql-init
mountPath: /docker-entrypoint-initdb.d
readOnly: true
# 安全上下文
securityContext:
runAsUser: 999
runAsGroup: 999
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
# 初始化容器
initContainers:
- name: init-mysql
image: mysql:8.0
command:
- bash
- "-c"
- |
set -ex
# 生成server-id
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
# 复制配置
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d/
else
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
- name: mysql-config
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
# 卷
volumes:
- name: mysql-config
emptyDir: {}
- name: config-map
configMap:
name: mysql-config
- name: mysql-init
configMap:
name: mysql-init
items:
- key: init.sql
path: init.sql
# 卷声明模板
volumeClaimTemplates:
- metadata:
name: mysql-data
labels:
app: mysql
component: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 20Gi
---
# MySQL服务
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: database
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
targetPort: mysql
selector:
app: mysql
clusterIP: None
---
# MySQL Headless服务
apiVersion: v1
kind: Service
metadata:
name: mysql-read
namespace: database
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
---
# 配置映射
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-config
namespace: database
data:
master.cnf: |
[mysqld]
log-bin
binlog_format=ROW
gtid_mode=ON
enforce_gtid_consistency=ON
log_slave_updates=ON
slave.cnf: |
[mysqld]
super-read-only=1
---
# 初始化SQL
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-init
namespace: database
data:
init.sql: |
CREATE DATABASE IF NOT EXISTS appdb;
CREATE USER IF NOT EXISTS 'appuser'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON appdb.* TO 'appuser'@'%';
FLUSH PRIVILEGES;
---
# 密钥
apiVersion: v1
kind: Secret
metadata:
name: mysql-secrets
namespace: database
type: Opaque
stringData:
root-password: "RootPass123!"
user-password: "UserPass456!"
部署有状态应用：
# 创建命名空间
$ kubectl create namespace database
namespace/database created
# 部署MySQL StatefulSet
$ kubectl apply -f stateful-app-with-pvc.yaml
statefulset.apps/mysql created
service/mysql created
service/mysql-read created
configmap/mysql-config created
configmap/mysql-init created
secret/mysql-secrets created
# 查看StatefulSet
$ kubectl get statefulset -n database
NAME READY AGE
mysql 3/3 30s
# 查看Pod
$ kubectl get pods -n database
NAME READY STATUS RESTARTS AGE
mysql-0 1/1 Running 0 40s
mysql-1 1/1 Running 0 30s
mysql-2 1/1 Running 0 20s
# 查看PVC
$ kubectl get pvc -n database
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-data-mysql-0 Bound pvc-abc123-def4-5678-90ab-cdef01234567 20Gi RWO gp3 40s
mysql-data-mysql-1 Bound pvc-def456-7890-1234-5678-90abcdef01234 20Gi RWO gp3 30s
mysql-data-mysql-2 Bound pvc-789012-3456-7890-1234-567890abcdef 20Gi RWO gp3 20s
# 查看PV
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-abc123-def4-5678-90ab-cdef01234567 20Gi RWO Delete Bound database/mysql-data-mysql-0 gp3 40s
pvc-def456-7890-1234-5678-90abcdef01234 20Gi RWO Delete Bound database/mysql-data-mysql-1 gp3 30s
pvc-789012-3456-7890-1234-567890abcdef 20Gi RWO Delete Bound database/mysql-data-mysql-2 gp3 20s
# 测试MySQL连接
$ kubectl exec -n database -it mysql-0 -- mysql -u root -pRootPass123! -e "SHOW DATABASES;"
+--------------------+
| Database |
+--------------------+
| appdb |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
13.6 存储卷快照
13.6.1 VolumeSnapshotClass
# volume-snapshot-class.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-aws-vsc
annotations:
snapshot.storage.kubernetes.io/is-default-class: "true"
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
tagKey: backup
tagValue: daily
13.6.2 VolumeSnapshot
# volume-snapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: mysql-backup-$(date +%Y%m%d-%H%M%S)
namespace: database
labels:
app: mysql
backup-type: daily
spec:
volumeSnapshotClassName: csi-aws-vsc
source:
persistentVolumeClaimName: mysql-data-mysql-0
13.6.3 从快照恢复
# restore-from-snapshot.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-restored-pvc
namespace: database
spec:
storageClassName: gp3
dataSource:
name: mysql-backup-20240101-120000
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
14. 安全最佳实践
14.1 RBAC权限控制
14.1.1 ServiceAccount配置
# serviceaccount-rbac.yaml
---
# 服务账户
apiVersion: v1
kind: ServiceAccount
metadata:
name: web-app-sa
namespace: production
annotations:
iam.gke.io/gcp-service-account: web-app@my-project.iam.gserviceaccount.com
automountServiceAccountToken: false
---
# 集群角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: web-app-cluster-role
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
# Pod相关权限
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/portforward"]
verbs: ["create"]
# 服务相关权限
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# 配置相关权限
- apiGroups: [""]
resources: ["configmaps", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# 事件相关权限
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch"]
# 持久化存储相关
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# Deployment相关权限
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# 自动扩缩容相关
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# 网络策略相关
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies", "ingresses"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# 存储类相关
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
# 自定义资源相关
- apiGroups: ["custom.example.com"]
resources: ["customresources"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# 节点相关权限
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
# 命名空间相关
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list", "watch"]
---
# 角色绑定
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: web-app-role-binding
namespace: production
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: web-app-role
subjects:
- kind: ServiceAccount
name: web-app-sa
namespace: production
---
# 集群角色绑定
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: web-app-cluster-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: web-app-cluster-role
subjects:
- kind: ServiceAccount
name: web-app-sa
namespace: production
---
# 只读角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: view-only-role
rules:
- apiGroups: [""]
resources: ["pods", "services", "configmaps", "secrets", "persistentvolumeclaims", "namespaces"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list", "watch"]
---
# 开发人员角色
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer-role
namespace: development
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "services", "configmaps", "secrets", "persistentvolumeclaims"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# 管理员角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: admin-role
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
- nonResourceURLs: ["*"]
verbs: ["*"]
验证RBAC配置：
# 创建RBAC资源
$ kubectl apply -f serviceaccount-rbac.yaml
serviceaccount/web-app-sa created
clusterrole.rbac.authorization.k8s.io/web-app-cluster-role created
rolebinding.rbac.authorization.k8s.io/web-app-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/web-app-cluster-role-binding created
clusterrole.rbac.authorization.k8s.io/view-only-role created
role.rbac.authorization.k8s.io/developer-role created
clusterrole.rbac.authorization.k8s.io/admin-role created
# 查看ServiceAccount
$ kubectl get serviceaccount -n production
NAME SECRETS AGE
default 1 1d
web-app-sa 0 10s
# 查看RoleBinding
$ kubectl get rolebinding -n production
NAME ROLE AGE
web-app-role-binding Role/web-app-role 10s
# 查看ClusterRoleBinding
$ kubectl get clusterrolebinding | grep web-app
NAME ROLE AGE
web-app-cluster-role-binding ClusterRole/web-app-cluster-role 10s
# 测试权限
$ kubectl auth can-i get pods --as=system:serviceaccount:production:web-app-sa
yes
$ kubectl auth can-i create deployments --as=system:serviceaccount:production:web-app-sa
yes
$ kubectl auth can-i delete nodes --as=system:serviceaccount:production:web-app-sa
no
14.2 Pod安全策略（PSP）
# pod-security-policy.yaml
---
# 受限策略
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default'
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default'
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
spec:
privileged: false
# 必须设置非特权运行
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
# 允许的卷类型
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# 必须设置非root用户
rule: 'MustRunAsNonRoot'
seLinux:
# 必须遵循Pod的SELinux上下文
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# 禁止添加root组
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: true
---
# 特权策略
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
volumes:
- '*'
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
---
# 受限策略的集群角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- restricted
---
# 特权策略的集群角色
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:privileged
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- privileged
---
# 绑定到服务账户
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:restricted:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:restricted
subjects:
- kind: Group
name: system:serviceaccounts
apiGroup: rbac.authorization.k8s.io
---
# 绑定到特权服务账户
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:privileged:privileged-sa
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:privileged
subjects:
- kind: ServiceAccount
name: privileged-sa
namespace: kube-system
14.3 网络策略
14.3.1 默认拒绝策略
# network-policies.yaml
---
# 默认拒绝所有入站流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: production
spec:
podSelector: {} # 选择所有Pod
policyTypes:
- Ingress
---
# 默认拒绝所有出站流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: production
spec:
podSelector: {} # 选择所有Pod
policyTypes:
- Egress
---
# Web应用网络策略
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-app-network-policy
namespace: production
spec:
podSelector:
matchLabels:
app: web-app
component: backend
policyTypes:
- Ingress
- Egress
# 入站规则
ingress:
# 允许来自Ingress控制器的HTTP/HTTPS流量
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
- podSelector:
matchLabels:
app: ingress-nginx
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 8443
# 允许来自监控系统的流量
- from:
- namespaceSelector:
matchLabels:
name: monitoring
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 9090
# 允许来自同一命名空间的前端Pod
- from:
- podSelector:
matchLabels:
app: web-app
component: frontend
ports:
- protocol: TCP
port: 8080
# 出站规则
egress:
# 允许访问数据库
- to:
- podSelector:
matchLabels:
app: mysql
component: database
ports:
- protocol: TCP
port: 3306
# 允许访问Redis
- to:
- podSelector:
matchLabels:
app: redis
component: cache
ports:
- protocol: TCP
port: 6379
# 允许访问Elasticsearch
- to:
- podSelector:
matchLabels:
app: elasticsearch
component: logging
ports:
- protocol: TCP
port: 9200
- protocol: TCP
port: 9300
# 允许DNS查询
- to:
- namespaceSelector:
matchLabels:
name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# 允许访问外部HTTPS
- to:
- ipBlock:
cidr: 0.0.0.0/0
ports:
- protocol: TCP
port: 443
# 允许访问外部HTTP
- to:
- ipBlock:
cidr: 0.0.0.0/0
ports:
- protocol: TCP
port: 80
验证网络策略：
# 创建网络策略
$ kubectl apply -f network-policies.yaml
networkpolicy.networking.k8s.io/default-deny-ingress created
networkpolicy.networking.k8s.io/default-deny-egress created
networkpolicy.networking.k8s.io/web-app-network-policy created
# 查看网络策略
$ kubectl get networkpolicies -n production
NAME POD-SELECTOR AGE
default-deny-egress <none> 10s
default-deny-ingress <none> 10s
web-app-network-policy app=web-app,component=backend 10s
# 测试网络连通性
$ kubectl run -n production test-pod --image=alpine --restart=Never --rm -it -- sh
/ # wget -O- http://web-app-service:8080
Connecting to web-app-service:8080 (10.96.123.123:8080)
wget: can't connect to remote host (10.96.123.123): Connection refused
/ # wget -O- http://mysql:3306
Connecting to mysql:3306 (10.96.234.234:3306)
wget: can't connect to remote host (10.96.234.234): Connection refused
15. 性能优化指南
15.1 资源优化配置
15.1.1 资源请求和限制
# resources-optimization.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: optimized-app
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: optimized-app
template:
metadata:
labels:
app: optimized-app
spec:
containers:
- name: app
image: nginx:1.25-alpine
imagePullPolicy: IfNotPresent
# CPU优化配置
resources:
# 请求资源 - 调度依据
requests:
# CPU: 250毫核 (0.25个CPU核心)
# 设置合理值避免资源浪费和节点过载
cpu: "250m"
# 内存: 64MB
# 基于应用实际使用量设置，通常为平均使用量的80%
memory: "64Mi"
# 临时存储: 1GB
# 用于/var/log, /tmp等临时目录
ephemeral-storage: "1Gi"
# 大页内存 (可选)
# 对于需要大页内存的应用
hugepages-2Mi: "128Mi"
# 限制资源 - 运行限制
limits:
# CPU限制: 500毫核
# 防止应用占用过多CPU
cpu: "500m"
# 内存限制: 128MB
# 超过此限制会被OOM Killer终止
memory: "128Mi"
# 临时存储限制: 2GB
ephemeral-storage: "2Gi"
# 大页内存限制
hugepages-2Mi: "256Mi"
# QoS类别计算
# 1. Guaranteed: requests = limits (CPU和内存)
# 2. Burstable: requests < limits
# 3. BestEffort: 未设置requests和limits
# 本示例属于Burstable QoS类别
# 健康检查
livenessProbe:
httpGet:
path: /healthz
port: 80
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 80
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 1
# 启动探针
startupProbe:
httpGet:
path: /startup
port: 80
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 30
# 安全上下文
securityContext:
# 以非root用户运行
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
seccompProfile:
type: RuntimeDefault
15.2 自动扩缩容配置
15.2.1 HPA配置
# hpa-optimized.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: optimized-hpa
namespace: production
labels:
app: optimized-app
spec:
# 目标引用
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: optimized-app
# 最小副本数
minReplicas: 2
# 最大副本数
maxReplicas: 10
# 行为配置
behavior:
# 缩容行为
scaleDown:
# 稳定窗口: 5分钟
stabilizationWindowSeconds: 300
policies:
# 策略1: 每分钟最多缩容10%
- type: Percent
value: 10
periodSeconds: 60
# 策略2: 每分钟最多缩容1个Pod
- type: Pods
value: 1
periodSeconds: 60
# 选择策略: 取最小值
selectPolicy: Min
# 扩容行为
scaleUp:
# 稳定窗口: 0秒（立即扩容）
stabilizationWindowSeconds: 0
policies:
# 策略1: 15秒内最多扩容100%
- type: Percent
value: 100
periodSeconds: 15
# 策略2: 15秒内最多扩容4个Pod
- type: Pods
value: 4
periodSeconds: 15
# 选择策略: 取最大值
selectPolicy: Max
# 指标配置
metrics:
# CPU利用率指标
- type: Resource
resource:
name: cpu
target:
type: Utilization
# 目标CPU利用率: 70%
# 经验值: 70-80%为佳
averageUtilization: 70
# 内存利用率指标
- type: Resource
resource:
name: memory
target:
type: Utilization
# 目标内存利用率: 80%
averageUtilization: 80
# 自定义指标: QPS
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
# 目标QPS: 每个Pod 1000请求/秒
averageValue: 1000
# 自定义指标: 响应时间
- type: Object
object:
metric:
name: latency_p99
describedObject:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: optimized-app-ingress
target:
type: Value
# 目标P99延迟: 200ms
value: 200m
验证HPA：
# 创建HPA
$ kubectl apply -f hpa-optimized.yaml
horizontalpodautoscaler.autoscaling/optimized-hpa created
# 查看HPA状态
$ kubectl get hpa -n production
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
optimized-hpa Deployment/optimized-app <unknown>/70%, <unknown>/80% 2 10 3 10s
# 查看HPA详情
$ kubectl describe hpa optimized-hpa -n production
Name: optimized-hpa
Namespace: production
Labels: app=optimized-app
Annotations: <none>
CreationTimestamp: Mon, 01 Jan 2024 10:00:00 +0800
Reference: Deployment/optimized-app
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 70%
resource memory on pods (as a percentage of request): <unknown> / 80%
Min replicas: 2
Max replicas: 10
Behavior:
Scale Up:
Stabilization Window: 0 seconds
Select Policy: Max
Policies:
- Type: Pods Value: 4 Period: 15 seconds
- Type: Percent Value: 100 Period: 15 seconds
Scale Down:
Stabilization Window: 300 seconds
Select Policy: Min
Policies:
- Type: Pods Value: 1 Period: 60 seconds
- Type: Percent Value: 10 Period: 60 seconds
Deployment pods: 3 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
Events: <none>
15.3 节点优化
15.3.1 节点压力驱逐配置
# kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# CPU管理策略
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 10s
# 拓扑管理器策略
topologyManagerPolicy: best-effort
# 内存管理
memoryManagerPolicy: Static
# 预留系统资源
systemReserved:
cpu: "500m"
memory: "1Gi"
ephemeral-storage: "5Gi"
kubeReserved:
cpu: "250m"
memory: "512Mi"
ephemeral-storage: "1Gi"
# 驱逐配置
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
imagefs.inodesFree: "5%"
evictionSoft:
memory.available: "300Mi"
nodefs.available: "15%"
nodefs.inodesFree: "10%"
evictionSoftGracePeriod:
memory.available: 1m30s
nodefs.available: 1m30s
nodefs.inodesFree: 1m30s
evictionPressureTransitionPeriod: 5m0s
evictionMaxPodGracePeriod: 30
# 最大Pod数
maxPods: 110
# Pod PDB
podPidsLimit: -1
# 镜像垃圾回收
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
# 容器日志配置
containerLogMaxSize: 10Mi
containerLogMaxFiles: 5
# 旋转证书
rotateCertificates: true
# 流式连接空闲超时
streamingConnectionIdleTimeout: 4h0m0s
# 节点状态更新频率
nodeStatusUpdateFrequency: 10s
# 节点状态报告频率
nodeStatusReportFrequency: 5m0s
16. 常见错误与解决方案
16.1 启动错误
错误1: ImagePullBackOff
# 错误现象
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7c8d9f5c8-abc12 0/1 ImagePullBackOff 0 2m
# 查看Pod详情
$ kubectl describe pod nginx-7c8d9f5c8-abc12
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m10s default-scheduler Successfully assigned default/nginx-7c8d9f5c8-abc12 to node-1
Normal Pulling 2m9s kubelet Pulling image "nginx:non-existent"
Warning Failed 2m8s kubelet Failed to pull image "nginx:non-existent": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:non-existent": failed to resolve reference "docker.io/library/nginx:non-existent": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 2m8s kubelet Error: ErrImagePull
Normal BackOff 2m7s kubelet Back-off pulling image "nginx:non-existent"
Warning Failed 117s kubelet Error: ImagePullBackOff
# 解决方案
1. 检查镜像名称是否正确
$ kubectl set image deployment/nginx nginx=nginx:1.25
2. 检查镜像仓库权限
$ kubectl create secret docker-registry regcred \
--docker-server=registry.example.com \
--docker-username=username \
--docker-password=password \
--docker-email=email@example.com
$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}'
3. 检查网络连通性
$ kubectl run -it --rm --image=busybox debug-pod -- ping registry.example.com
错误2: CrashLoopBackOff
# 错误现象
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
crashing-app-abc12 0/1 CrashLoopBackOff 3 1m
# 查看日志
$ kubectl logs crashing-app-abc12
Error: Unable to connect to database
# 解决方案
1. 检查应用配置
$ kubectl describe pod crashing-app-abc12
$ kubectl get configmap app-config -o yaml
$ kubectl get secret app-secrets -o yaml
2. 检查依赖服务
$ kubectl get services
$ kubectl get endpoints
3. 调试容器
$ kubectl exec -it crashing-app-abc12 -- sh
$ kubectl debug crashing-app-abc12 -it --image=busybox
错误3: Pending状态
# 错误现象
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pending-pod 0/1 Pending 0 5m
# 查看详情
$ kubectl describe pod pending-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m default-scheduler 0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 Insufficient cpu.
# 解决方案
1. 检查节点资源
$ kubectl describe nodes
$ kubectl top nodes
2. 检查节点状态
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-1 NotReady <none> 5d v1.28.0
node-2 Ready <none> 5d v1.28.0
node-3 Ready <none> 5d v1.28.0
3. 检查污点
$ kubectl describe node node-1 | grep Taint
Taints: node.kubernetes.io/not-ready:NoSchedule
4. 增加资源或调整调度
$ kubectl scale deployment myapp --replicas=2
# 或调整资源请求
$ kubectl edit deployment myapp
16.2 网络错误
错误4: DNS解析失败
# 错误现象
$ kubectl exec -it test-pod -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'
# 解决方案
1. 检查CoreDNS Pod
$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-7c8d9f5c8-abc12 1/1 Running 0 5d
coredns-7c8d9f5c8-def34 1/1 Running 0 5d
2. 检查CoreDNS日志
$ kubectl logs -n kube-system coredns-7c8d9f5c8-abc12
3. 检查CoreDNS配置
$ kubectl get configmap -n kube-system coredns -o yaml
4. 测试DNS解析
$ kubectl run -it --rm --image=busybox dns-test -- nslookup kubernetes.default
错误5: 服务无法访问
# 错误现象
$ kubectl exec -it test-pod -- curl http://my-service:80
curl: (7) Failed to connect to my-service port 80: Connection refused
# 解决方案
1. 检查Service
$ kubectl get services
$ kubectl describe service my-service
$ kubectl get endpoints my-service
2. 检查Pod标签
$ kubectl get pods --show-labels
$ kubectl get service my-service -o yaml | grep selector
3. 检查网络策略
$ kubectl get networkpolicies
$ kubectl describe networkpolicy my-np
4. 直接访问Pod IP
$ kubectl get pods -o wide
$ kubectl exec -it test-pod -- curl http://10.244.1.2:80
16.3 存储错误
错误6: PVC Pending
# 错误现象
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-pvc Pending standard 5m
# 解决方案
1. 检查StorageClass
$ kubectl get storageclass
$ kubectl describe storageclass standard
2. 检查PV
$ kubectl get pv
$ kubectl describe pv
3. 检查Provisioner日志
$ kubectl get pods -n kube-system | grep provisioner
$ kubectl logs -n kube-system ebs-csi-controller-abc123
4. 检查节点标签
$ kubectl get nodes --show-labels
错误7: Mount错误
# 错误现象
$ kubectl describe pod app-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 2m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-abc123]: timed out waiting for the condition
# 解决方案
1. 检查存储后端
$ kubectl get pv
$ kubectl describe pv pvc-abc123
2. 检查节点状态
$ kubectl describe node node-1
3. 检查CSI驱动
$ kubectl get pods -n kube-system | grep csi
$ kubectl logs -n kube-system csi-node-abc123
17. 监控与运维
17.1 监控配置
17.1.1 Prometheus配置
# prometheus-config.yaml
---
# Prometheus ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitoring
---
# ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
# ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitoring
---
# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
namespace: monitoring
labels:
name: prometheus-server-conf
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_timeout: 10s
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
---
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: monitoring
labels:
app: prometheus
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: prom/prometheus:v2.45.0
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
- "--web.console.libraries=/etc/prometheus/console_libraries"
- "--web.console.templates=/etc/prometheus/consoles"
- "--web.enable-lifecycle"
- "--storage.tsdb.retention.time=30d"
ports:
- containerPort: 9090
# 17.1.2 完整的Prometheus监控方案
接续Prometheus Deployment配置：
```yaml
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "1000m"
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
livenessProbe:
httpGet:
path: /-/healthy
port: 9090
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /-/ready
port: 9090
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
persistentVolumeClaim:
claimName: prometheus-pvc
---
# PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-pvc
namespace: monitoring
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 50Gi
---
# Service
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: monitoring
labels:
app: prometheus
spec:
selector:
app: prometheus
ports:
- name: web
port: 80
targetPort: 9090
type: ClusterIP
---
# Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-ingress
namespace: monitoring
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
spec:
ingressClassName: nginx
tls:
- hosts:
- prometheus.example.com
secretName: prometheus-tls
rules:
- host: prometheus.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-service
port:
number: 80
---
# Node Exporter DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: monitoring
labels:
app: node-exporter
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
hostNetwork: true
hostPID: true
containers:
- name: node-exporter
image: prom/node-exporter:v1.6.0
args:
- "--path.procfs=/host/proc"
- "--path.sysfs=/host/sys"
- "--path.rootfs=/host/root"
- "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($|/)"
ports:
- containerPort: 9100
hostPort: 9100
resources:
requests:
memory: "30Mi"
cpu: "100m"
limits:
memory: "50Mi"
cpu: "200m"
volumeMounts:
- name: proc
mountPath: /host/proc
readOnly: true
- name: sys
mountPath: /host/sys
readOnly: true
- name: root
mountPath: /host/root
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
- name: sys
hostPath:
path: /sys
- name: root
hostPath:
path: /
tolerations:
- effect: NoSchedule
operator: Exists
---
# kube-state-metrics
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
replicas: 2
selector:
matchLabels:
app: kube-state-metrics
template:
metadata:
labels:
app: kube-state-metrics
spec:
serviceAccountName: kube-state-metrics
containers:
- name: kube-state-metrics
image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.9.2
ports:
- containerPort: 8080
- containerPort: 8081
resources:
requests:
memory: "150Mi"
cpu: "100m"
limits:
memory: "300Mi"
cpu: "200m"
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
---
# Service for kube-state-metrics
apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: monitoring
labels:
app: kube-state-metrics
spec:
ports:
- name: http-metrics
port: 8080
targetPort: 8080
- name: telemetry
port: 8081
targetPort: 8081
selector:
app: kube-state-metrics
clusterIP: None
---
# AlertManager配置
apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-config
namespace: monitoring
data:
alertmanager.yml: |
global:
smtp_smarthost: 'smtp.example.com:587'
smtp_from: 'alertmanager@example.com'
smtp_auth_username: 'alertmanager'
smtp_auth_password: 'password'
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'web.hook'
routes:
- match:
severity: critical
receiver: 'critical-alerts'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://webhook.example.com'
- name: 'critical-alerts'
email_configs:
- to: 'alerts@example.com'
send_resolved: true
---
# AlertManager Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: alertmanager
namespace: monitoring
spec:
replicas: 3
selector:
matchLabels:
app: alertmanager
template:
metadata:
labels:
app: alertmanager
spec:
containers:
- name: alertmanager
image: prom/alertmanager:v0.25.0
args:
- '--config.file=/etc/alertmanager/alertmanager.yml'
- '--storage.path=/alertmanager'
- '--cluster.advertise-address=0.0.0.0:9093'
ports:
- containerPort: 9093
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "400Mi"
cpu: "200m"
volumeMounts:
- name: config-volume
mountPath: /etc/alertmanager
- name: storage-volume
mountPath: /alertmanager
livenessProbe:
httpGet:
path: /-/healthy
port: 9093
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /-/ready
port: 9093
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: config-volume
configMap:
name: alertmanager-config
- name: storage-volume
emptyDir: {}
---
# AlertManager Service
apiVersion: v1
kind: Service
metadata:
name: alertmanager
namespace: monitoring
spec:
selector:
app: alertmanager
ports:
- name: web
port: 80
targetPort: 9093
type: ClusterIP
部署和验证监控系统：
# 创建监控命名空间
$ kubectl create namespace monitoring
namespace/monitoring created
# 部署监控组件
$ kubectl apply -f prometheus-config.yaml
serviceaccount/prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prometheus-server-conf created
deployment.apps/prometheus-deployment created
persistentvolumeclaim/prometheus-pvc created
service/prometheus-service created
ingress.networking.k8s.io/prometheus-ingress created
daemonset.apps/node-exporter created
deployment.apps/kube-state-metrics created
service/kube-state-metrics created
configmap/alertmanager-config created
deployment.apps/alertmanager created
service/alertmanager created
# 查看监控组件状态
$ kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/alertmanager-7c8d9f5c8-abc12 1/1 Running 0 30s
pod/alertmanager-7c8d9f5c8-def34 1/1 Running 0 30s
pod/alertmanager-7c8d9f5c8-ghi56 1/1 Running 0 30s
pod/kube-state-metrics-7c8d9f5c8-jkl78 1/1 Running 0 30s
pod/kube-state-metrics-7c8d9f5c8-mno90 1/1 Running 0 30s
pod/node-exporter-abc12 1/1 Running 0 30s
pod/node-exporter-def34 1/1 Running 0 30s
pod/node-exporter-ghi56 1/1 Running 0 30s
pod/prometheus-deployment-7c8d9f5c8-pqr1 1/1 Running 0 30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager ClusterIP 10.96.123.123 <none> 80/TCP 30s
service/kube-state-metrics ClusterIP None <none> 8080/TCP,8081/TCP 30s
service/prometheus-service ClusterIP 10.96.234.234 <none> 80/TCP 30s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE AGE
daemonset.apps/node-exporter 3 3 3 3 3 30s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/alertmanager 3/3 3 3 30s
deployment.apps/kube-state-metrics 2/2 2 2 30s
deployment.apps/prometheus-deployment 1/1 1 1 30s
# 访问Prometheus UI
$ kubectl port-forward -n monitoring svc/prometheus-service 9090:80
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
# 在浏览器中访问 http://localhost:9090
17.2 日志收集
17.2.1 ELK Stack配置
# elk-stack.yaml
---
# Elasticsearch StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: logging
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
initContainers:
- name: fix-permissions
image: busybox:1.35
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox:1.35
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox:1.35
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2
env:
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: cluster.name
value: "k8s-logs"
- name: discovery.seed_hosts
value: "elasticsearch-0.elasticsearch,elasticsearch-1.elasticsearch,elasticsearch-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
- name: xpack.security.enabled
value: "false"
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2000m"
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
livenessProbe:
tcpSocket:
port: 9200
initialDelaySeconds: 20
periodSeconds: 10
readinessProbe:
httpGet:
path: /_cluster/health?local=true
port: 9200
initialDelaySeconds: 20
periodSeconds: 5
volumes:
- name: data
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "gp3"
resources:
requests:
storage: 50Gi
---
# Elasticsearch Service
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
namespace: logging
labels:
app: elasticsearch
spec:
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
selector:
app: elasticsearch
clusterIP: None
---
# Kibana Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: logging
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:8.10.2
env:
- name: ELASTICSEARCH_URL
value: "http://elasticsearch:9200"
ports:
- containerPort: 5601
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /api/status
port: 5601
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /api/status
port: 5601
initialDelaySeconds: 5
periodSeconds: 5
---
# Kibana Service
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: logging
spec:
ports:
- port: 80
targetPort: 5601
selector:
app: kibana
type: ClusterIP
---
# Kibana Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kibana-ingress
namespace: logging
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
spec:
ingressClassName: nginx
tls:
- hosts:
- kibana.example.com
secretName: kibana-tls
rules:
- host: kibana.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kibana
port:
number: 80
---
# Fluentd DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: logging
labels:
app: fluentd
spec:
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch.logging.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENT_UID
value: "0"
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "500Mi"
cpu: "500m"
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluentd-config
mountPath: /fluentd/etc
- name: fluentd-buffers
mountPath: /var/log/fluentd-buffers
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluentd-config
configMap:
name: fluentd-config
- name: fluentd-buffers
hostPath:
path: /var/log/fluentd-buffers
---
# Fluentd ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: logging
data:
fluent.conf: |
<source>
@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
<parse>
@type json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
<match kubernetes.var.log.containers.**_kube-system_**.log>
@type null
</match>
<match **>
@type elasticsearch
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME']}"
logstash_format true
logstash_prefix fluentd
buffer_chunk_limit 2M
buffer_queue_limit 32
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 8
</match>
---
# Fluentd ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: logging
---
# Fluentd ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
# Fluentd ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluentd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluentd
subjects:
- kind: ServiceAccount
name: fluentd
namespace: logging
部署和验证日志系统：
# 创建日志命名空间
$ kubectl create namespace logging
namespace/logging created
# 部署ELK Stack
$ kubectl apply -f elk-stack.yaml
statefulset.apps/elasticsearch created
service/elasticsearch created
deployment.apps/kibana created
service/kibana created
ingress.networking.k8s.io/kibana-ingress created
daemonset.apps/fluentd created
configmap/fluentd-config created
serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created
# 查看ELK组件状态
$ kubectl get all -n logging
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-0 1/1 Running 0 1m
pod/elasticsearch-1 1/1 Running 0 50s
pod/elasticsearch-2 1/1 Running 0 40s
pod/fluentd-abc12 1/1 Running 0 1m
pod/fluentd-def34 1/1 Running 0 1m
pod/fluentd-ghi56 1/1 Running 0 1m
pod/kibana-7c8d9f5c8-jkl78 1/1 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 1m
service/kibana ClusterIP 10.96.123.123 <none> 80/TCP 1m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE AGE
daemonset.apps/fluentd 3 3 3 3 3 1m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana 1/1 1 1 1m
NAME READY AGE
statefulset.apps/elasticsearch 3/3 1m
# 测试Elasticsearch连接
$ kubectl run -n logging -it --rm --image=curlimages/curl test-curl -- curl http://elasticsearch:9200
{
"name" : "elasticsearch-0",
"cluster_name" : "k8s-logs",
"cluster_uuid" : "abc123-def4-5678-90ab-cdef01234567",
"version" : {
"number" : "8.10.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "abc123def456",
"build_date" : "2023-10-31T10:00:00.000Z",
"build_snapshot" : false,
"lucene_version" : "9.7.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
18. 进阶：多集群管理与GitOps
18.1 多集群管理（Cluster API）
# cluster-api-config.yaml
---
# Cluster API Provider
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: production-cluster
namespace: cluster-api
spec:
clusterNetwork:
pods:
cidrBlocks:
- 10.244.0.0/16
serviceDomain: cluster.local
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: production-control-plane
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
name: production-cluster
---
# Control Plane
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: production-control-plane
namespace: cluster-api
spec:
version: v1.28.0
replicas: 3
infrastructureTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
name: production-control-plane
kubeadmConfigSpec:
clusterConfiguration:
apiServer:
extraArgs:
cloud-provider: aws
controllerManager:
extraArgs:
cloud-provider: aws
initConfiguration:
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
kubeletExtraArgs:
cloud-provider: aws
joinConfiguration:
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
kubeletExtraArgs:
cloud-provider: aws
---
# Worker Node Pool
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: production-worker-pool
namespace: cluster-api
spec:
clusterName: production-cluster
replicas: 3
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: production-cluster
template:
spec:
clusterName: production-cluster
version: v1.28.0
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: production-worker-bootstrap
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
name: production-worker
18.2 GitOps工作流（ArgoCD）
# argocd-config.yaml
---
# ArgoCD命名空间
apiVersion: v1
kind: Namespace
metadata:
name: argocd
---
# ArgoCD安装
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/argoproj/argo-cd.git
targetRevision: stable
path: manifests/core
directory:
recurse: true
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
---
# 应用定义
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: production-apps
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/myorg/production-gitops.git
targetRevision: HEAD
path: apps
directory:
recurse: true
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
---
# 多集群应用
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: multi-cluster-apps
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
env: production
template:
metadata:
name: '{{name}}-apps'
spec:
project: production
source:
repoURL: https://github.com/myorg/production-gitops.git
targetRevision: HEAD
path: '{{path}}'
destination:
name: '{{name}}'
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
部署ArgoCD：
# 安装ArgoCD
$ kubectl create namespace argocd
namespace/argocd created
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-redis created
serviceaccount/argocd-repo-server created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-redis created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-notifications-secret created
secret/argocd-secret created
service/argocd-applicationset-controller created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-application-controller created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
networkpolicy.networking.k8s.io/argocd-server-network-policy created
# 获取ArgoCD管理员密码
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
admin-password-123
# 端口转发访问UI
$ kubectl port-forward svc/argocd-server -n argocd 8080:443
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
# 登录 http://localhost:8080
# 用户名: admin
# 密码: 上面获取的密码
19. Kubernetes发展趋势
19.1 Kubernetes 1.29+ 路线图
预计新特性：
-
Kubernetes 1.29 (计划中)
- 增强的Sidecar容器支持
- 改进的节点内存管理
- 更智能的调度器
- 增强的安全特性
-
Kubernetes 1.30+ 展望
- 更轻量级的发行版
- 增强的边缘计算支持
- AI/ML工作负载优化
- 服务网格原生集成
19.2 生态发展趋势
-
eBPF的深入集成
- Cilium作为默认CNI
- 基于eBPF的服务网格
- 增强的网络可观测性
-
WASM工作负载支持
- WebAssembly运行时集成
- 更安全的沙箱环境
- 快速启动的微服务
-
Serverless容器
- Knative深度集成
- 事件驱动的自动扩缩容
- 零冷启动优化
学习资源推荐
官方资源：
- https://kubernetes.io/docs/
- https://kubernetes.io/docs/reference/
- https://github.com/kubernetes/kubernetes
在线课程：
- Kubernetes官方教程
- Linux Foundation培训
- Coursera/edX相关课程
社区资源：
- Kubernetes Slack频道
- CNCF社区
- 本地Meetup活动
实践平台：
- https://www.katacoda.com/courses/kubernetes
- https://killercoda.com/kubernetes
- 各大云厂商的免费试用
更多推荐
8
0- 0
已为社区贡献8条内容
所有评论(0)