443 k8s配置开启nginx_[k8s]nginx-ingress配置4/7层测试

基本原理default-backend提供了2个功能:1. 404报错页面2. healthz页面# Any image is permissable as long as:# 1. It serves a 404 page at /# 2. It serves 200 on a /healthz endpoint创建svc,外面访问80 映射到容器的8080.deploy+svckubectl

巴黎巨星岬太郎

1391人浏览 · 2020-12-24 03:26:45

巴黎巨星岬太郎 · 2020-12-24 03:26:45 发布

基本原理

default-backend提供了2个功能:

1. 404报错页面

2. healthz页面

# Any image is permissable as long as:

# 1. It serves a 404 page at /

# 2. It serves 200 on a /healthz endpoint

创建svc,外面访问80 映射到容器的8080.

deploy+svc

kubectl create -f default-backend.yaml

nginx-ingress搭建

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \

| kubectl apply -f -

curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml \

| kubectl apply -f -

默认使用的镜像

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

gcr.io/google_containers/defaultbackend:1.4

docker pull lanny/gcr.io_google_containers_defaultbackend_1.4:v1.4

docker tag lanny/gcr.io_google_containers_defaultbackend_1.4:v1.4 gcr.io/google_containers/defaultbackend:1.4

贴上ingress的yaml

主要修改点:

通过18080访问状态页面(ingress-controller的nginx.conf决定)

http://192.168.x,x:18080/nginx_status

ingress-controller需要开启 hostNetwork: true

便于暴漏ingress的80端口和其他ingress-controller的nginx.conf暴漏的端口

namespace.yaml

apiVersion: v1

kind: Namespace

metadata:

default-backend.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

labels:

app: default-http-backend

namespace: ingress-nginx

spec:

replicas: 1

template:

metadata:

labels:

app: default-http-backend

spec:

terminationGracePeriodSeconds: 60

containers:

- name: default-http-backend

# Any image is permissable as long as:

# 1. It serves a 404 page at /

# 2. It serves 200 on a /healthz endpoint

image: gcr.io/google_containers/defaultbackend:1.4

livenessProbe:

httpGet:

path: /healthz

port: 8080

scheme: HTTP

initialDelaySeconds: 30

timeoutSeconds: 5

ports:

- containerPort: 8080

resources:

limits:

cpu: 10m

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

---

apiVersion: v1

kind: Service

metadata:

namespace: ingress-nginx

labels:

app: default-http-backend

spec:

ports:

- port: 80

targetPort: 8080

selector:

app: default-http-backend

without-rbac.yaml

这个yaml官网已经更新了, 多了个

sysctl -w net.core.somaxconn=32768; sysctl -w net.ipv4.ip_local_port_range="1024 65535"

而我这里还没更新.

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

namespace: ingress-nginx

spec:

replicas: 1

selector:

matchLabels:

app: ingress-nginx

template:

metadata:

labels:

app: ingress-nginx

annotations:

prometheus.io/port: '10254'

prometheus.io/scrape: 'true'

spec:

hostNetwork: true

containers:

- name: nginx-ingress-controller

image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

args:

- /nginx-ingress-controller

- --default-backend-service=$(POD_NAMESPACE)/default-http-backend

- --configmap=$(POD_NAMESPACE)/nginx-configuration

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

- --udp-services-configmap=$(POD_NAMESPACE)/udp-services

- --annotations-prefix=nginx.ingress.kubernetes.io

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

ports:

- name: http

containerPort: 80

- name: https

containerPort: 443

livenessProbe:

failureThreshold: 3

httpGet:

path: /healthz

port: 10254

scheme: HTTP

initialDelaySeconds: 10

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 1

readinessProbe:

failureThreshold: 3

httpGet:

path: /healthz

port: 10254

scheme: HTTP

periodSeconds: 10

successThreshold: 1

timeoutSeconds: 1

ingress-controller的本质是:

/nginx-ingress-controller 加启动参数

args:

- /nginx-ingress-controller

- --default-backend-service=$(POD_NAMESPACE)/default-http-backend

- --configmap=$(POD_NAMESPACE)/nginx-configuration

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

- --udp-services-configmap=$(POD_NAMESPACE)/udp-services

- --annotations-prefix=nginx.ingress.kubernetes.io

tcp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

namespace: ingress-nginx

udp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

namespace: ingress-nginx

启动nginx测试ingress的http 7层负载

kubectl run --image=nginx nginx --replicas=2

kubectl expose deployment nginx --port=80 ## 这里是svc端口,默认和容器的端口一致

nginx-ingress.conf

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

namespace: default

spec:

rules:

- host: mynginx.maotai.com

http:

paths:

- path: /

backend:

serviceName: nginx

servicePort: 80

注意: ingress虽然调用的是svc,貌似转发是client--nginx--svc--pod; 实际上ingress监控svc 自动将svc下的podip填充到nginx.conf.转发是client--nginx--pod

测试4层负载

udp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

namespace: ingress-nginx

data:

53: "kube-system/kube-dns:53"

修改后,apply即可,nginx-ingress可以热更新

kubectl apply -f udp-services-configmap.yaml

$ host -t A nginx.default.svc.cluster.local 192.168.14.132

Using domain server:

Name: 192.168.x.x

Address: 192.168.x.x#53

Aliases:

nginx.default.svc.cluster.local has address 10.254.160.155

另一个tcp的示例

kind: ConfigMap

apiVersion: v1

metadata:

namespace: ingress-nginx

data:

2200: "default/gitlab:22"

3306: "kube-public/mysql:3306"

2202: "kube-public/centos:22"

2203: "kube-public/mongodb:27017"

以下是nginx-ingress镜像的dockerfile.进这个容器可以看到

Dockerfile

FROM quay.io/kubernetes-ingress-controller/nginx-amd64:0.30

RUN clean-install \

diffutils \

dumb-init

# Create symlinks to redirect nginx logs to stdout and stderr docker log collector

# This only works if nginx is started with CMD or ENTRYPOINT

RUN mkdir -p /var/log/nginx \

&& ln -sf /dev/stdout /var/log/nginx/access.log \

&& ln -sf /dev/stderr /var/log/nginx/error.log

COPY . /

ENTRYPOINT ["/usr/bin/dumb-init"]

CMD ["/nginx-ingress-controller"]

默认启动后nginx-ingres的nginx.conf

root@n1:/etc/nginx# cat nginx.conf

daemon off;

worker_processes 4;

pid /run/nginx.pid;

worker_rlimit_nofile 15360;

worker_shutdown_timeout 10s ;

events {

multi_accept on;

worker_connections 16384;

use epoll;

}

http {

real_ip_header X-Forwarded-For;

real_ip_recursive on;

set_real_ip_from 0.0.0.0/0;

geoip_country /etc/nginx/GeoIP.dat;

geoip_city /etc/nginx/GeoLiteCity.dat;

geoip_proxy_recursive on;

vhost_traffic_status_zone shared:vhost_traffic_status:10m;

vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;

sendfile on;

aio threads;

aio_write on;

tcp_nopush on;

tcp_nodelay on;

log_subrequest on;

reset_timedout_connection on;

keepalive_timeout 75s;

keepalive_requests 100;

client_header_buffer_size 1k;

client_header_timeout 60s;

large_client_header_buffers 4 8k;

client_body_buffer_size 8k;

client_body_timeout 60s;

http2_max_field_size 4k;

http2_max_header_size 16k;

types_hash_max_size 2048;

server_names_hash_max_size 1024;

server_names_hash_bucket_size 32;

map_hash_bucket_size 64;

proxy_headers_hash_max_size 512;

proxy_headers_hash_bucket_size 64;

variables_hash_bucket_size 128;

variables_hash_max_size 2048;

underscores_in_headers off;

ignore_invalid_headers on;

include /etc/nginx/mime.types;

default_type text/html;

brotli on;

brotli_comp_level 4;

brotli_types application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;

gzip on;

gzip_comp_level 5;

gzip_http_version 1.1;

gzip_min_length 256;

gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;

gzip_proxied any;

gzip_vary on;

# Custom headers for response

server_tokens on;

# disable warnings

uninitialized_variable_warn off;

# Additional available variables:

# $namespace

# $ingress_name

# $service_name

log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

map $request_uri $loggable {

default 1;

}

access_log /var/log/nginx/access.log upstreaminfo if=$loggable;

error_log /var/log/nginx/error.log notice;

resolver 192.168.14.2 valid=30s;

# Retain the default nginx handling of requests without a "Connection" header

map $http_upgrade $connection_upgrade {

default upgrade;

close;

}

map $http_x_forwarded_for $the_real_ip {

default $remote_addr;

}

# trust http_x_forwarded_proto headers correctly indicate ssl offloading

map $http_x_forwarded_proto $pass_access_scheme {

default $http_x_forwarded_proto;

$scheme;

}

map $http_x_forwarded_port $pass_server_port {

default $http_x_forwarded_port;

$server_port;

}

map $http_x_forwarded_host $best_http_host {

default $http_x_forwarded_host;

$this_host;

}

map $pass_server_port $pass_port {

443 443;

default $pass_server_port;

}

# Obtain best http host

map $http_host $this_host {

default $http_host;

$host;

}

server_name_in_redirect off;

port_in_redirect off;

ssl_protocols TLSv1.2;

# turn on session caching to drastically improve performance

ssl_session_cache builtin:1000 shared:SSL:10m;

ssl_session_timeout 10m;

# allow configuring ssl session tickets

ssl_session_tickets on;

# slightly reduce the time-to-first-byte

ssl_buffer_size 4k;

# allow configuring custom ssl ciphers

ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';

ssl_prefer_server_ciphers on;

ssl_ecdh_curve auto;

proxy_ssl_session_reuse on;

upstream upstream-default-backend {

# Load balance algorithm; empty for round robin, which is the default

least_conn;

keepalive 32;

server 10.2.98.3:8080 max_fails=0 fail_timeout=0;

}

## start server _

server {

server_name _ ;

listen 80 default_server reuseport backlog=511;

listen [::]:80 default_server reuseport backlog=511;

set $proxy_upstream_name "-";

listen 443 default_server reuseport backlog=511 ssl http2;

listen [::]:443 default_server reuseport backlog=511 ssl http2;

# PEM sha: 479f4653ff7d901e313895dbaafbbe64b0805346

ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;

ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;

more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";

location / {

set $proxy_upstream_name "upstream-default-backend";

set $namespace "";

set $ingress_name "";

set $service_name "";

port_in_redirect off;

client_max_body_size "1m";

proxy_set_header Host $best_http_host;

# Pass the extracted client certificate to the backend

proxy_set_header ssl-client-cert "";

proxy_set_header ssl-client-verify "";

proxy_set_header ssl-client-dn "";

# Allow websocket connections

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection $connection_upgrade;

proxy_set_header X-Real-IP $the_real_ip;

proxy_set_header X-Forwarded-For $the_real_ip;

proxy_set_header X-Forwarded-Host $best_http_host;

proxy_set_header X-Forwarded-Port $pass_port;

proxy_set_header X-Forwarded-Proto $pass_access_scheme;

proxy_set_header X-Original-URI $request_uri;

proxy_set_header X-Scheme $pass_access_scheme;

# Pass the original X-Forwarded-For

proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

# mitigate HTTPoxy Vulnerability

# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

proxy_set_header Proxy "";

# Custom headers to proxied server

proxy_connect_timeout 5s;

proxy_send_timeout 60s;

proxy_read_timeout 60s;

proxy_redirect off;

proxy_buffering off;

proxy_buffer_size "4k";

proxy_buffers 4 "4k";

proxy_request_buffering "on";

proxy_http_version 1.1;

proxy_cookie_domain off;

proxy_cookie_path off;

# In case of errors try the next upstream server before returning an error

proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;

proxy_pass http://upstream-default-backend;

}

# health checks in cloud providers require the use of port 80

location /healthz {

access_log off;

return 200;

}

# this is required to avoid error if nginx is being monitored

# with an external software (like sysdig)

location /nginx_status {

allow 127.0.0.1;

allow ::1;

deny all;

access_log off;

stub_status on;

}

## end server _

# default server, used for NGINX healthcheck and access to nginx stats

server {

# Use the port 18080 (random value just to avoid known ports) as default port for nginx.

# Changing this value requires a change in:

# https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go

listen 18080 default_server reuseport backlog=511;

listen [::]:18080 default_server reuseport backlog=511;

set $proxy_upstream_name "-";

location /healthz {

access_log off;

return 200;

}

location /nginx_status {

set $proxy_upstream_name "internal";

vhost_traffic_status_display;

vhost_traffic_status_display_format html;

}

location / {

set $proxy_upstream_name "upstream-default-backend";

proxy_pass http://upstream-default-backend;

}

stream {

log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

access_log /var/log/nginx/access.log log_stream;

error_log /var/log/nginx/error.log;

# TCP services

# UDP services

}

多路径ingress

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

annotations:

ingress.kubernetes.io/rewrite-target: "/"

kubernetes.io/ingress.class: "nginx"

labels:

app: stultified-snail-monocular

chart: "monocular-0.5.0"

release: "stultified-snail"

heritage: "Tiller"

spec:

rules:

- http:

paths:

- backend:

serviceName: stultified-snail-monocular-ui

servicePort: 80

path: /

- backend:

serviceName: stultified-snail-monocular-api

servicePort: 80

path: /api/

host:

DNS Policy导致hostnetwork网络问题

参考: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

遇到个坑: nginx-ingress是hostnetwork模式, 而这种模式的pod的/etc/resolve.conf的dns是继承宿主机的(相当于dokcer run --net=host,共享宿主机网络协议栈,因此也继承了网卡dns),导致访问集群里的 api svc无法解析

By default, DNS policy for a pod is ‘ClusterFirst’. So pods running with hostNetwork cannot resolve DNS names. To have DNS options set along with hostNetwork, you should specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. Update the busybox.yaml as following:

nginx-ingress生产中使用方法(因为我没加证书,所以只能手动引导ingress连http的api)

args:

- /nginx-ingress-controller

- --apiserver-host=http://kube-api-http.kube-public

- --default-backend-service=$(POD_NAMESPACE)/default-http-backend

- --tcp-services-configmap=$(POD_NAMESPACE)/nginx-tcp-ingress-configmap

apiVersion: v1

kind: Pod

metadata:

namespace: default

spec:

containers:

- image: busybox

command:

- sleep

- "3600"

imagePullPolicy: IfNotPresent

restartPolicy: Always

hostNetwork: true

dnsPolicy: ClusterFirstWithHostNet

使用mysql测试nginx的4层负载

mysql的deploy和4层端口ingress创建

[root@m1 yaml]# cat mysql/mysql-deploy.yaml

apiVersion: v1

kind: ReplicationController

metadata:

spec:

replicas: 1

selector:

app: mysql

template:

metadata:

labels:

app: mysql

spec:

containers:

- name: mysql

image: mysql

ports:

- containerPort: 3306

env:

- name: MYSQL_ROOT_PASSWORD

value: "123456"

[root@m1 yaml]# cat ingress/tcp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

namespace: ingress-nginx

data:

3306: "default/mysql:3306"

注:

1.这里发现apply还有,ingress的nginx.conf配置没发生变化,因为我重启过了启动,重新建了下ingress正常了.

2.ingress创建的端口只在ingress-controller的节点上监听.

nginx配置参考

[root@m1 yaml]# cat ingress/tcp-services-configmap.yaml

kind: ConfigMap

apiVersion: v1

metadata:

namespace: ingress-nginx

data:

3306: "default/mysql:3306"

错误日志记录

日志json格式

stub_status & 开启认证

404错误页配置,并重定向

某些后缀文件拒绝访问(default.conf)

配置include(简化)

log-format-upstream: '{ "time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr",

"x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$request_id", "remote_user":

"$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":

$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri",

"request_query": "$args", "request_length": $request_length, "duration": $request_time,

"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent":

"$http_user_agent" }'

字段名

字段解释

$remote_addr, $http_x_forwarded_for

记录客户端IP地址

$remote_user

记录客户端用户名称

$request

记录请求的URL和HTTP协议

$status

记录请求状态

$body_bytes_sent

发送给客户端的字节数，不包括响应头的大小；该变量与Apache模块mod_log_config里的“%B”参数兼容。

$bytes_sent

发送给客户端的总字节数。

$connection

连接的序列号。

$connection_requests

当前通过一个连接获得的请求数量。

$msec

日志写入时间。单位为秒，精度是毫秒。

$pipe

如果请求是通过HTTP流水线(pipelined)发送，pipe值为“p”，否则为“.”。

$http_referer

记录从哪个页面链接访问过来的

$http_user_agent

记录客户端浏览器相关信息

$request_length

请求的长度(包括请求行，请求头和请求正文)。

$request_time

请求处理时间，单位为秒，精度毫秒；从读入客户端的第一个字节开始，直到把最后一个字符发送给客户端后进行日志写入为止。

$time_iso8601

ISO8601标准格式下的本地时间。

$time_local

通用日志格式下的本地时间。

nginx快速测试

mkdir -p /data/nginx-html

echo "maotai" > /data/nginx-html/index.html

docker run -d \

--net=host \

--restart=always \

-v /etc/nginx/nginx.conf:/etc/nginx/nginx.conf:ro \

-v /etc/localtime:/etc/localtime:ro \

-v /data/nginx-html:/usr/share/nginx/html \

--name nginx \

nginx

nginx4层负载和7层负载示例

单个端口映射负载:一对多

cat > nginx.conf <

error_log stderr notice;

worker_processes auto;

events {

multi_accept on;

use epoll;

worker_connections 1024;

}

stream {

upstream kube_apiserver {

least_conn;

server 192.168.8.161:6443;

server 192.168.8.162:6443;

server 192.168.8.163:6443;

}

server {

listen 127.0.0.1:6443;

proxy_pass kube_apiserver;

proxy_timeout 10m;

proxy_connect_timeout 1s;

}

EOF

单个端口映射:一对一

cat > nginx.conf <

error_log stderr notice;

worker_processes auto;

events {

multi_accept on;

use epoll;

worker_connections 1024;

}

stream {

log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

access_log /var/log/nginx/access.log log_stream;

error_log /var/log/nginx/error.log;

# TCP services

# UDP services

upstream udp-53-kube-system-kube-dns-53 {

server 10.2.54.9:53;

}

server {

listen 53 udp;

listen [::]:53 udp;

proxy_responses 1;

proxy_timeout 600s;

proxy_pass udp-53-kube-system-kube-dns-53;

}

EOF

一个tcp&一个udp端口映射

stream {

log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

access_log /var/log/nginx/access.log log_stream;

error_log /var/log/nginx/error.log;

# TCP services

upstream tcp-3306-default-mysql-3306 {

server 10.2.30.2:3306;

}

server {

listen 3306;

listen [::]:3306;

proxy_timeout 600s;

proxy_pass tcp-3306-default-mysql-3306;

}

# UDP services

upstream udp-53-kube-system-kube-dns-53 {

server 10.2.54.9:53;

}

server {

listen 53 udp;

listen [::]:53 udp;

proxy_responses 1;

proxy_timeout 600s;

proxy_pass udp-53-kube-system-kube-dns-53;

}

nginx的proxy-pass7层负载配置

user nginx nginx;

worker_processes auto;

worker_rlimit_nofile 65535;

error_log /usr/local/nginxlogs/error.log;

# pid logs/nginx.pid;

events {

use epoll;

worker_connections 51200;

}

http {

include mime.types;

default_type application/octet-stream;

log_format main '$remote_addr $remote_user [$time_local] "$request" $http_host '

'$status $upstream_status $body_bytes_sent "$http_referer" '

'"$http_user_agent" $ssl_protocol $ssl_cipher $upstream_addr '

'$request_time $upstream_response_time';

server_info off;

server_tag off;

server_name_in_redirect off;

access_log /usr/local/tengine-2.1.2/logs/access.log main;

client_max_body_size 80m;

client_header_buffer_size 16k;

large_client_header_buffers 4 16k;

sendfile on;

tcp_nopush on;

keepalive_timeout 65;

server_tokens on;

gzip on;

gzip_min_length 1k;

gzip_buffers 4 16k;

gzip_proxied any;

gzip_http_version 1.1;

gzip_comp_level 3;

gzip_types text/plain application/x-javascript text/css application/xml;

gzip_vary on;

upstream kube-api-vip {

least_conn;

server 192.168.x.20:8080;

server 192.168.x.21:8080;

server 192.168.x.22:8080;

}

server {

listen 80;

server_name kube-api-vip.maotai.net;

proxy_connect_timeout 1s;

# proxy_read_timeout 600;

# proxy_send_timeout 600;

proxy_buffer_size 128k;

proxy_buffers 4 256k;

proxy_busy_buffers_size 256k;

location / {

proxy_next_upstream error timeout invalid_header http_500 http_503 http_404 http_502 http_504;

proxy_pass http://kube-api-vip;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

混合:端口映射和7层映射放在一起(这是个ingress的例子)

root@n2:/# cat /etc/nginx/nginx.conf

daemon off;

worker_processes 4;

pid /run/nginx.pid;

worker_rlimit_nofile 15360;

worker_shutdown_timeout 10s ;

events {

multi_accept on;

worker_connections 16384;

use epoll;

}

http {

real_ip_header X-Forwarded-For;

real_ip_recursive on;

set_real_ip_from 0.0.0.0/0;

geoip_country /etc/nginx/GeoIP.dat;

geoip_city /etc/nginx/GeoLiteCity.dat;

geoip_proxy_recursive on;

vhost_traffic_status_zone shared:vhost_traffic_status:10m;

vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;

sendfile on;

aio threads;

aio_write on;

tcp_nopush on;

tcp_nodelay on;

log_subrequest on;

reset_timedout_connection on;

keepalive_timeout 75s;

keepalive_requests 100;

client_header_buffer_size 1k;

client_header_timeout 60s;

large_client_header_buffers 4 8k;

client_body_buffer_size 8k;

client_body_timeout 60s;

http2_max_field_size 4k;

http2_max_header_size 16k;

types_hash_max_size 2048;

server_names_hash_max_size 1024;

server_names_hash_bucket_size 64;

map_hash_bucket_size 64;

proxy_headers_hash_max_size 512;

proxy_headers_hash_bucket_size 64;

variables_hash_bucket_size 128;

variables_hash_max_size 2048;

underscores_in_headers off;

ignore_invalid_headers on;

include /etc/nginx/mime.types;

default_type text/html;

brotli on;

brotli_comp_level 4;

gzip on;

gzip_comp_level 5;

gzip_http_version 1.1;

gzip_min_length 256;

gzip_proxied any;

gzip_vary on;

# Custom headers for response

server_tokens on;

# disable warnings

uninitialized_variable_warn off;

# Additional available variables:

# $namespace

# $ingress_name

# $service_name

map $request_uri $loggable {

default 1;

}

access_log /var/log/nginx/access.log upstreaminfo if=$loggable;

error_log /var/log/nginx/error.log notice;

resolver 192.168.14.2 valid=30s;

# Retain the default nginx handling of requests without a "Connection" header

map $http_upgrade $connection_upgrade {

default upgrade;

'' close;

}

map $http_x_forwarded_for $the_real_ip {

default $remote_addr;

}

# trust http_x_forwarded_proto headers correctly indicate ssl offloading

map $http_x_forwarded_proto $pass_access_scheme {

default $http_x_forwarded_proto;

'' $scheme;

}

map $http_x_forwarded_port $pass_server_port {

default $http_x_forwarded_port;

'' $server_port;

}

map $http_x_forwarded_host $best_http_host {

default $http_x_forwarded_host;

'' $this_host;

}

map $pass_server_port $pass_port {

443 443;

default $pass_server_port;

}

# Obtain best http host

map $http_host $this_host {

default $http_host;

'' $host;

}

server_name_in_redirect off;

port_in_redirect off;

ssl_protocols TLSv1.2;

# turn on session caching to drastically improve performance

ssl_session_cache builtin:1000 shared:SSL:10m;

ssl_session_timeout 10m;

# allow configuring ssl session tickets

ssl_session_tickets on;

# slightly reduce the time-to-first-byte

ssl_buffer_size 4k;

# allow configuring custom ssl ciphers

ssl_prefer_server_ciphers on;

ssl_ecdh_curve auto;

proxy_ssl_session_reuse on;

upstream kube-public-spring-web {

# Load balance algorithm; empty for round robin, which is the default

least_conn;

keepalive 32;

server 10.2.54.4:8080 max_fails=0 fail_timeout=0;

}

upstream upstream-default-backend {

# Load balance algorithm; empty for round robin, which is the default

least_conn;

keepalive 32;

server 10.2.30.3:8080 max_fails=0 fail_timeout=0;

}

## start server _

server {

server_name _ ;

listen 80 default_server reuseport backlog=511;

listen [::]:80 default_server reuseport backlog=511;

set $proxy_upstream_name "-";

listen 443 default_server reuseport backlog=511 ssl http2;

listen [::]:443 default_server reuseport backlog=511 ssl http2;

# PEM sha: 20103470e60aa51135afee9244c7c831559f04b8

ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;

ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;

more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";

location / {

set $proxy_upstream_name "upstream-default-backend";

set $namespace "";

set $ingress_name "";

set $service_name "";

port_in_redirect off;

client_max_body_size "1m";

proxy_set_header Host $best_http_host;

# Pass the extracted client certificate to the backend

proxy_set_header ssl-client-cert "";

proxy_set_header ssl-client-verify "";

proxy_set_header ssl-client-dn "";

# Allow websocket connections

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection $connection_upgrade;

proxy_set_header X-Real-IP $the_real_ip;

proxy_set_header X-Forwarded-For $the_real_ip;

proxy_set_header X-Forwarded-Host $best_http_host;

proxy_set_header X-Forwarded-Port $pass_port;

proxy_set_header X-Forwarded-Proto $pass_access_scheme;

proxy_set_header X-Original-URI $request_uri;

proxy_set_header X-Scheme $pass_access_scheme;

# Pass the original X-Forwarded-For

proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

# mitigate HTTPoxy Vulnerability

# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

proxy_set_header Proxy "";

# Custom headers to proxied server

proxy_connect_timeout 5s;

proxy_send_timeout 60s;

proxy_read_timeout 60s;

proxy_redirect off;

proxy_buffering off;

proxy_buffer_size "4k";

proxy_buffers 4 "4k";

proxy_request_buffering "on";

proxy_http_version 1.1;

proxy_cookie_domain off;

proxy_cookie_path off;

# In case of errors try the next upstream server before returning an error

proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;

proxy_pass http://upstream-default-backend;

}

# health checks in cloud providers require the use of port 80

location /healthz {

access_log off;

return 200;

}

# this is required to avoid error if nginx is being monitored

# with an external software (like sysdig)

location /nginx_status {

allow 127.0.0.1;

allow ::1;

deny all;

access_log off;

stub_status on;

}

## end server _

## start server spring.maotai.com

server {

server_name spring.maotai.com ;

listen 80;

listen [::]:80;

set $proxy_upstream_name "-";

location / {

set $proxy_upstream_name "kube-public-spring-web";

set $namespace "kube-public";

set $ingress_name "spring";

set $service_name "spring";

port_in_redirect off;

client_max_body_size "1m";

proxy_set_header Host $best_http_host;

# Pass the extracted client certificate to the backend

proxy_set_header ssl-client-cert "";

proxy_set_header ssl-client-verify "";

proxy_set_header ssl-client-dn "";

# Allow websocket connections

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection $connection_upgrade;

proxy_set_header X-Real-IP $the_real_ip;

proxy_set_header X-Forwarded-For $the_real_ip;

proxy_set_header X-Forwarded-Host $best_http_host;

proxy_set_header X-Forwarded-Port $pass_port;

proxy_set_header X-Forwarded-Proto $pass_access_scheme;

proxy_set_header X-Original-URI $request_uri;

proxy_set_header X-Scheme $pass_access_scheme;

# Pass the original X-Forwarded-For

proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

# mitigate HTTPoxy Vulnerability

# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/

proxy_set_header Proxy "";

# Custom headers to proxied server

proxy_connect_timeout 5s;

proxy_send_timeout 60s;

proxy_read_timeout 60s;

proxy_redirect off;

proxy_buffering off;

proxy_buffer_size "4k";

proxy_buffers 4 "4k";

proxy_request_buffering "on";

proxy_http_version 1.1;

proxy_cookie_domain off;

proxy_cookie_path off;

# In case of errors try the next upstream server before returning an error

proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;

proxy_pass http://kube-public-spring-web;

}

## end server spring.maotai.com

# default server, used for NGINX healthcheck and access to nginx stats

server {

# Use the port 18080 (random value just to avoid known ports) as default port for nginx.

# Changing this value requires a change in:

# https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go

listen 18080 default_server reuseport backlog=511;

listen [::]:18080 default_server reuseport backlog=511;

set $proxy_upstream_name "-";

location /healthz {

access_log off;

return 200;

}

location /nginx_status {

set $proxy_upstream_name "internal";

vhost_traffic_status_display;

vhost_traffic_status_display_format html;

}

location / {

set $proxy_upstream_name "upstream-default-backend";

proxy_pass http://upstream-default-backend;

}

stream {

log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

access_log /var/log/nginx/access.log log_stream;

error_log /var/log/nginx/error.log;

# TCP services

upstream tcp-3306-default-mysql-3306 {

server 10.2.30.2:3306;

}

server {

listen 3306;

listen [::]:3306;

proxy_timeout 600s;

proxy_pass tcp-3306-default-mysql-3306;

}

# UDP services

upstream udp-53-kube-system-kube-dns-53 {

server 10.2.54.9:53;

}

server {

listen 53 udp;

listen [::]:53 udp;

proxy_responses 1;

proxy_timeout 600s;

proxy_pass udp-53-kube-system-kube-dns-53;

}

nginx-ingress-vts里backend显示,但是server zone不显示

原来是本地ie浏览器连接里配了代理导致.

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub