k8s部署，系统获取真实客户端ip

我们生产中使用的是kong网关环境的架构也不同，第一种kong网管后走nginx，第二种kong网管后不走nginx

kong网管后走ingress-nginx

1. 修改kong的配置

配置要信任的原始IP地址列表，这里配置为全部信任

trusted_ips = 0.0.0.0/0,::/0

yaml文件添加:
        - name: KONG_TRUSTED_IPS
          value: 0.0.0.0/0,::/0
        - name: KONG_REAL_IP_HEADER
          value: X-Forwarded-For
        - name: KONG_REAL_IP_RECURSIVE
          value: "on"

2. 第二步：把kong改为三副本，分别跑在三个master节点上，增加反亲和策略

反亲和策略：
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - kong
            topologyKey: kubernetes.io/hostname
      containers:
    
节点调度：
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
          operator: Exists
      nodeSelector:
        node-role.kubernetes.io/master: ""

3. 配置kong的svc为local模式

kubectl -n kong patch svc kong-proxy -p '{"spec":{"externalTrafficPolicy":"Local"}}'

4. 配置nginx的svc为local模式

kubectl -n ingress-nginx patch svc ingress-nginx -p '{"spec":{"externalTrafficPolicy":"Local"}}'

5. 把nginx改为三副本，分别跑在三个master节点上

反亲和配置： 
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
            topologyKey: kubernetes.io/hostname
      containers:
      
节点调度：
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
          operator: Exists
      nodeSelector:
        node-role.kubernetes.io/master: ""

6. 配置nginx的configmap

apiVersion: v1
data:
  compute-full-forwarded-for: "true"
  forwarded-for-header: X-Forwarded-For
  proxy-body-size: 200m
  use-forwarded-headers: "true"
kind: ConfigMap

​ 通过上面的配置再nginx的中有源ip，通过抓包可以看到ip再x_original_forwarded_for，只要系统开启日志，设置好日志格式就可以取到真实ip

​ 系统nginx的部分配置

nginx.conf: |-
    user  root;
    worker_processes  1;
    worker_rlimit_nofile 100480;
    events {
        worker_connections  65536;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        charset utf-8;
        client_max_body_size 100m;
        log_format main ' $http_x_original_forwarded_for [$time_local] "$request" "$http_host"' '"$status" $body_bytes_sent "$http_referer"' '"$http_user_agent" $request_length';
        #客户端ip就在x_original_forwarded_for
        server {
                listen 80;
                server_name 域名
                index index.html index.htm index.jsp index.htm index.do index.action;
                root   /datadh/hanweb-jpaas-ui;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                large_client_header_buffers 4 16k;
                client_max_body_size 300m;
                client_body_buffer_size 128k;
                proxy_connect_timeout 600;
                proxy_read_timeout 600;
                proxy_send_timeout 600;
                proxy_buffer_size 64k;
                proxy_buffers 4 32k;
                proxy_busy_buffers_size 64k;
                proxy_temp_file_write_size 64k;
                access_log /var/log/nginx/access-001.log main;  #这个必须有

​ 那接下来说下我这遇到的问题：应用人和我说他们没有取到客户端ip，取到的都是虚拟机ip和pod的虚ip，我和是奇怪，就帮他们查看，日志确实没有，我就开始排查多次修改log_format main看看有啥效果，但是问题出现了，日志格式尽然没有变化过，所以我猜测他们的日志没有开启，结果还真是。。。。。能取到你才怪呢，这奥特曼也给你取不到啊。

他们的配置是这个样的

```
http {
    include       mime.types;
 ssl_protocols TLSv1.1 TLSv1.2;
 default_type  application/octet-stream;
    client_max_body_size 1024m;
  fastcgi_buffer_size 1024k;
  fastcgi_buffers 32 1024k;
  fastcgi_busy_buffers_size 2048k;
  fastcgi_temp_file_write_size 2048k;
    server_tokens off;
    sendfile        on;
    keepalive_timeout  65;
    proxy_connect_timeout 180;
    proxy_read_timeout 180;
    proxy_send_timeout 180;
    proxy_ignore_client_abort on;
     log_format main '$http_x_original_forwarded_for [$time_local] "$request" "$http_host"' '"$status" $body_bytes_sent "$http_referer"' '"$http_user_agent" $request_length';
```

```
    server {
        listen       80;
        server_name  yitihua;
        proxy_connect_timeout 240;
        proxy_read_timeout 240;
        proxy_send_timeout 240;

        #charset koi8-r;

        #access_log  /var/log/nginx/access.log  main;
        #日志都不开始取ip?
``

​ 所以这个问题是系统自己人傻！还屁颠屁颠说配置没问题。

7. 在系统应用获取真实ip后又遇到一个问题，如果客户端用伪装ip的话这个样取不到真实客户ip
   
   我的解决办法，修改系统nginx配置

nginx.conf: |-
    user  root;
    worker_processes  1;
    worker_rlimit_nofile 100480;
    events {
        worker_connections  65536;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        charset utf-8;
        client_max_body_size 100m;
        log_format main ' $http_x_original_forwarded_for [$time_local] "$request" "$http_host"' '"$status" $body_bytes_sent "$http_referer"' '"$http_user_agent" $request_length';
        #客户端ip就在x_original_forwarded_for
        server {
                listen 80;
                server_name 域名
                index index.html index.htm index.jsp index.htm index.do index.action;
                root   /datadh/hanweb-jpaas-ui;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                #将此处$proxy_add_x_forwarded_for修改为$remote_addr这样就可以取到真实ip和伪装ip
                large_client_header_buffers 4 16k;
                client_max_body_size 300m;
                client_body_buffer_size 128k;
                proxy_connect_timeout 600;
                proxy_read_timeout 600;
                proxy_send_timeout 600;
                proxy_buffer_size 64k;
                proxy_buffers 4 32k;
                proxy_busy_buffers_size 64k;
                proxy_temp_file_write_size 64k;
                access_log /var/log/nginx/access-001.log main;  #这个必须有

结果如下

kong网管后不走ingress-nginx，直接到系统

其实kong的配置是一样的，只是少了nginx的配置

这里需要注意，走nginx和不走nginx有区别，在走一层nginx，ip会放在xoff，不走nginx，ip会在xff,，所以针对系统取真实ip日志回去header有区别

通过抓包可以看到ip在x_forwarded_for，只要系统开启日志，设置好日志格式就可以取到真实ip