Here's my nginx conf:

http {
    include                        mime.types;
    default_type                   application/octet-stream;
    keepalive_timeout              65;
    server_names_hash_max_size     10000;
    server_names_hash_bucket_size  128;
    client_max_body_size           2m;

    server {
        listen    80;
        server_name     testheader.com;
        ignore_invalid_headers off;
        proxy_http_version 1.1;

        location ~* ^/testheader {
            proxy_request_buffering off;
            proxy_next_upstream off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Connection "";
            proxy_read_timeout 180s;

            set $upstream 127.0.0.1:8888;
            proxy_pass http://$upstream ;
        }
    }
}

I sent a sample request as following

POST /testhead/magic?null HTTP/1.1: 
content-type: text/plain; charset=UTF-8
test_trace_id: 1234567890123456
test_span_id: 1234567890123456
x-forwarded-for: 127.0.0.1:8080
x-test-client-host: 127.0.0.1:8888
x-test-request.toplevel.uuid: xxxxxxx-xxxxxx-xxxxxx-xxxxxxx
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding: gzip
connection: keep-alive
accept-language: null
host: top.ctrip.uat.qa.nt.ctripcorp.com
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Content-Length: 264


{"ActCode":2}

I expected to get all the invalid headers like test_trace_id, test_span_id, all x-{$value} from my upstream server 127.0.0.1:8888, but none of the headers was shown. In the mean time, nginx log didn't report any errors in sending invalid headers phase.

I also tried capturing requests at the target upstream server using tcpdump, and I'm for sure nginx did not even try to send them out.

However, in another sample:

POST /testhead/magic?null HTTP/1.1: 
content-type: text/plain; charset=UTF-8
test_trace_id: 1234567890123456
test_span_id: 1234567890123456
host: top.ctrip.uat.qa.nt.ctripcorp.com
x-forwarded-for: 127.0.0.1:8080
x-test-request.toplevel.uuid: xxxxxxx-xxxxxx-xxxxxx-xxxxxxx
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding: gzip
connection: keep-alive
accept-language: null
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Content-Length: 264


{"ActCode":3}

The invalid header x-test-request.toplevel.uuid was sent to the upstream server.

I don't think the missing two headers in the second sample is the cause of the issue. I tried many requests and it seems like this problem happens randomly, sometimes the directive works perfectly, sometimes it doesn't.

Is it a bug? The config file clearly tells nginx to ignore invalid headers and pass whatever it gets from client to the upstream. Nginx has no reason to throw those headers away.

Does anyone know how to fix it? Thanks in advance.

I have met the same problem.

You can have a interesting test. Send a request with some invalid headers before the Host header and some invalid headers after it.

You will find you can receive the invalid headers after the Host header, and the ones before the Host header are lost.

The solution to this problem is to set ignore_invalid_headers to off in http level.

About the cause of the problem. I think Nginx need to read the Host header before entering the server scope because the domain, in other words the server name, stored in the Host header. And Nginx discards all invalid headers before the Host header in this phase, because the directive ignore_invalid_headers off in server scope doesn't work in this phase. All other headers will be read in server scope, so the invalid headers after Host header can be read, because the directive ignore_invalid_headers off in server scope tells Nginx to ignore the invalid headers.

So you need to tell Nginx not to discard the invalid headers before the Host header using the directive ignore_invalid_headers off in http level.