在nginx和php-fpm下一访问nginx就瞬间502的问题 php-fpmsignal 7 (SIGBUS)

故障现象
使用TinkPHP3.2.x框架，页面偶尔会出现一访问nginx就报502 bad gateway，并不是等一段时间后nginx才报502，打开页面的一瞬间就502了。

php-fpm日志

[28-Sep-2015 23:25:07] WARNING: [pool www] child 2965 exited on signal 7 (SIGBUS) after 319.686109 seconds from start

[28-Sep-2015 23:25:07] NOTICE: [pool www] child 3223 started

[28-Sep-2015 23:25:49] WARNING: [pool www] child 1251 exited on signal 7 (SIGBUS) after 1005.893950 seconds from start

[28-Sep-2015 23:25:49] NOTICE: [pool www] child 3242 started

[28-Sep-2015 23:26:46] WARNING: [pool www] child 1467 exited on signal 7 (SIGBUS) after 1057.791921 seconds from start

[28-Sep-2015 23:26:46] NOTICE: [pool www] child 3284 started

[28-Sep-2015 23:30:33] WARNING: [pool www] child 1178 exited on signal 7 (SIGBUS) after 1289.872260 seconds from start

[28-Sep-2015 23:30:33] NOTICE: [pool www] child 3450 started

[28-Sep-2015 23:30:35] WARNING: [pool www] child 1115 exited on signal 7 (SIGBUS) after 1292.014337 seconds from start

[28-Sep-2015 23:30:35] NOTICE: [pool www] child 3451 started

[28-Sep-2015 23:30:35] WARNING: [pool www] child 1487 exited on signal 7 (SIGBUS) after 1285.395573 seconds from start

[28-Sep-2015 23:30:35] NOTICE: [pool www] child 3452 started

[28-Sep-2015 23:30:36] WARNING: [pool www] child 1958 exited on signal 7 (SIGBUS) after 1181.206336 seconds from start

[28-Sep-2015 23:30:36] NOTICE: [pool www] child 3453 started

可以看到有大量的 php-fpm进程收到 signal 7 退出了， 由于fastcgi php-fpm 瞬间退出了，nginx就瞬间502了

对php-fpm进行调试
既然看不出来头绪就需要进行调试了，这里需要把php-fpm退出的时候导出coredump

php-fpm coredump方法
核心文件，也称核心转储，是操作系统在进程收到某些信号而终止运行时，将此时进程地址空间的内容以及有关进程状态的其他信息写出的一个磁盘文件。这种信息往往用于调试。 核心文件一词来源于磁芯内存。

php需要打开debug参数，如果编译的时候没有打开，需要重新编译，以下是我这里的

--enable-debug
--prefix=/usr/local/php --with-config-file-path=/usr/local/php/etc --enable-fpm --with-fpm-user=www --with-fpm-group=www --with-mysql=mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --with-iconv-dir --with-freetype-dir=/usr/local/freetype --with-jpeg-dir --with-png-dir --with-zlib --with-libxml-dir=/usr --enable-xml --disable-rpath --enable-bcmath --enable-shmop --enable-sysvsem --enable-inline-optimization --with-curl --enable-mbregex --enable-mbstring --with-mcrypt --enable-ftp --with-gd --enable-gd-native-ttf --with-openssl --with-mhash --enable-pcntl --enable-sockets --with-xmlrpc --enable-zip --enable-soap --with-gettext --disable-fileinfo --enable-debug

修改dump出来的路径

echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern

允许程序崩溃的时候dump

ulimit -c unlimited

关闭dump功能

ulimit -c 0

用gdb进行调试

gdb /usr/local/php/sbin/php-fpm core.php-fpm.4555.web1.iamle.com.1443458079

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/local/php/sbin/php-fpm...done.

[New Thread 4555]

Missing separate debuginfo for /usr/lib/libmcrypt.so.4

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c5/699ec0c783a6055a7f983f9eb64146371eb2e6

Missing separate debuginfo for /usr/lib/libpng12.so.0

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9e/65badd636c1f2e81682a8c498af098b5bbd063

Missing separate debuginfo for /usr/lib/libjpeg.so.62

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/6b2c6a8cb0b42dda3d57a1b5d3670ab99c83bd

Missing separate debuginfo for /usr/local/lib/libiconv.so.2

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/df/7400febafa74aeef81e48b638c9fde24cb2beb

Missing separate debuginfo for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/64/1aa8440943921a4d553e380282d15828f18377

Missing separate debuginfo for

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fa/36da47e774d9bc5a739905887a5d25790090f7

Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypt.so.1

Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libz.so.1

Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/librt.so.1

Reading symbols from /usr/lib/libmcrypt.so.4...done.

Loaded symbols for /usr/lib/libmcrypt.so.4

Reading symbols from /usr/lib/libpng12.so.0...Missing separate debuginfo for /usr/lib/libpng12.so.0

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9e/65badd636c1f2e81682a8c498af098b5bbd063.debug

(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libpng12.so.0

Reading symbols from /usr/lib/libjpeg.so.62...Missing separate debuginfo for /usr/lib/libjpeg.so.62

Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/6b2c6a8cb0b42dda3d57a1b5d3670ab99c83bd.debug

(no debugging symbols found)...done.

Loaded symbols for /usr/lib/libjpeg.so.62

Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libcurl.so.4

Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnsl.so.1

Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libxml2.so.2

Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssl.so.10

Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libcrypto.so.10

Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libfreetype.so.6

Reading symbols from /usr/local/lib/libiconv.so.2...done.

Loaded symbols for /usr/local/lib/libiconv.so.2

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libresolv.so.2

Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libfreebl3.so

Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

[Thread debugging using libthread_db enabled]

Loaded symbols for /lib64/libpthread.so.0

Reading symbols from /lib64/libidn.so.11...(no debugging symbols found)...done.

Loaded symbols for /lib64/libidn.so.11

Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libldap-2.4.so.2

Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libgssapi_krb5.so.2

Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkrb5.so.3

Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.

Loaded symbols for /lib64/libk5crypto.so.3

Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcom_err.so.2

Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssl3.so

Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libsmime3.so

Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libnss3.so

Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libnssutil3.so

Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libplds4.so

Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libplc4.so

Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnspr4.so

Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libssh2.so.1

Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/liblber-2.4.so.2

Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libsasl2.so.2

Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkrb5support.so.0

Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkeyutils.so.1

Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libselinux.so.1

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/networkbench.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/networkbench.so

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

Core was generated by `php-fpm: pool www                                                             '.

Program terminated with signal 7, Bus error.

#0  lex_scan (zendlval=0x7fff8083fae8) at Zend/zend_language_scanner.c:2265

2265 switch (yych) {

Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6.x86_64 freetype-2.3.11-14.el6_3.1.x86_64 glibc-2.12-1.149.el6_6.5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libcurl-7.19.7-40.el6_6.1.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6.x86_64 libxml2-2.7.6-17.el6_6.1.x86_64 nspr-4.10.6-1.el6_5.x86_64 nss-3.16.1-14.el6.x86_64 nss-softokn-freebl-3.14.3-17.el6.x86_64 nss-util-3.16.1-3.el6.x86_64 openldap-2.4.39-8.el6.x86_64 openssl-1.0.1e-30.el6_6.4.x86_64 zlib-1.2.3-29.el6.x86_64

(gdb) bt

#0  lex_scan (zendlval=0x7fff8083fae8) at Zend/zend_language_scanner.c:1088

#1  0x00000000007ad730 in zendlex (zendlval=0x7fff8083fae0) at /root/oneshell-master/php-5.4.45/Zend/zend_compile.c:6545

#2  0x0000000000795aca in zendparse () at /root/oneshell-master/php-5.4.45/Zend/zend_language_parser.c:3471

#3  0x00000000007a2990 in compile_file (file_handle=0x7fff8083fe30, type=2) at Zend/zend_language_scanner.l:585

#4  0x000000000063255a in phar_compile_file (file_handle=0x7fff8083fe30, type=2) at /root/oneshell-master/php-5.4.45/ext/phar/phar.c:3414

#5  0x00000000007a20bb in compile_filename (type=2, filename=0x299d018) at Zend/zend_language_scanner.l:628

#6  0x00000000008156de in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0x7f46209c6158) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:27443

#7  0x0000000000832c80 in execute (op_array=0x27b4158) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#8  0x00007f4614b27db6 in nb_execute (op_array=0x27b4158) at /tingyun/nbprof/nbprof.c:318

#9  0x00000000007c0e14 in zend_call_function (fci=0x7fff808401b0, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956

#10 0x00000000006d8ae5 in zif_call_user_func_array (ht=<value optimized out>, return_value=0x29733b0, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>,

    return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/ext/standard/basic_functions.c:4754

#11 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c5f78, return_value_used=1) at /tingyun/nbprof/nbprof.c:403

#12 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645

#13 0x0000000000832c80 in execute (op_array=0x7f46209f8a70) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#14 0x00007f4614b27db6 in nb_execute (op_array=0x7f46209f8a70) at /tingyun/nbprof/nbprof.c:318

#15 0x00000000007c0e14 in zend_call_function (fci=0x7fff80840540, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956

#16 0x00000000007e41bf in zend_call_method (object_pp=0x0, obj_ce=<value optimized out>, fn_proxy=0x7f46209f6618, function_name=0xb02423 "__callstatic",

    function_name_len=<value optimized out>, retval_ptr_ptr=0x7fff80840668, param_count=2, arg1=0x29730a8, arg2=0x29735c0)

    at /root/oneshell-master/php-5.4.45/Zend/zend_interfaces.c:97

#17 0x00000000007efb39 in zend_std_callstatic_user_call (ht=<value optimized out>, return_value=0x2972f58, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>,

    return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_object_handlers.c:1110

#18 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c5d88, return_value_used=0) at /tingyun/nbprof/nbprof.c:403

#19 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645

#20 0x0000000000832c80 in execute (op_array=0x304d188) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#21 0x00007f4614b27db6 in nb_execute (op_array=0x304d188) at /tingyun/nbprof/nbprof.c:318

#22 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#23 0x0000000000832c80 in execute (op_array=0x2831020) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#24 0x00007f4614b27db6 in nb_execute (op_array=0x2831020) at /tingyun/nbprof/nbprof.c:318

#25 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#26 0x0000000000832c80 in execute (op_array=0x27a48a0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#27 0x00007f4614b27db6 in nb_execute (op_array=0x27a48a0) at /tingyun/nbprof/nbprof.c:318

#28 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#29 0x0000000000832c80 in execute (op_array=0x27a25b8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#30 0x00007f4614b27db6 in nb_execute (op_array=0x27a25b8) at /tingyun/nbprof/nbprof.c:318

#31 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#32 0x0000000000832c80 in execute (op_array=0x2832410) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#33 0x00007f4614b27db6 in nb_execute (op_array=0x2832410) at /tingyun/nbprof/nbprof.c:318

---Type <return> to continue, or q <return> to quit---

#34 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#35 0x0000000000832c80 in execute (op_array=0x282c1e0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#36 0x00007f4614b27db6 in nb_execute (op_array=0x282c1e0) at /tingyun/nbprof/nbprof.c:318

#37 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#38 0x0000000000832c80 in execute (op_array=0x2896bb8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#39 0x00007f4614b27db6 in nb_execute (op_array=0x2896bb8) at /tingyun/nbprof/nbprof.c:318

#40 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#41 0x0000000000832c80 in execute (op_array=0x286ced8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#42 0x00007f4614b27db6 in nb_execute (op_array=0x286ced8) at /tingyun/nbprof/nbprof.c:318

#43 0x00000000007c0e14 in zend_call_function (fci=0x7fff80841170, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956

#44 0x000000000064ad3f in zim_reflection_method_invoke (ht=<value optimized out>, return_value=0x288e4e8, return_value_ptr=<value optimized out>, this_ptr=0x2859330,

    return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/ext/reflection/php_reflection.c:2898

#45 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c0530, return_value_used=0) at /tingyun/nbprof/nbprof.c:403

#46 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645

#47 0x0000000000832c80 in execute (op_array=0x27a5a58) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#48 0x00007f4614b27db6 in nb_execute (op_array=0x27a5a58) at /tingyun/nbprof/nbprof.c:318

#49 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#50 0x0000000000832c80 in execute (op_array=0x27a5b58) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#51 0x00007f4614b27db6 in nb_execute (op_array=0x27a5b58) at /tingyun/nbprof/nbprof.c:318

#52 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#53 0x0000000000832c80 in execute (op_array=0x27b66a0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#54 0x00007f4614b27db6 in nb_execute (op_array=0x27b66a0) at /tingyun/nbprof/nbprof.c:318

#55 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673

#56 0x0000000000832c80 in execute (op_array=0x7f46209f4108) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#57 0x00007f4614b27e84 in nb_execute (op_array=0x7f46209f4108) at /tingyun/nbprof/nbprof.c:318

#58 0x0000000000816a9c in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER (execute_data=0x7f46209bd318) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:2650

#59 0x0000000000832c80 in execute (op_array=0x7f46209f35b8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410

#60 0x00007f4614b27dff in nb_execute (op_array=0x7f46209f35b8) at /tingyun/nbprof/nbprof.c:318

#61 0x00000000007cc5ae in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /root/oneshell-master/php-5.4.45/Zend/zend.c:1329

#62 0x000000000077221e in php_execute_script (primary_file=0x7fff80845fc0) at /root/oneshell-master/php-5.4.45/main/main.c:2502

#63 0x000000000087b34e in main (argc=<value optimized out>, argv=<value optimized out>) at /root/oneshell-master/php-5.4.45/sapi/fpm/fpm/fpm_main.c:1938

(gdb)

发现 php的lex_scan执行的时候崩溃了，通过搜索明白是某php正在被词法分析的时候又被修改了。

最后确定是TinkPHP开启调试模式后，模板是动态编译的，这样当有并发访问的时候就发生了php文件又在被解析，又在被写入的情况导致lex_scan的时候php-fpm就挂掉了。

当有并发的时候关闭调试模式即可解决

也算是见识了php程序的问题可以让php-fpm进程都挂掉

参考
提供SIGABRT的 backtrace ，如何提供backtrace, 请参看:
http://bugs.php.net/bugs-generating-backtrace.php for *NIX and
http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32

Generating core-dump for php5-fpm
https://rtcamp.com/tutorials/php/core-dump-php5-fpm/

对nginx报502bad gateway的一次原因定位
http://blog.chunshiban.com/2013/07/05/%E5%AF%B9nginx%E6%8A%A5502bad-gateway%E7%9A%84%E4%B8%80%E6%AC%A1%E5%8E%9F%E5%9B%A0%E5%AE%9A%E4%BD%8D/

一例php进程的SIGBUS故障
http://blog.druggo.org/post/2013/05/02/%E4%B8%80%E4%BE%8Bphp%E8%BF%9B%E7%A8%8B%E7%9A%84SIGBUS%E6%95%85%E9%9A%9C

低并发502错误signal 7 (SIGBUS)
http://kokahkhk.blog.163.com/blog/static/209428040201411595622729/

压力测试下httpd进程由于lex_scan和Runtime缓存文件而崩溃
http://www.thinkphp.cn/topic/27464.html

php进程崩溃跟踪方法
http://blog.chinaunix.net/uid-23504396-id-4819011.html