问题描述:

警告文件磁盘已经满了,通过:df -h, du -sh * 等指令配合使用,发现当前系统任然存在大量可以使用的空间。大量剩余的磁盘空间不清楚怎么丢失了…

原因分析:

通过指令:lsof | grep deleted 指令,查看当前系统句柄未释放情况

[root@bigdata-hadoop2:/proc/21893/fd]# lsof | grep deleted
zabbix_ag  1791        zabbix    3wW     REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag  1806        zabbix    3w      REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag  1807        zabbix    3w      REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
zabbix_ag  1808        zabbix    3w      REG              253,1         4     132502 /var/tmp/zabbix_agentd.pid (deleted)
python     3946          root   14w      REG              253,1         0     393228 /tmp/cmflistener-stderr---agent-3916-1532518455-HYSskK.log (deleted)
catalogd  21893        impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893        impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893        impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893        impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893        impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893        impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893        impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893        impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893        impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893        impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  2502  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  2502  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  2502  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  2502  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  2502  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  2502  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  2502  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  2502  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  2502  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  2502  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  5443  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  5443  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  5443  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  5443  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  5443  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  5443  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  5443  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  5443  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  5443  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  5443  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  5834  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  5834  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  5834  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  5834  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  5834  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  5834  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  5834  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  5834  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  5834  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  5834  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  6688  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  6688  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  6688  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  6688  impala  259r      REG              253,1  72198042     396110 /tmp/9a5ce9d5-fa60-465c-b400-3ea2eeca9a45.jar (deleted)
catalogd  21893  6688  impala  260r      REG              253,1         1     396111 /tmp/fc8ba5b5-ddd8-4317-9e18-bd335584c04f.jar (deleted)
catalogd  21893  6688  impala  262r      REG              253,1 153712947     396112 /tmp/a7378ee0-82cf-4435-b05d-5fbe8248942b.jar (deleted)
catalogd  21893  6688  impala  265r      REG              253,1         1     396113 /tmp/f29f8a20-060f-4286-84e6-b1a958604c8c.jar (deleted)
catalogd  21893  6688  impala  267r      REG              253,1 153714277     396114 /tmp/5fdd988e-fbfa-43ae-a790-c277cab2304b.jar (deleted)
catalogd  21893  6688  impala  269r      REG              253,1         1     396115 /tmp/e37795e5-f53f-49da-8b21-27516241c95f.jar (deleted)
catalogd  21893  6688  impala  270r      REG              253,1         1     396183 /tmp/b0f21624-5850-4c60-b2f4-125656d5ac40.jar (deleted)
catalogd  21893  6689  impala  254r      REG              253,1 163923099     393230 /tmp/225e43c3-03a8-4b24-ba2e-14637e4d5bba.jar (deleted)
catalogd  21893  6689  impala  255r      REG              253,1  77023830     396109 /tmp/ee702167-27f2-4685-8217-8cec85f53a3c.jar (deleted)
catalogd  21893  6689  impala  258r      REG              253,1  72198042     393231 /tmp/2aeb7bb8-5bed-402d-9201-5c23859754d3.jar (deleted)
catalogd  21893  6689  impala  259r      REG              253,1  72198042 

通过结果可以发现,当前系统存在大量文件已经被删除(deleted),可是文件句柄并没有被释放的情况。

问题已经有答案了,因为系统中存在大量文件句柄没有释放,导致当前占用的磁盘空间无法释放。文件句柄释放不掉的根本原因,是因为当前有线程对句柄进行引用。

解决方案:

通过查询发现,有两种解决方案:

  1. 将当前线程进行重启,关闭线程,从而让句柄释放,释放空间
  2. 找到指定的文件句柄,将当前文件句柄的大小设置为空

根据当前的业务情况,需要对外提供实时服务,所以频繁重启不适合当前业务场景,那么只能选择将文件句柄内容置空。文件句柄内容置空,文件的内容大小被置空,释放了磁盘空间,但是文件句柄并没有释放,可能导致【Too Many File is Open】 错误。

lsof输出各列信息的意义如下:

lsof
command     PID USER   FD      type             DEVICE     SIZE       NODE NAME

lsof输出各列信息的意义如下:

COMMAND:进程的名称
PID:进程标识符
PPID:父进程标识符(需要指定-R参数)
USER:进程所有者
PGID:进程所属组
FD:文件描述符,应用程序通过文件描述符识别该文件。

通过losf 获取当前文件的 PID 和 FD 两个参数
置空当前的文件内容:

echo > /proc/PID/fd/FD

原文链接:https://blog.csdn.net/u014730165/article/details/82147150?spm=1001.2101.3001.6650.1&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.no_search_link&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.no_search_link

Logo

更多推荐