Filesystem Full due to open files

Status of the root file system 

bash# df -kh /

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             4.0G  3.8G   16M 100% /

Notice here that the sum of  directories only adds up to around  1G by using command

  

bash# du -sh *

 8.7M    bin
23M     boot
80K     dev
113M    etc
27M     export

188K    gbin
71M     home
245M    lib
31M     lib64
16K     lost+found
8.0K    media
0       misc
8.0K    mnt

Next was to look at open files. It is worth mentioning here that even if a file is deleted, it’s space may not be reclaimed if the process that created it, or still using it, is still running. Using the lsof ( list open files ) utility will show these files.

bash# lsof |grep deleted

frmweb     3561 applmchp   81u      REG                8,2    7984736     458799 /tmp/file59plfd.TMP (deleted)

frmweb     5052 applmchp   50u      REG                8,2     457552     458897 /tmp/file22QA5y.TMP (deleted)

java            5584 applmchp  244w      REG                8,6      47650     262193 /var/tmp/bne.log (deleted)

gconfd-2   9595   oravnc   12wW     REG                8,2        609         54 /tmp/gconfd-oravnc/lock/0t1389892324ut687211u3027p9595r259524570k3221222552 (deleted)

gconfd-2   9652  oravnc2   12wW     REG                8,2        617         59 /tmp/gconfd-oravnc2/lock/0t1389892327ut429457u3404p9652r1030077876k3221222552 (deleted)

gconfd-2   9734  backvnc   12wW     REG                8,2        617         64 /tmp/gconfd-backvnc/lock/0t1389892330ut648713u3405p9734r92839103k3221222552 (deleted)

gconfd-2   9879   sysvnc   12wW     REG                8,2        609         69 /tmp/gconfd-sysvnc/lock/0t1389892333ut788397u3406p9879r1486644921k3221222552 (deleted)

frmweb    11186 applmchp   75u      REG                8,2  222875648     458790 /tmp/file96BiXl.TMP (deleted)

frmweb    20806 applmchp   74u      REG                8,2 3394768896     458893 /tmp/fileMvAktd.TMP (deleted)

frmweb    24330 applmchp  127u      REG                8,2    1807968     458795 /tmp/filez3JO0E.TMP (deleted)

frmweb    27349 applmchp   49u      REG                8,2    2702160     458902 /tmp/filecP6Ikd.TMP (deleted)

frmweb    28865 applmchp  109u      REG                8,2     715264     458808 /tmp/filemVtYQg.TMP (deleted)

frmweb    29012 applmchp   51u      REG                8,2     347024     458776 /tmp/filelImpTc.TMP (deleted)

[root@db265 tmp]# ps -ef|grep -i 3561

applmchp  3561  4066  0 18:05 ?        00:00:30 /app01/appl/mchpora/10.1.2/bin/frmweb server webfile=HTTP-0,0,1,default,34.237.63.199

This job was probably cancelled out for some reason, but was not cleaned up although the files were deleted. The background process was still running at the OS level.After Kill those process at the OS level, the space was reclaimed.

bash # df -h  /

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             4.0G  343M  3.5G   9% /

File system got full due to open files
lsof -a +L1 /dev/mapper/Test--APP01-app01|grep deleted

use below command to kill deleted process. 

sudo lsof | grep "(deleted)$" | sed -re 's/^\S+\s+(\S+)\s+\S+\s+([0-9]+).*/\1\/fd\/\2/' | while read file; do sudo bash -c ": > /proc/$file"; done