Filesystem Full due to open files

Status of the root file system 

bash# df -kh /

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             4.0G  3.8G   16M 100% /

Notice here that the sum of  directories only adds up to around  1G by using command

  

bash# du -sh *

 8.7M    bin
23M     boot
80K     dev
113M    etc
27M     export

188K    gbin
71M     home
245M    lib
31M     lib64
16K     lost+found
8.0K    media
0       misc
8.0K    mnt

Next was to look at open files. It is worth mentioning here that even if a file is deleted, it’s space may not be reclaimed if the process that created it, or still using it, is still running. Using the lsof ( list open files ) utility will show these files.

bash# lsof |grep deleted

frmweb     3561 applmchp   81u      REG                8,2    7984736     458799 /tmp/file59plfd.TMP (deleted)

frmweb     5052 applmchp   50u      REG                8,2     457552     458897 /tmp/file22QA5y.TMP (deleted)

java            5584 applmchp  244w      REG                8,6      47650     262193 /var/tmp/bne.log (deleted)

gconfd-2   9595   oravnc   12wW     REG                8,2        609         54 /tmp/gconfd-oravnc/lock/0t1389892324ut687211u3027p9595r259524570k3221222552 (deleted)

gconfd-2   9652  oravnc2   12wW     REG                8,2        617         59 /tmp/gconfd-oravnc2/lock/0t1389892327ut429457u3404p9652r1030077876k3221222552 (deleted)

gconfd-2   9734  backvnc   12wW     REG                8,2        617         64 /tmp/gconfd-backvnc/lock/0t1389892330ut648713u3405p9734r92839103k3221222552 (deleted)

gconfd-2   9879   sysvnc   12wW     REG                8,2        609         69 /tmp/gconfd-sysvnc/lock/0t1389892333ut788397u3406p9879r1486644921k3221222552 (deleted)

frmweb    11186 applmchp   75u      REG                8,2  222875648     458790 /tmp/file96BiXl.TMP (deleted)

frmweb    20806 applmchp   74u      REG                8,2 3394768896     458893 /tmp/fileMvAktd.TMP (deleted)

frmweb    24330 applmchp  127u      REG                8,2    1807968     458795 /tmp/filez3JO0E.TMP (deleted)

frmweb    27349 applmchp   49u      REG                8,2    2702160     458902 /tmp/filecP6Ikd.TMP (deleted)

frmweb    28865 applmchp  109u      REG                8,2     715264     458808 /tmp/filemVtYQg.TMP (deleted)

frmweb    29012 applmchp   51u      REG                8,2     347024     458776 /tmp/filelImpTc.TMP (deleted)

[root@db265 tmp]# ps -ef|grep -i 3561

applmchp  3561  4066  0 18:05 ?        00:00:30 /app01/appl/mchpora/10.1.2/bin/frmweb server webfile=HTTP-0,0,1,default,34.237.63.199

This job was probably cancelled out for some reason, but was not cleaned up although the files were deleted. The background process was still running at the OS level.After Kill those process at the OS level, the space was reclaimed.

bash # df -h  /

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             4.0G  343M  3.5G   9% /

File system got full due to open files
lsof -a +L1 /dev/mapper/Test--APP01-app01|grep deleted

use below command to kill deleted process. 

sudo lsof | grep "(deleted)$" | sed -re 's/^\S+\s+(\S+)\s+\S+\s+([0-9]+).*/\1\/fd\/\2/' | while read file; do sudo bash -c ": > /proc/$file"; done

 

  

Nmap ("Network Mapper")

Nmap ("Network Mapper") is an open source tool for network exploration and security auditing. It was designed to rapidly scan large networks, although it works fine against single hosts. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. While Nmap is commonly used for security audits, many systems and network administrators find it useful for routine tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime.

Examples:

scan specific ports

bash # nmap -v -A -p 80,443 google.com

Scan all ports with * wildcard

bash # nmap -p "*" 10.128.10.10

 

Show all packets sent and received

bash # nmap --packet-trace 10.128.10.10

 

Show open ports

bash # nmap --open 10.128.10.10

prtdiag hung

The Platform Information and Control Library (PICL) provides a mechanism to publish platform-specific information for clients to access in a platform-independent way. picld maintains and controls access to the PICL information from clients and plug-in modules. The daemon is started in both single-user and multi-user boot mode.

The prtdiag command summarizes system component status in table format.

bash# prtdiag

svcs--> Provide information about services, including their status

bash# svcs -a|grep -i picl

online         00:49:58 svc:/system/picl:default

svcadm--> Manage the state of service instances

bash# svcadm restart picl

bash#  svcs -a|grep -i picl

online         14:49:58 svc:/system/picl:default

bash# prtdiag

System Configuration:  Oracle Corporation  sun4v SPARC Enterprise T2000

Memory size: 32640 Megabytes

================================ Virtual CPUs ================================