FS OUt of Space - Deleted Instead of Truncating - Cant Stop Process

Situation:

• File system out of space
• No room to move or compress files
• Deleted large files instead of truncating
• Disk space has not been freed up
• File is still held open because a running process still has the file handle open
• Can’t terminate process because it is important

Check disk usage and the size of our example file:

df -h .
du -sh /var/log/nginx/error.log

Check which process has this file open:

lsof /var/log/nginx/error.log

ps -ef |grep -i nginx

Truncate file

This is what you should do. It will clear the space used by the file without actually deleting it. The space will be freed imediately.

echo > /var/log/nginx/error.log

Delete file

Don’t do this if the file is being held open by a process. For example, log files will be held open constantly for writing. If you delete a file the space won’t be freed until all processes that had the file open are restarted. Alternately, if you have a way to tell the process to close the file that would work too.

rm /var/log/nginx/error.log

Disk space is not cleared because process is still writing to the file handle.

df -h .

Truncate Delted Files with Proc File System

List all file handles that still exist but the files have been deleted:

find /proc/*/fd -ls | grep deleted

Filter for the desired process and split out the column for the file descriptor. Make sure you get the right column. You might need to change the column number that is printed.

find /proc/*/fd -ls | grep deleted|grep nginx | awk '{print $11 }'

Loop over all of these file handles and truncate them all in one line ( check with the above commands before doing this ):

for i in `find /proc/*/fd -ls |grep deleted|grep nginx | awk '{print $11 }'`; do echo >$i; done

Problem solved … for now …. This should get you by until the process can be safely restarted. Keep in mind that this is still a suboptimal situation because we are no longer logging events.