Low Orbit Flux Logo 2 F

FS OUt of Space - Deleted Instead of Truncating - Cant Stop Process


Check disk usage and the size of our example file:

df -h .
du -sh /var/log/nginx/error.log

Check which process has this file open:

lsof /var/log/nginx/error.log

ps -ef |grep -i nginx

Truncate file

This is what you should do. It will clear the space used by the file without actually deleting it. The space will be freed imediately.

echo > /var/log/nginx/error.log

Delete file

Don’t do this if the file is being held open by a process. For example, log files will be held open constantly for writing. If you delete a file the space won’t be freed until all processes that had the file open are restarted. Alternately, if you have a way to tell the process to close the file that would work too.

rm /var/log/nginx/error.log

Disk space is not cleared because process is still writing to the file handle.

df -h .

Truncate Delted Files with Proc File System

List all file handles that still exist but the files have been deleted:

find /proc/*/fd -ls | grep deleted

Filter for the desired process and split out the column for the file descriptor. Make sure you get the right column. You might need to change the column number that is printed.

find /proc/*/fd -ls | grep deleted|grep nginx | awk '{print $11 }'

Loop over all of these file handles and truncate them all in one line ( check with the above commands before doing this ):

for i in `find /proc/*/fd -ls |grep deleted|grep nginx | awk '{print $11 }'`; do echo >$i; done

Problem solved … for now …. This should get you by until the process can be safely restarted. Keep in mind that this is still a suboptimal situation because we are no longer logging events.