Piszki Lab | EN

Today I met with a surprising message, he appeared in a production environment while trying to deploy a new virtual machine (OVF). The message sounded like this: Failed to open the /var/log/journal for write (There is no space left on device). It surprised me so much, that we keep logs on a drive other than root.

A quick check showed anything wrong:

The conclusion is simple, ran out of inodes, you can check this very quickly by this command (localcli system visorfs ramdisk list):

OK, the root disk exhausted the available inodes. Such errors are heavy in the diagnosis, i-nodes can be used by large numbers of small files or eg. The log file with a broken descriptor (we deleted a file to which the system is writing, we think that the file does not exist but it still grows). Fortunately, I managed to find a KB that covered my problem. To my incessant surprise, in /var/run/sfcb I found a lot of files:

The procedure of “cleaning” is simple, stop the watchdog, remove the files and start watchdog. Check inodes:

The number of busy inodes has fallen by half! How to defend in the future against this problem? Accordance with the KB redirect writing of those files to a RAM disk:

esxcli system visorfs ramdisk add –name sfcbtickets –min-size 0 –max-size 1024 –permissions 0755 –target /var/run/sfcb