Piszki Lab | EN

My case study in the clouds…

ESXi 5.5 – Exhaustion of root disk inodes (There is no space left on device).


Today I met with a surprising message, he appeared in a production environment while trying to deploy a new virtual machine (OVF). The message sounded like this: Failed to open the /var/log/journal for write (There is no space left on device). It surprised me so much, that we keep logs on a drive other than root.


A quick check showed anything wrong:


The conclusion is simple, ran out of inodes, you can check this very quickly by this command (localcli system visorfs ramdisk list):


OK, the root disk exhausted the available inodes. Such errors are heavy in the diagnosis, i-nodes can be used by large numbers of small files or eg. The log file with a broken descriptor (we deleted a file to which the system is writing, we think that the file does not exist but it still grows). Fortunately, I managed to find a KB that covered my problem. To my incessant surprise, in /var/run/sfcb I found a lot of files:


The procedure of “cleaning” is simple, stop the watchdog, remove the files and start watchdog. Check inodes:


The number of busy inodes has fallen by half! How to defend in the future against this problem? Accordance with the KB redirect writing of those files to a RAM disk:

esxcli system visorfs ramdisk add –name sfcbtickets –min-size 0 –max-size 1024 –permissions 0755 –target /var/run/sfcb


Rate this article:
[Total: 2 Average: 1.5]

Author: Piotr Pisz

Computer always, since I got a Commodore 64 at the end of primary school, through his beloved Amiga and Linux infinite number of consoles, until today, fully virtual day. Since 2001, Unix/Linux Systems Administrator, for seven years a faithful companion and protector of Solaris system, until his sad end. In the year 2011 came in the depths of virtualization, then smoothly ascended into the clouds and continues there today. Professionally working as Systems Architect in the Polish Security Printing Works.

Leave a Reply

Required fields are marked *.