Piszki Lab | EN

My case study in the clouds…

ESXi 5.5 – Exhaustion of root disk inodes (There is no space left on device).

| 0 comments

Today I met with a surprising message, he appeared in a production environment while trying to deploy a new virtual machine (OVF). The message sounded like this: Failed to open the /var/log/journal for write (There is no space left on device). It surprised me so much, that we keep logs on a drive other than root.

journal1

A quick check showed anything wrong:

journal2

The conclusion is simple, ran out of inodes, you can check this very quickly by this command (localcli system visorfs ramdisk list):

journal3

OK, the root disk exhausted the available inodes. Such errors are heavy in the diagnosis, i-nodes can be used by large numbers of small files or eg. The log file with a broken descriptor (we deleted a file to which the system is writing, we think that the file does not exist but it still grows). Fortunately, I managed to find a KB that covered my problem. To my incessant surprise, in /var/run/sfcb I found a lot of files:

journal4

The procedure of “cleaning” is simple, stop the watchdog, remove the files and start watchdog. Check inodes:

journal5

The number of busy inodes has fallen by half! How to defend in the future against this problem? Accordance with the KB redirect writing of those files to a RAM disk:

esxcli system visorfs ramdisk add –name sfcbtickets –min-size 0 –max-size 1024 –permissions 0755 –target /var/run/sfcb

journal6

Rate this article:
[Total: 2 Average: 1.5]

Author: Piotr Pisz

Computer always, since I got a Commodore 64 at the end of primary school, through his beloved Amiga and Linux infinite number of consoles, until today, fully virtual day. Since 2001, Unix/Linux Systems Administrator, for seven years a faithful companion and protector of Solaris system, until his sad end. In the year 2011 came in the depths of virtualization, then smoothly ascended into the clouds and continues there today. Professionally working as Systems Architect in the Polish Security Printing Works.

Leave a Reply

Required fields are marked *.