Fedora 26 No Space Left on Device with Plenty of Space Available

scottalanmiller

Here is a new one. We have one Fedora 26 server that has "filled up" recently, but has never come close to being full. The system works, then suddenly says "no space left on device" when working on the root filesystem (ext4.)

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        989M     0  989M   0% /dev
tmpfs          1000M   28K 1000M   1% /dev/shm
tmpfs          1000M  564K  999M   1% /run
tmpfs          1000M     0 1000M   0% /sys/fs/cgroup
/dev/vda1        40G   24G   14G  63% /
tmpfs          1000M     0 1000M   0% /tmp
tmpfs           200M     0  200M   0% /run/user/0

And here is du...

# du -shx /
24G	/

And from mount...

/dev/vda1 on / type ext4 (rw,relatime,data=ordered)

But if you go to / and do anything, no luck...

# cd /
# touch test
touch: cannot touch 'test': No space left on device

It is acting like it has gone read only, but mount doesn't show that as having happened. You can make files in /tmp of course, as that is a different filesystem (RAM disk.) So the issue is definitely the one partition.

scottalanmiller

And I should note, doing a reboot fixes it as if there was no problem. And of course logs stop because it can't write to them.

stacksofplates

How big is your LV?

stacksofplates

Also assuming inodes aren't full?

scottalanmiller

@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

Also assuming inodes aren't full?

F Me!!! Of course they are.

RamblingBiped

lsof +L1

FTW...

RamblingBiped

I've seen this behavior a fair amount on systems where log rotation for java applications is misconfigured. Log files are removed while the java application is still writing to them so it shows the file being deleted even though the connection to the inode is still held open.

A reboot is usually the quickest way to release the inodes.

Alex Sage

@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

Also assuming inodes aren't full?

F Me!!! Of course they are.

@scottalanmiller Need a video on inodes

scottalanmiller

@aaronstuder said in Fedora 26 No Space Left on Device with Plenty of Space Available:

@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

Also assuming inodes aren't full?

F Me!!! Of course they are.

@scottalanmiller Need a video on inodes

I do? I might, lol.

travisdh1

@aaronstuder said in Fedora 26 No Space Left on Device with Plenty of Space Available:

@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:

@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

Also assuming inodes aren't full?

F Me!!! Of course they are.

@scottalanmiller Need a video on inodes

"This is the database that tracks where your files are stored." That'd take me 15 seconds

Obsolesce

Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.

Maybe I need a good explanation, if someone doesn't mind.

stacksofplates

@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:

Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.

Maybe I need a good explanation, if someone doesn't mind.

This is the same limitation any FS has. It's just metadata. If you have a ton of files you will run out of available inodes. Containers are an area where this can happen easily since all of the containers can share the same FS as the host.

This is also usually more of an issue with EXT file systems (like we had here). INodes are set at FS creation and cannot be changed in EXT. Whereas XFS is dynamic (but has a limit that's based on percentage).

wirestyle22

"An inode is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. A data structure is a way of storing data so that it can be used efficiently."

wirestyle22

@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:

@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:

Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.

Maybe I need a good explanation, if someone doesn't mind.

This is the same limitation any FS has. It's just metadata. If you have a ton of files you will run out of available inodes. Containers are an area where this can happen easily since all of the containers can share the same FS as the host.

This is also usually more of an issue with EXT file systems (like we had here). INodes are set at FS creation and cannot be changed in EXT. Whereas XFS is dynamic (but has a limit that's based on percentage).

~~Does the inode only store the information when the file is in use~~? It doesn't.

wirestyle22

"When a file is created, it is assigned both a name and an inode number, which is an integer that is unique within the filesystem. Both the file names and their corresponding inode numbers are stored as entries in the directory that appears to the user to contain the files. That is, the directory associates file names with inodes.

Whenever a user or a program refers to a file by name, the operating system uses that name to look up the corresponding inode, which then enables the system to obtain the information it needs about the file to perform further operations. That is, a file name in a Unix-like operating system is merely an entry in a table with inode numbers, rather than being associated directly with a file (in contrast to other operating systems such as the Microsoft Windows systems). The inode numbers and their corresponding inodes are held in inode tables, which are stored in strategic locations in a filesystem, including near its beginning."

wirestyle22

So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently

Obsolesce

Thanks for the explanations. It helped me search better, and I eventually found this: https://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html

Shows how to check available inode space and how to fix it as well.

As for Scott's problem, I'm guessing something creating a lot of temp files that are cleared during a reboot.

wirestyle22

@tim_g If it's the same file system wouldn't that keep the number of temporary files down to a minimum? Legitimately asking, not being snarky

stacksofplates

@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently

Usually comes from orphaned INodes. Can happen if the process halts or server is shutdown incorrectly. This is what the lost+found directory is for.

scottalanmiller

@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:

So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently

It did not. Deleting millions of unneeded files did.