Fedora 26 No Space Left on Device with Plenty of Space Available
-
lsof +L1
FTW...
-
I've seen this behavior a fair amount on systems where log rotation for java applications is misconfigured. Log files are removed while the java application is still writing to them so it shows the file being deleted even though the connection to the inode is still held open.
A reboot is usually the quickest way to release the inodes.
-
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Also assuming inodes aren't full?
F Me!!! Of course they are.
@scottalanmiller Need a video on inodes
-
@aaronstuder said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Also assuming inodes aren't full?
F Me!!! Of course they are.
@scottalanmiller Need a video on inodes
I do? I might, lol.
-
@aaronstuder said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Also assuming inodes aren't full?
F Me!!! Of course they are.
@scottalanmiller Need a video on inodes
"This is the database that tracks where your files are stored." That'd take me 15 seconds
-
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
This is the same limitation any FS has. It's just metadata. If you have a ton of files you will run out of available inodes. Containers are an area where this can happen easily since all of the containers can share the same FS as the host.
This is also usually more of an issue with EXT file systems (like we had here). INodes are set at FS creation and cannot be changed in EXT. Whereas XFS is dynamic (but has a limit that's based on percentage).
-
"An inode is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. A data structure is a way of storing data so that it can be used efficiently."
-
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
This is the same limitation any FS has. It's just metadata. If you have a ton of files you will run out of available inodes. Containers are an area where this can happen easily since all of the containers can share the same FS as the host.
This is also usually more of an issue with EXT file systems (like we had here). INodes are set at FS creation and cannot be changed in EXT. Whereas XFS is dynamic (but has a limit that's based on percentage).
Does the inode only store the information when the file is in use? It doesn't. -
"When a file is created, it is assigned both a name and an inode number, which is an integer that is unique within the filesystem. Both the file names and their corresponding inode numbers are stored as entries in the directory that appears to the user to contain the files. That is, the directory associates file names with inodes.
Whenever a user or a program refers to a file by name, the operating system uses that name to look up the corresponding inode, which then enables the system to obtain the information it needs about the file to perform further operations. That is, a file name in a Unix-like operating system is merely an entry in a table with inode numbers, rather than being associated directly with a file (in contrast to other operating systems such as the Microsoft Windows systems). The inode numbers and their corresponding inodes are held in inode tables, which are stored in strategic locations in a filesystem, including near its beginning."
-
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
-
Thanks for the explanations. It helped me search better, and I eventually found this: https://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html
Shows how to check available inode space and how to fix it as well.
As for Scott's problem, I'm guessing something creating a lot of temp files that are cleared during a reboot.
-
@tim_g If it's the same file system wouldn't that keep the number of temporary files down to a minimum? Legitimately asking, not being snarky
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
Usually comes from orphaned INodes. Can happen if the process halts or server is shutdown incorrectly. This is what the lost+found directory is for.
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
It did not. Deleting millions of unneeded files did.
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
It's one of those weird things that on tests and in classrooms people talk about constantly as if it is an every day thing. But in the real world, with thousands of servers over decades of use, I've actually had this happen now... once. I am an idiot for not checking that straight away, but it is so insanely rare in the real world that it never occurred to me that it was a real possibility. Especially when the system had no reason to be making so many files.
-
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems like a major Linux turn-off if this happens for no reason.
It's not Linux related. It's filesystem related. And it is so insanely rare that it's not a thing to worry about in the real world. Windows has inodes too, they call them File ID.
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
-
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
