Fedora 26 No Space Left on Device with Plenty of Space Available
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
This is the same limitation any FS has. It's just metadata. If you have a ton of files you will run out of available inodes. Containers are an area where this can happen easily since all of the containers can share the same FS as the host.
This is also usually more of an issue with EXT file systems (like we had here). INodes are set at FS creation and cannot be changed in EXT. Whereas XFS is dynamic (but has a limit that's based on percentage).
-
"An inode is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. A data structure is a way of storing data so that it can be used efficiently."
-
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
This is the same limitation any FS has. It's just metadata. If you have a ton of files you will run out of available inodes. Containers are an area where this can happen easily since all of the containers can share the same FS as the host.
This is also usually more of an issue with EXT file systems (like we had here). INodes are set at FS creation and cannot be changed in EXT. Whereas XFS is dynamic (but has a limit that's based on percentage).
Does the inode only store the information when the file is in use? It doesn't. -
"When a file is created, it is assigned both a name and an inode number, which is an integer that is unique within the filesystem. Both the file names and their corresponding inode numbers are stored as entries in the directory that appears to the user to contain the files. That is, the directory associates file names with inodes.
Whenever a user or a program refers to a file by name, the operating system uses that name to look up the corresponding inode, which then enables the system to obtain the information it needs about the file to perform further operations. That is, a file name in a Unix-like operating system is merely an entry in a table with inode numbers, rather than being associated directly with a file (in contrast to other operating systems such as the Microsoft Windows systems). The inode numbers and their corresponding inodes are held in inode tables, which are stored in strategic locations in a filesystem, including near its beginning."
-
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
-
Thanks for the explanations. It helped me search better, and I eventually found this: https://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html
Shows how to check available inode space and how to fix it as well.
As for Scott's problem, I'm guessing something creating a lot of temp files that are cleared during a reboot.
-
@tim_g If it's the same file system wouldn't that keep the number of temporary files down to a minimum? Legitimately asking, not being snarky
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
Usually comes from orphaned INodes. Can happen if the process halts or server is shutdown incorrectly. This is what the lost+found directory is for.
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
It did not. Deleting millions of unneeded files did.
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
It's one of those weird things that on tests and in classrooms people talk about constantly as if it is an every day thing. But in the real world, with thousands of servers over decades of use, I've actually had this happen now... once. I am an idiot for not checking that straight away, but it is so insanely rare in the real world that it never occurred to me that it was a real possibility. Especially when the system had no reason to be making so many files.
-
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems like a major Linux turn-off if this happens for no reason.
It's not Linux related. It's filesystem related. And it is so insanely rare that it's not a thing to worry about in the real world. Windows has inodes too, they call them File ID.
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
-
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
-
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
-
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
Empty I think. Tons were just folders.
-
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? More files = more inodes being used
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant. Pay attention to what is said.
-
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant.
It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.
Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.
-
Ya I've never run into it in the real world only on trick questions
