How to Stop XenServer from Mounting /var/log
-
So, things did not go so well overnight. (Note this is a test XS not my production box.)
Couple things happened. Not sure if it is a coincidence, or something more sinister.
I could not connect to this XS from either XC or Putty. You could connect through the local console.
The first thing I did, of course, was the check to see if the logging was still working. /var/log was sitting there with big fat red text. Hmmm. So I figured I would check the LV on /run/sr-mount to see when the logs stopped. Except that was no longer mounted there. Uh oh.
I also noticed there seemed to be something wrong with the reporting of the devices.
So, I decided to reboot the server, and hope it came back up. After taking a looooooooooong time to shut down, it did indeed come back up, but the same problem existed. The LV was not attached, and it had no network connectivity.
The LV still appears to be there, but I wonder if something else went wacky with this XS install. I mean, could putting a subfolder in the VM storage directory really cause this much havoc?
-
I'm leaning towards something wacky, because even if the SR was detached for some reason, why would the networking not work as well?
-
Looks more like a crash than anything. So it is back up and working after the reboot?
-
@scottalanmiller said
Looks more like a crash than anything. So it is back up and working after the reboot?
The SR is not mounted, and the networking is not working.
So no, not working. Which is why I wonder if the entire thing just went south for some reason.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
@scottalanmiller said
Looks more like a crash than anything. So it is back up and working after the reboot?
The SR is not mounted, and the networking is not working.
So no, not working. Which is why I wonder if the entire thing just went south for some reason.
Something is wrong with that first lsblk output. System has crashed/been corrupted by something.
-
Well that sucks.
-
@scottalanmiller said in How to Stop XenServer from Mounting /var/log:
Well that sucks.
The good news is that this is Xen. Reinstall, import the storage repository, and import the configs, shouldn't take more than an hour.
I'd be nervous about putting that into production till I figured out what caused the issue.
-
@travisdh1 said
Something is wrong with that first lsblk output. System has crashed/been corrupted by something.
It came back "normal" after a reboot, but obviously a lot of functionality was hosed.
I tried running a dd on this machine the other day, and it was taking forever.
After about 90 minutes I stopped it, and it had only done 15GB thus far. (Of a 64 GB stick.)
Possible maybe the USB stick was going bad, and finally gave up the ghost?
Is it possible cancelling (^C) dd would have done anything to the source USB stick?
-
dd only reads from the if device, it does not modify it. So no.
-
@travisdh1 said
I'd be nervous about putting that into production till I figured out what caused the issue.
Oh yeah. Fortunately this was on my test machine.
I'm more nervous that moving the logs caused this. Is there any chance that XS needs those logs set up the way they have them?
Is anyone here actually moving their logs, or does everyone just write them to USB?
-
@scottalanmiller said
dd only reads from the if device, it does not modify it. So no.
Would the crazy slow speed perhaps indicate potential impending failure? I mean, even at USB2 that is slow. It was doing 3MBps.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is it possible cancelling (^C) dd would have done anything to the source USB stick?
dd input is read only, I'd suspect the flash drive initially. While I run hypervisors on them, I don't trust them at all. Gotta keep those backups.
dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is anyone here actually moving their logs, or does everyone just write them to USB?
Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.
-
@scottalanmiller said
Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.
I am pretty sure once I get back from my mini trip over the weekend that I am going to be reinstalling XS on the extra RAID array I have in my production server.
I know everyone at ML loves hypervisors on USB, but there just seems to be too much crashing of XS on USB, with no clear cut way (as of now) to prevent it.
Other than to keep a clone and pray.
-
@travisdh1 said
dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.
It was dd-ing the boot device, so that all had to be included...
-
After a little more poking around...
The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
After a little more poking around...
The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*
Sounds like the DM device has died perhaps beneath the LV.
-
@scottalanmiller said
Sounds like the DM device has died perhaps beneath the LV.
Since the USB stick showed all those weird characters, wouldn't it be more probable that the USB stick died, and not the hard drive that LV was on?
-
@BRRABill That is what I would assume as well.
That the USB has something wrong with and needs to be replaced.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
My production server is also booting off USB.
I am now trying to figure out if it is worth messing with it (aka, powering it down and cloning it, and hoping nothing goes wrong) before I go away for a few days. Or just leave it and hope for the best.
I'm the only guy here, so if it dies while I am gone, that's a problem.
