How to Stop XenServer from Mounting /var/log
-
@scottalanmiller said in How to Stop XenServer from Mounting /var/log:
Well that sucks.
The good news is that this is Xen. Reinstall, import the storage repository, and import the configs, shouldn't take more than an hour.
I'd be nervous about putting that into production till I figured out what caused the issue.
-
@travisdh1 said
Something is wrong with that first lsblk output. System has crashed/been corrupted by something.
It came back "normal" after a reboot, but obviously a lot of functionality was hosed.
I tried running a dd on this machine the other day, and it was taking forever.
After about 90 minutes I stopped it, and it had only done 15GB thus far. (Of a 64 GB stick.)
Possible maybe the USB stick was going bad, and finally gave up the ghost?
Is it possible cancelling (^C) dd would have done anything to the source USB stick?
-
dd only reads from the if device, it does not modify it. So no.
-
@travisdh1 said
I'd be nervous about putting that into production till I figured out what caused the issue.
Oh yeah. Fortunately this was on my test machine.
I'm more nervous that moving the logs caused this. Is there any chance that XS needs those logs set up the way they have them?
Is anyone here actually moving their logs, or does everyone just write them to USB?
-
@scottalanmiller said
dd only reads from the if device, it does not modify it. So no.
Would the crazy slow speed perhaps indicate potential impending failure? I mean, even at USB2 that is slow. It was doing 3MBps.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is it possible cancelling (^C) dd would have done anything to the source USB stick?
dd input is read only, I'd suspect the flash drive initially. While I run hypervisors on them, I don't trust them at all. Gotta keep those backups.
dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is anyone here actually moving their logs, or does everyone just write them to USB?
Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.
-
@scottalanmiller said
Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.
I am pretty sure once I get back from my mini trip over the weekend that I am going to be reinstalling XS on the extra RAID array I have in my production server.
I know everyone at ML loves hypervisors on USB, but there just seems to be too much crashing of XS on USB, with no clear cut way (as of now) to prevent it.
Other than to keep a clone and pray.
-
@travisdh1 said
dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.
It was dd-ing the boot device, so that all had to be included...
-
After a little more poking around...
The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
After a little more poking around...
The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*
Sounds like the DM device has died perhaps beneath the LV.
-
@scottalanmiller said
Sounds like the DM device has died perhaps beneath the LV.
Since the USB stick showed all those weird characters, wouldn't it be more probable that the USB stick died, and not the hard drive that LV was on?
-
@BRRABill That is what I would assume as well.
That the USB has something wrong with and needs to be replaced.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
My production server is also booting off USB.
I am now trying to figure out if it is worth messing with it (aka, powering it down and cloning it, and hoping nothing goes wrong) before I go away for a few days. Or just leave it and hope for the best.
I'm the only guy here, so if it dies while I am gone, that's a problem.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
Odd because it is brand new, and from Samsung.
It's almost like it went crazy for a bit, the data got corrupted, but then now it looks OK again, just doesn't work.
Ghost in the machine...
-
I took that USB drive out of the XS, stuck it into my Windows machine, reformatted it, and ran a CHKDSK on it, checking all sectors.
Nary an issue.
Maybe just a fluke and the drive is really OK? Or trash it?
I'm leaning towards trashing it.
-
Today I rebuilt the crashed XS7. Installed XS7 to a new USB. Once XS was set up, I stopped the mounting of /var/log and redirected it to a folder on the VM SR. Then I set up 2 test VMs on the XS7. Everything was working fine.
Then guess what? Same issue happened. I rebooted the XS, and it crashed in the exact same way. No network and the SR had been unmounted.
What the hell is going on? At this point I am ruling out hardware. Is it possible putting anything on the VM storage LV crashes XS? Is it possibly a bug?
What is this?????
-
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.
It is going to the PV that the VMs are stored on.
When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.
It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).
I mean, could doing that really crash rthe server?
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.
It is going to the PV that the VMs are stored on.
When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.
It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).
I mean, could doing that really crash rthe server?
Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.
