How to Stop XenServer from Mounting /var/log
-
@scottalanmiller said
Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.
I am pretty sure once I get back from my mini trip over the weekend that I am going to be reinstalling XS on the extra RAID array I have in my production server.
I know everyone at ML loves hypervisors on USB, but there just seems to be too much crashing of XS on USB, with no clear cut way (as of now) to prevent it.
Other than to keep a clone and pray.
-
@travisdh1 said
dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.
It was dd-ing the boot device, so that all had to be included...
-
After a little more poking around...
The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
After a little more poking around...
The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*
Sounds like the DM device has died perhaps beneath the LV.
-
@scottalanmiller said
Sounds like the DM device has died perhaps beneath the LV.
Since the USB stick showed all those weird characters, wouldn't it be more probable that the USB stick died, and not the hard drive that LV was on?
-
@BRRABill That is what I would assume as well.
That the USB has something wrong with and needs to be replaced.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
My production server is also booting off USB.
I am now trying to figure out if it is worth messing with it (aka, powering it down and cloning it, and hoping nothing goes wrong) before I go away for a few days. Or just leave it and hope for the best.
I'm the only guy here, so if it dies while I am gone, that's a problem.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
Odd because it is brand new, and from Samsung.
It's almost like it went crazy for a bit, the data got corrupted, but then now it looks OK again, just doesn't work.
Ghost in the machine...
-
I took that USB drive out of the XS, stuck it into my Windows machine, reformatted it, and ran a CHKDSK on it, checking all sectors.
Nary an issue.
Maybe just a fluke and the drive is really OK? Or trash it?
I'm leaning towards trashing it.
-
Today I rebuilt the crashed XS7. Installed XS7 to a new USB. Once XS was set up, I stopped the mounting of /var/log and redirected it to a folder on the VM SR. Then I set up 2 test VMs on the XS7. Everything was working fine.
Then guess what? Same issue happened. I rebooted the XS, and it crashed in the exact same way. No network and the SR had been unmounted.
What the hell is going on? At this point I am ruling out hardware. Is it possible putting anything on the VM storage LV crashes XS? Is it possibly a bug?
What is this?????
-
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.
It is going to the PV that the VMs are stored on.
When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.
It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).
I mean, could doing that really crash rthe server?
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.
It is going to the PV that the VMs are stored on.
When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.
It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).
I mean, could doing that really crash rthe server?
Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.
-
@travisdh1 said i
Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.
I mean this is what I am doing. Is it possible any of these steps could be hosing something else?
- dd the line in fstab that mounts /var/log
- reboot the host
- once the server comes back up it is logging to the "normal" spot
- stop rsyslog
- del /var/log
- mkdir in /run/sr-mount/xxxxx called xenserverlogs
- symlink /var/log to that folder from #6
- restart rsyslog
After doing these steps, the logging is working perfectly to the folder in #6.
Sometime (I think it might be around midnight) it seems to crash on reboot and not come back up. The PV lists in pvs and lvs but is not in /dev/mapper or listed under /dev/dm* I also lose networking
See anything odd?
-
@BRRABill The only thing that springs to mind is making sure the volume group the logical volume sits on is ok. It should be with the logical volume being ok. The other thing is making sure /var/log is mounting via fstab correctly (df). Kinda grasping at straws at the moment.
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill The only thing that springs to mind is making sure the volume group the logical volume sits on is ok. It should be with the logical volume being ok. The other thing is making sure /var/log is mounting via fstab correctly (df). Kinda grasping at straws at the moment.
The VG/LV should be OK. They are brand new and work fine until the change.
I do not mount /var/log ... just symlink to it. Is that incorrect? It is originally in fstab because they mount it to a partition on the boot device.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
I do not mount /var/log ... just symlink to it. Is that incorrect? It is originally in fstab because they mount it to a partition on the boot device.
Your goal is to not mount /var/log as its own filesystem, that is correct.
-
@scottalanmiller said
Your goal is to not mount /var/log as its own filesystem, that is correct.
@scottalanmiller What is your take on this new wrinkle?
-
@BRRABill Did you create a separate logical volume for /var/log, or is both that and the storage repository on the same lv?
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Did you create a separate logical volume for /var/log, or is both that and the storage repository on the same lv?
No. XS used 100% of the space I had for its own LV.
So we thought putting a directory with the VHD files would be ok.
So /run/sr-mount/xxxxxxxxx
has
vm1.vhd
vm2.vhd
vm3.vhd
lost+found
xenserverlogs (the directory i created)Is it possible putting a directory there would cause this big an issue?
