How to Stop XenServer from Mounting /var/log
-
@scottalanmiller said
Sounds like the DM device has died perhaps beneath the LV.
Since the USB stick showed all those weird characters, wouldn't it be more probable that the USB stick died, and not the hard drive that LV was on?
-
@BRRABill That is what I would assume as well.
That the USB has something wrong with and needs to be replaced.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
My production server is also booting off USB.
I am now trying to figure out if it is worth messing with it (aka, powering it down and cloning it, and hoping nothing goes wrong) before I go away for a few days. Or just leave it and hope for the best.
I'm the only guy here, so if it dies while I am gone, that's a problem.
-
@DustinB3403 said
That the USB has something wrong with and needs to be replaced.
Odd because it is brand new, and from Samsung.
It's almost like it went crazy for a bit, the data got corrupted, but then now it looks OK again, just doesn't work.
Ghost in the machine...
-
I took that USB drive out of the XS, stuck it into my Windows machine, reformatted it, and ran a CHKDSK on it, checking all sectors.
Nary an issue.
Maybe just a fluke and the drive is really OK? Or trash it?
I'm leaning towards trashing it.
-
Today I rebuilt the crashed XS7. Installed XS7 to a new USB. Once XS was set up, I stopped the mounting of /var/log and redirected it to a folder on the VM SR. Then I set up 2 test VMs on the XS7. Everything was working fine.
Then guess what? Same issue happened. I rebooted the XS, and it crashed in the exact same way. No network and the SR had been unmounted.
What the hell is going on? At this point I am ruling out hardware. Is it possible putting anything on the VM storage LV crashes XS? Is it possibly a bug?
What is this?????
-
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.
It is going to the PV that the VMs are stored on.
When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.
It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).
I mean, could doing that really crash rthe server?
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none /tmp tmpfs defaults 0 0Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.
It is going to the PV that the VMs are stored on.
When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.
It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).
I mean, could doing that really crash rthe server?
Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.
-
@travisdh1 said i
Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.
I mean this is what I am doing. Is it possible any of these steps could be hosing something else?
- dd the line in fstab that mounts /var/log
- reboot the host
- once the server comes back up it is logging to the "normal" spot
- stop rsyslog
- del /var/log
- mkdir in /run/sr-mount/xxxxx called xenserverlogs
- symlink /var/log to that folder from #6
- restart rsyslog
After doing these steps, the logging is working perfectly to the folder in #6.
Sometime (I think it might be around midnight) it seems to crash on reboot and not come back up. The PV lists in pvs and lvs but is not in /dev/mapper or listed under /dev/dm* I also lose networking
See anything odd?
-
@BRRABill The only thing that springs to mind is making sure the volume group the logical volume sits on is ok. It should be with the logical volume being ok. The other thing is making sure /var/log is mounting via fstab correctly (df). Kinda grasping at straws at the moment.
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill The only thing that springs to mind is making sure the volume group the logical volume sits on is ok. It should be with the logical volume being ok. The other thing is making sure /var/log is mounting via fstab correctly (df). Kinda grasping at straws at the moment.
The VG/LV should be OK. They are brand new and work fine until the change.
I do not mount /var/log ... just symlink to it. Is that incorrect? It is originally in fstab because they mount it to a partition on the boot device.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
I do not mount /var/log ... just symlink to it. Is that incorrect? It is originally in fstab because they mount it to a partition on the boot device.
Your goal is to not mount /var/log as its own filesystem, that is correct.
-
@scottalanmiller said
Your goal is to not mount /var/log as its own filesystem, that is correct.
@scottalanmiller What is your take on this new wrinkle?
-
@BRRABill Did you create a separate logical volume for /var/log, or is both that and the storage repository on the same lv?
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Did you create a separate logical volume for /var/log, or is both that and the storage repository on the same lv?
No. XS used 100% of the space I had for its own LV.
So we thought putting a directory with the VHD files would be ok.
So /run/sr-mount/xxxxxxxxx
has
vm1.vhd
vm2.vhd
vm3.vhd
lost+found
xenserverlogs (the directory i created)Is it possible putting a directory there would cause this big an issue?
-
possible, yes. I'd say unlikely, but if it causes the software to freak out because it doesn't expect it there, yes.
-
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is it possible putting a directory there would cause this big an issue?
In no way should it create issues like this... in the real world however, well.
Do you have free space available that you could shrink the LV and create another LV just for the log files?
-
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is it possible putting a directory there would cause this big an issue?
In no way should it create issues like this... in the real world however, well.
Do you have free space available that you could shrink the LV and create another LV just for the log files?
yes, shrinking is a technical possibility.
-
@scottalanmiller said in How to Stop XenServer from Mounting /var/log:
@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill said in How to Stop XenServer from Mounting /var/log:
Is it possible putting a directory there would cause this big an issue?
In no way should it create issues like this... in the real world however, well.
Do you have free space available that you could shrink the LV and create another LV just for the log files?
yes, shrinking is a technical possibility.
The other question I'd think about is if it's an LV or VG. ProxMox (good riddance, it's gone) actually uses a volume group when you mount local storage LVM containers.
