How to Stop XenServer from Mounting /var/log

scottalanmiller

dd only reads from the if device, it does not modify it. So no.

BRRABill

I'd be nervous about putting that into production till I figured out what caused the issue.

Oh yeah. Fortunately this was on my test machine.

I'm more nervous that moving the logs caused this. Is there any chance that XS needs those logs set up the way they have them?

Is anyone here actually moving their logs, or does everyone just write them to USB?

BRRABill

@scottalanmiller said

dd only reads from the if device, it does not modify it. So no.

Would the crazy slow speed perhaps indicate potential impending failure? I mean, even at USB2 that is slow. It was doing 3MBps.

travisdh1

@BRRABill said in How to Stop XenServer from Mounting /var/log:

Is it possible cancelling (^C) dd would have done anything to the source USB stick?

dd input is read only, I'd suspect the flash drive initially. While I run hypervisors on them, I don't trust them at all. Gotta keep those backups.

dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.

scottalanmiller

@BRRABill said in How to Stop XenServer from Mounting /var/log:

Is anyone here actually moving their logs, or does everyone just write them to USB?

Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.

BRRABill

@scottalanmiller said

Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.

I am pretty sure once I get back from my mini trip over the weekend that I am going to be reinstalling XS on the extra RAID array I have in my production server.

I know everyone at ML loves hypervisors on USB, but there just seems to be too much crashing of XS on USB, with no clear cut way (as of now) to prevent it.

Other than to keep a clone and pray.

BRRABill

@travisdh1 said

dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.

It was dd-ing the boot device, so that all had to be included...

BRRABill

After a little more poking around...

The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*

scottalanmiller

@BRRABill said in How to Stop XenServer from Mounting /var/log:

After a little more poking around...

The LV shows up in LVS, but is not listed under /dev/mapper nor is it listed under /dev/dm*

Sounds like the DM device has died perhaps beneath the LV.

BRRABill

@scottalanmiller said

Sounds like the DM device has died perhaps beneath the LV.

Since the USB stick showed all those weird characters, wouldn't it be more probable that the USB stick died, and not the hard drive that LV was on?

DustinB3403

@BRRABill That is what I would assume as well.

That the USB has something wrong with and needs to be replaced.

BRRABill

@DustinB3403 said

That the USB has something wrong with and needs to be replaced.

My production server is also booting off USB.

I am now trying to figure out if it is worth messing with it (aka, powering it down and cloning it, and hoping nothing goes wrong) before I go away for a few days. Or just leave it and hope for the best.

I'm the only guy here, so if it dies while I am gone, that's a problem.

BRRABill

@DustinB3403 said

That the USB has something wrong with and needs to be replaced.

Odd because it is brand new, and from Samsung.

It's almost like it went crazy for a bit, the data got corrupted, but then now it looks OK again, just doesn't work.

Ghost in the machine...

BRRABill

I took that USB drive out of the XS, stuck it into my Windows machine, reformatted it, and ran a CHKDSK on it, checking all sectors.

Nary an issue.

Maybe just a fluke and the drive is really OK? Or trash it?

I'm leaning towards trashing it.

BRRABill

Today I rebuilt the crashed XS7. Installed XS7 to a new USB. Once XS was set up, I stopped the mounting of /var/log and redirected it to a folder on the VM SR. Then I set up 2 test VMs on the XS7. Everything was working fine.

Then guess what? Same issue happened. I rebooted the XS, and it crashed in the exact same way. No network and the SR had been unmounted.

What the hell is going on? At this point I am ruling out hardware. Is it possible putting anything on the VM storage LV crashes XS? Is it possibly a bug?

What is this?????

travisdh1

@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)

none     /tmp     tmpfs     defaults     0 0

Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.

BRRABill

@travisdh1 said in How to Stop XenServer from Mounting /var/log:

@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none     /tmp     tmpfs     defaults     0 0
Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.

It's not going to one of the VMs.

It is going to the PV that the VMs are stored on.

When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.

It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).

I mean, could doing that really crash rthe server?

travisdh1

@BRRABill said in How to Stop XenServer from Mounting /var/log:

@travisdh1 said in How to Stop XenServer from Mounting /var/log:
@BRRABill Hrm. I don't know that having the host use one of it's own VMs as the target for /var/log will work real well, it's going to start writing to the log before the guest is up. Have you tried mounting it to memory only? (tmpfs)
none     /tmp     tmpfs     defaults     0 0
Obviously if you do that, you'd want to get the log files shipped to that VM you have setup anyway, just a roundabout way of doing it.
It's not going to one of the VMs.

It is going to the PV that the VMs are stored on.

When XS installed, it took the entire space I had for storafe for VM storage. So we created a directory there.

It's under /run/sr-mount/xxxxxx where xxxxx is the PV name (a long list of letters and numbers).

I mean, could doing that really crash rthe server?

Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.

BRRABill

@travisdh1 said i

Ah, I gotcha. Yeah, shouldn't be a problem. Bug report time I'd say.

I mean this is what I am doing. Is it possible any of these steps could be hosing something else?

dd the line in fstab that mounts /var/log
reboot the host
once the server comes back up it is logging to the "normal" spot
stop rsyslog
del /var/log
mkdir in /run/sr-mount/xxxxx called xenserverlogs
symlink /var/log to that folder from #6
restart rsyslog

After doing these steps, the logging is working perfectly to the folder in #6.

Sometime (I think it might be around midnight) it seems to crash on reboot and not come back up. The PV lists in pvs and lvs but is not in /dev/mapper or listed under /dev/dm* I also lose networking

See anything odd?

travisdh1

@BRRABill The only thing that springs to mind is making sure the volume group the logical volume sits on is ok. It should be with the logical volume being ok. The other thing is making sure /var/log is mounting via fstab correctly (df). Kinda grasping at straws at the moment.