How to Stop XenServer from Mounting /var/log



  • @scottalanmiller said

    Correct. Since /var/log is a mount point for you, though, that is not as simple as deleting or renaming. You have to stop it mounting first. Then you can rmdir /var/log. Then you can symlink.

    You can fork this if you want.

    How exactly does one do that, stop it from mounting?



  • You have to find the configuration file that is causing it to mount. You already posted the contents of /etc/fstab in another thread and it is not listed in there. So we need to know where XS is getting the information from.



  • Just as a reminder, this is fstab

    
    LABEL=root-qidkls    /         ext3     defaults   1  1
    LABEL=swap-qidkls          swap      swap   defaults   0  0
    LABEL=logs-qidkls    /var/log         ext3     defaults   0  2
    /opt/xensource/packages/iso/XenCenter.iso   /var/xen/xc-install   iso9660   loop,ro   0  0


  • Oh, I thought that you posted it and it wasn't there. Just remove the third line and you are done.



  • @scottalanmiller

    I posted it before I learned about posting with the "```" so it was probably hard to read.

    So that is it, eh?



  • Can I unmount it while the server is running?

    AKA ... what should be the process for the whole job?

    1. Stop rsyslog
    2. Copy files from /var/log to the new location
    3. Unmount /var/log
    4. Delete /var/log folder
    5. Symlink /var/log to other storage
    6. Restart rsyslog

    Does that sound about right?



  • Oooh, I think I am getting somewhere!

    lrwxrwxrwx  1 root root   64 Jul 11 13:01 log -> /run/sr-mount/cbe8175f-43b3-3a64-cdac-3c1475aa2d01/xenserverlogs


  • The one thing to note is that in my system, XS7 used my entire array for VM storage. I boot off USB and only have one presented drive (Raid 5 SSD) in the server.

    After discussing with @scottalanmiller it was determined it was probably OK to create a directory alongside the .VHD files in SR-Mount and store the logs there.

    So far, so good. Nothing has imploded.



  • Other options would be to use LVM to create two logical volumes on that RAID device or use the RAID device's LVM system to do the same.



  • Can you think of any reason why XS would object to creating a folder in sr-mount?



  • @BRRABill said in How to Stop XenServer from Mounting /var/log:

    Can you think of any reason why XS would object to creating a folder in sr-mount?

    Yes, if it is scanning that directory for folders believing that the existence of a folder in that location implies something, then yes.



  • @scottalanmiller said

    Yes, if it is scanning that directory for folders believing that the existence of a folder in that location implies something, then yes.

    Would it be safer, do you think, to go with your other method of shrinking the volume and adding another volume for the logging?

    I mean, I'd like to figure this out not just for me, but for all the people in the future who ML recommends XS on USB to.



  • You've already done it, so I would just check and see how it is working.



  • @scottalanmiller said

    You've already done it, so I would just check and see how it is working.

    Seems fine.

    I just like to, you know, check.

    I figure someone here would know if it's an OK idea or say
    NO FOR THE LOVE OF ALL THAT IS GOOD DO NOT DO THAT!!!!

    We have other XS users ... what are they doing?



  • @BRRABill said in How to Stop XenServer from Mounting /var/log:

    @scottalanmiller said

    You've already done it, so I would just check and see how it is working.

    Seems fine.

    I just like to, you know, check.

    I figure someone here would know if it's an OK idea or say
    NO FOR THE LOVE OF ALL THAT IS GOOD DO NOT DO THAT!!!!

    We have other XS users ... what are they doing?

    Just letting it log to the usb drive for now. Yes, I've got a copy of the current config that gets taken every day should the usb stick croak.



  • @travisdh1 said

    Just letting it log to the usb drive for now. Yes, I've got a copy of the current config that gets taken every day should the usb stick croak.

    Out of curiosity, how are you copying the current config?

    Cloning the USB? Or making backups through XC/CLI?



  • So, things did not go so well overnight. (Note this is a test XS not my production box.)

    Couple things happened. Not sure if it is a coincidence, or something more sinister.

    I could not connect to this XS from either XC or Putty. You could connect through the local console.

    The first thing I did, of course, was the check to see if the logging was still working. /var/log was sitting there with big fat red text. Hmmm. So I figured I would check the LV on /run/sr-mount to see when the logs stopped. Except that was no longer mounted there. Uh oh.

    0_1468333526911_IMG_6791.JPG

    I also noticed there seemed to be something wrong with the reporting of the devices.

    0_1468333538528_IMG_6793.JPG

    So, I decided to reboot the server, and hope it came back up. After taking a looooooooooong time to shut down, it did indeed come back up, but the same problem existed. The LV was not attached, and it had no network connectivity.

    The LV still appears to be there, but I wonder if something else went wacky with this XS install. I mean, could putting a subfolder in the VM storage directory really cause this much havoc?

    0_1468333809010_IMG_6794.JPG



  • I'm leaning towards something wacky, because even if the SR was detached for some reason, why would the networking not work as well?



  • Looks more like a crash than anything. So it is back up and working after the reboot?



  • @scottalanmiller said

    Looks more like a crash than anything. So it is back up and working after the reboot?

    The SR is not mounted, and the networking is not working.

    So no, not working. Which is why I wonder if the entire thing just went south for some reason.



  • @BRRABill said in How to Stop XenServer from Mounting /var/log:

    @scottalanmiller said

    Looks more like a crash than anything. So it is back up and working after the reboot?

    The SR is not mounted, and the networking is not working.

    So no, not working. Which is why I wonder if the entire thing just went south for some reason.

    Something is wrong with that first lsblk output. System has crashed/been corrupted by something.



  • Well that sucks.



  • @scottalanmiller said in How to Stop XenServer from Mounting /var/log:

    Well that sucks.

    The good news is that this is Xen. Reinstall, import the storage repository, and import the configs, shouldn't take more than an hour.

    I'd be nervous about putting that into production till I figured out what caused the issue.



  • @travisdh1 said

    Something is wrong with that first lsblk output. System has crashed/been corrupted by something.

    It came back "normal" after a reboot, but obviously a lot of functionality was hosed.

    I tried running a dd on this machine the other day, and it was taking forever.

    After about 90 minutes I stopped it, and it had only done 15GB thus far. (Of a 64 GB stick.)

    Possible maybe the USB stick was going bad, and finally gave up the ghost?

    Is it possible cancelling (^C) dd would have done anything to the source USB stick?



  • dd only reads from the if device, it does not modify it. So no.



  • @travisdh1 said

    I'd be nervous about putting that into production till I figured out what caused the issue.

    Oh yeah. Fortunately this was on my test machine.

    I'm more nervous that moving the logs caused this. Is there any chance that XS needs those logs set up the way they have them?

    Is anyone here actually moving their logs, or does everyone just write them to USB?



  • @scottalanmiller said

    dd only reads from the if device, it does not modify it. So no.

    Would the crazy slow speed perhaps indicate potential impending failure? I mean, even at USB2 that is slow. It was doing 3MBps.



  • @BRRABill said in How to Stop XenServer from Mounting /var/log:

    Is it possible cancelling (^C) dd would have done anything to the source USB stick?

    dd input is read only, I'd suspect the flash drive initially. While I run hypervisors on them, I don't trust them at all. Gotta keep those backups.

    dd would take a long, long time if you have it copying something like /proc, /sys, or /dev.



  • @BRRABill said in How to Stop XenServer from Mounting /var/log:

    Is anyone here actually moving their logs, or does everyone just write them to USB?

    Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.



  • @scottalanmiller said

    Ours had to be built to spinning SATA sadly due to USB port issues. They are not back up yet after the move (much lower priority than the big Scale cluster so have not looked at them yet and won't until I am back in Rochester) but once they are back up, log shipping will be going to ELK and we will attempt to stop local writing even so.

    I am pretty sure once I get back from my mini trip over the weekend that I am going to be reinstalling XS on the extra RAID array I have in my production server.

    I know everyone at ML loves hypervisors on USB, but there just seems to be too much crashing of XS on USB, with no clear cut way (as of now) to prevent it.

    Other than to keep a clone and pray. 🙂


Log in to reply