Stopping XenServer From Writing To A USB Boot Drive

BRRABill

There has been discussion on various threads about this topic, so I thought it would be good to give it its own area.

Basically, we have been directing a lot of people to install XS, and install it on USB. However, XS does a good deal of writing to the boot disk, and that cannot happen for the safety of the USB boot device.

I think it would be prudent to figure out exactly WHAT it writes, and how to stop it.

QUESTION 1:
How do we determine what is being written? Is there a way to track what XS is writing?

I have found these two articles, both of which talk about some of the files it does write. But how do we know these are the only files being written?

http://xenserver.org/discuss-virtualization/virtualization-blog/entry/xenserver-root-disk-maintenance.html

http://xenserver.org/discuss-virtualization/virtualization-blog/entry/log-rotation-and-syslog-forwarding.html

BRRABill

With remote syslog turned ON, these are still some of the files that appear to be writing to the disk.

This is from /var/log from today.

-rw------- 1 root root 552045 Jul 8 11:54 daemon.log
-rw------- 1 root root 36569171 Jul 8 11:54 xensource.log
-rw------- 1 root root 6993911 Jul 8 11:54 xenstored-access.log
-rw------- 1 root root 3314999 Jul 8 11:53 secure
-rw------- 1 root root 1651148 Jul 8 11:53 audit.log
-rw-r--r-- 1 root root 38273316 Jul 8 11:53 lastlog
-rw------- 1 root root 1092509 Jul 8 11:53 SMlog
-rw-rw-r-- 1 root utmp 24960 Jul 8 11:53 wtmp
-rw------- 1 root root 39140 Jul 8 11:52 xcp-rrdd-plugins.log
-rw------- 1 root root 7910 Jul 8 11:50 cron
-rw------- 1 root root 25237 Jul 8 11:49 user.log
-rw------- 1 root root 2197 Jul 8 11:27 kern.log
drwxr-xr-x 2 root root 4096 Jul 8 11:27 blktap
-rw-r--r-- 1 root root 0 Jul 8 04:02 boot.log
-rw------- 1 root root 0 Jul 8 04:02 crit.log
drwxr-xr-x 2 root root 4096 Jul 8 04:02 xen
-rw------- 1 root root 0 Jul 8 04:02 maillog
-rw------- 1 root root 0 Jul 8 04:02 messages
-rw------- 1 root root 0 Jul 8 04:02 spooler
-rw------- 1 root root 25430 Jul 8 04:02 cron.1
-rw------- 1 root root 1397607 Jul 8 04:02 daemon.log.1
-rw-r--r-- 1 root root 0 Jul 8 04:02 interface-rename.log
-rw------- 1 root root 21282 Jul 8 04:02 user.log.1
-rw------- 1 root root 17914819 Jul 8 04:01 xenstored-access.log.1
-rw------- 1 root root 2878021 Jul 8 04:01 audit.log.1
-rw------- 1 root root 6119984 Jul 8 04:01 secure.1
-rw------- 1 root root 1438123 Jul 8 04:01 SMlog.1
-rw------- 1 root root 96116 Jul 8 03:57 xcp-rrdd-plugins.log.1
-rw------- 1 root root 54918053 Jul 8 00:39 xensource.log.1
drwxr-xr-x 2 root root 4096 Jul 8 00:00 sa

scottalanmiller

You have to not just turn on remote, but stop it from writing locally.

BRRABill

@scottalanmiller said

You have to not just turn on remote, but stop it from writing locally.

Right. I am going to try the steps they suggested in that article. Perhaps it will work this time.

Well, step one, already an issue...

This stpe
"Finally, select "OK" and the stand-alone XenServer (or pool) will update its Syslog configuration, or more specifically, /var/lib/syslog.conf. "

/var/lib/syslog.conf is no longer there

BRRABill

I think that file has been replaced with /etc/rsyslog.d/xenserver.conf

Which reads...

# Suppress duplicate messages and report "Last line repeated n times"
$RepeatedMsgReduction on

# Don't rate-limit messages - this isn't the right way to go about 
# reducing log size!
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0

# Ensure critical and higher level errors are logged synchronously.
*.crit;mail.none;authpriv.none;cron.none		/var/log/crit.log

# Log by facility.
kern.*							-/var/log/kern.log
daemon.*						-/var/log/daemon.log
user.*							-/var/log/user.log

# The authpriv file has restricted access.
authpriv.*						-/var/log/secure

# Log all the mail messages in one place.
mail.*							-/var/log/maillog

# Log cron stuff
cron.*							-/var/log/cron

# Save boot messages also to boot.log
local7.*						/var/log/boot.log

# Xapi rbac audit log echoes to syslog local6
local6.*						-/var/log/audit.log

# Xapi, xenopsd echo to syslog local5
local5.*						-/var/log/xensource.log

# V6d echo to syslog local4
local4.*						-/var/log/v6d.log

# xenstore access to syslog local3
local3.info						-/var/log/xenstored-access.log

# Storage Manager to syslog local2
local2.*						-/var/log/SMlog

# xcp-rrdd-plugins (info and above) to local0
local0.info						-/var/log/xcp-rrdd-plugins.log

# ignore default rules

*.* @10.0.4.26
*.* ~

BRRABill

If you turn off remote logging, this is what you get at the bottom...

# ignore default rules

*.* ~

Danp

Found this: http://discussions.citrix.com/topic/378269-xenserver-7-varlogmessages/

BRRABill

Another thing to ponder...

With XS7, there is a separate 4GB log partition.

Is there a way to forward that elsewhere?

BRRABill

@Danp said

Found this: http://discussions.citrix.com/topic/378269-xenserver-7-varlogmessages/

Yeah, I found that, too.

But I am wondering...is that the file that tells all the possible logs, and where they write to?

And can we tell if anything else is writing?

DustinB3403

Does the link here not already address this issue?

"I mentioned before that Syslog can forward messages to other hosts. Furthermore, it can forward Syslog messages to other hosts without writing a copy of the log to local disk. What this means is that a single XenServer or a pool of XenServers can send their log data to a "Syslog Aggregator". "

DustinB3403

Followed by

Debian Syslog Server and Centralized RSYSLOG Server monitoring

DustinB3403

Or even this last guide on the very same page, All-In-One-Guide.

BRRABill

@DustinB3403 said

Does the link here not already address this issue?

"I mentioned before that Syslog can forward messages to other hosts. Furthermore, it can forward Syslog messages to other hosts without writing a copy of the log to local disk. What this means is that a single XenServer or a pool of XenServers can send their log data to a "Syslog Aggregator". "

No, because further in the article, you will see...

"Certain logs will still continue to record Syslog on the host, so it may be desirable to edit /var/lib/syslog.conf and add comments to lines where a "-/var/log/some_filename" is specified as lines with "@x.x.x.x" dictate to forward to the Syslog aggregator."

And in XS7, that file has been totally replaced with the one I posted, which has different syntax.

DustinB3403

@BRRABill Wouldn't the same essentially apply to the log file you have there?

Your file in /etc/rsyslog.d/xenserver.conf:

# Save boot messages also to boot.log
local7.*

Would be changed to:

# Save boot messages also to boot.log
local7.*      @your-syslog-ip
#local7.*

DustinB3403

And in XS6.5 the syslog location is stored at

/etc/syslog.conf

Which still has the same flags.

BRRABill

@DustinB3403

Well, in 6.5, the syntax was as follows...

# Save boot messages also to boot.log
local7.*             @10.0.0.1
# local7.*         /var/log/boot.log

In this new version, it just puts the IP at the bottom.

@scottalanmiller mentioned, as I think you did as well, that I should just redirect /var/log somewhere else.

Being a little fresh in Linux partitioning, I most post a thread for help with that.

DustinB3403

@BRRABill Why are you not commenting out the records as needed and adding them under each field in order?

You can edit the file (on XS7) with:

sudo nano /etc/rsyslog.d/xenserver.conf

Or on XS 6.5

sudo nano /etc/syslog.conf

DustinB3403

I'll follow the setup on my lab tonight that is posted in the first link, and make all of the changes on XS6.5.

Then I'll see how it performs. from there I'll tear down my installation and repeat for XS7.

Danp

Here's one to watch from the Citrix forums: http://discussions.citrix.com/topic/379454-booting-xenserver-off-usb-safe/

P.S. Which one of you guys wrote this?

BRRABill

@Danp said in Stopping XenServer From Writing To A USB Boot Drive:

Here's one to watch from the Citrix forums: http://discussions.citrix.com/topic/379454-booting-xenserver-off-usb-safe/

P.S. Which one of you guys wrote this?

That was me.

Those guys know the nuts and bolts of XS, though they aren't real responsive to threads all the time.

On ML that would have have 500 posts already and been forked 6 times.