Final Call ... XenServer Boot Media
-
@scottalanmiller said
I thought that the commenter was on 6.5. When I read it, I read it as the author was on 6.2 and the commenter was on 6.5.
Here is the post...
COMMENT POSTER:
Following your article, I updated /var/lib/syslog.conf and commented out the local storage lines for /var/log/messages and /var/log/xensource.log since we are logging to a remote ELK (Elasticsearch, Logstash, Kibana) stack.However, when I restart syslog, /var/lib/syslog.conf get rewritten back to the original configuration, changing my commented lines back to active.
This is on XenServer 6.2.
Any idea why this is happening and how to make my changes stick?
AUTHOR RESPONSE:
Thanks for the comment and pardon the delay! I needed to check on some things between XenServer 6.2 and 6.5 to answer your question.-
In XenCenter, did you enable "Log Forwarding"?
-
If you didn't, that is odd.
-
If you did, here is a dirty, dirty trick you can do. It will not live through a major upgrade, so be sure to make a backup of these conf files.... and I don't recommend it, but if you back it up... I'd like to hear how it went!!
- Make the changes to /var/lib/syslog.conf as you want
- Make a backup of /etc/syslog.conf, as in:
cd /etc
cp /etc/syslog.conf /etc/backup.syslog.config - Then, replace /etc/syslog.conf with /var/lib/syslog.conf by executing:
cp /var/lib/syslog.conf /etc/syslog.conf - Finally, make /etc/syslog.conf and /var/lib/syslog.conf READ ONLY:
chmod 400 /etc/syslog.conf
chmod 400 /var/lib/syslog.conf
This is a permanent cludge to ensure that:
- Whenever the syslog daemon is restarted (along with elastic syslog) any scripts, such as items mentioned in Tobias' comments above, don't make a copy of /etc/syslog.conf, inject the destination IP over and over again, and muck up your /var/lib/syslog.conf
-
-
If /var/lib/syslog.conf isn't surviving a reboot, then there must be a further upstream file that's changing it. Right?
-
@Dashrender said in Final Call ... XenServer Boot Media:
If /var/lib/syslog.conf isn't surviving a reboot, then there must be a further upstream file that's changing it. Right?
Presumably...
-
@Dashrender said in Final Call ... XenServer Boot Media:
If /var/lib/syslog.conf isn't surviving a reboot, then there must be a further upstream file that's changing it. Right?
Correct
-
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
Well, in 6.5 it is located at /etc/syslog.conf.
You sure? That goes against what we had determined.
chuckle - well, here are more semantics. that is the location that the system reads, but as shown above, XS has a script that changes that file upon startup, so the real place you need to edit is the location listed previously.
Correct, it is not the place where the configuration of the system is stored or made, it is where it is temporarily placed while running, it's a scratch file.
I'm sorry - what is a scratch file? If you're saying that /etc/syslog.conf is a scratch file, I don't currently agree. it's the file that almost all of linux systems use to set the syslog settings. To me that makes it the actual config file.
If I'm thinking about that wrongly, please explain where my thinking goes awry.It's a scratch file because it is a temp file made at run time for ephemeral settings. It IS the file, we think, ready by the rsyslog process (we could determine that pretty easily) but it is ephemeral, just a scratch file written by the real one whenever it wants to make a change. It's not where the configuration is stored, it's just part of the communications chain to the process - like a PID file or a network socket.
-
@BRRABill said in Final Call ... XenServer Boot Media:
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
Well, in 6.5 it is located at /etc/syslog.conf.
You sure? That goes against what we had determined.
Determined where? All we determined was that the article was from 6.2 and in 7 they changed everything.
No, we determined that it was 6.2 and that all evidence said that the change was in 6.5. We know 100% that things had changed by 7, and there is zero reason to not think that it changed in 6.5 and everything in that thread showed that it had indeed changed in 6.5.
Right - it's like the GPO example given above. XS has a script (like GPO) that it runs that edits /etc/syslog.conf so editing /etc/syslog.conf directly is pointless, like editing a windows machine registry is pointless because they will be over written by the script/GPO
The other file ALSO got overwritten.
@DustinB3403 can check and confirm or deny this, as well.
What is the other one? And did you check that there is no GUI running?
-
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
Well, in 6.5 it is located at /etc/syslog.conf.
You sure? That goes against what we had determined.
Determined where? All we determined was that the article was from 6.2 and in 7 they changed everything.
No, we determined that it was 6.2 and that all evidence said that the change was in 6.5. We know 100% that things had changed by 7, and there is zero reason to not think that it changed in 6.5 and everything in that thread showed that it had indeed changed in 6.5.
Right - it's like the GPO example given above. XS has a script (like GPO) that it runs that edits /etc/syslog.conf so editing /etc/syslog.conf directly is pointless, like editing a windows machine registry is pointless because they will be over written by the script/GPO
The other file ALSO got overwritten.
@DustinB3403 can check and confirm or deny this, as well.
What is the other one? And did you check that there is no GUI running?
/etc/syslog.conf
and
/var/lib/syslog.conf
-
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
Well, in 6.5 it is located at /etc/syslog.conf.
You sure? That goes against what we had determined.
Determined where? All we determined was that the article was from 6.2 and in 7 they changed everything.
No, we determined that it was 6.2 and that all evidence said that the change was in 6.5. We know 100% that things had changed by 7, and there is zero reason to not think that it changed in 6.5 and everything in that thread showed that it had indeed changed in 6.5.
Right - it's like the GPO example given above. XS has a script (like GPO) that it runs that edits /etc/syslog.conf so editing /etc/syslog.conf directly is pointless, like editing a windows machine registry is pointless because they will be over written by the script/GPO
The other file ALSO got overwritten.
@DustinB3403 can check and confirm or deny this, as well.
What is the other one? And did you check that there is no GUI running?
Do not know about the GUI. What would that affect?
-
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
Well, in 6.5 it is located at /etc/syslog.conf.
You sure? That goes against what we had determined.
Determined where? All we determined was that the article was from 6.2 and in 7 they changed everything.
No, we determined that it was 6.2 and that all evidence said that the change was in 6.5. We know 100% that things had changed by 7, and there is zero reason to not think that it changed in 6.5 and everything in that thread showed that it had indeed changed in 6.5.
Right - it's like the GPO example given above. XS has a script (like GPO) that it runs that edits /etc/syslog.conf so editing /etc/syslog.conf directly is pointless, like editing a windows machine registry is pointless because they will be over written by the script/GPO
The other file ALSO got overwritten.
@DustinB3403 can check and confirm or deny this, as well.
What is the other one? And did you check that there is no GUI running?
/etc/syslog.conf
and
/var/lib/syslog.conf
Those don't seem right.
-
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
Well, in 6.5 it is located at /etc/syslog.conf.
You sure? That goes against what we had determined.
Determined where? All we determined was that the article was from 6.2 and in 7 they changed everything.
No, we determined that it was 6.2 and that all evidence said that the change was in 6.5. We know 100% that things had changed by 7, and there is zero reason to not think that it changed in 6.5 and everything in that thread showed that it had indeed changed in 6.5.
Right - it's like the GPO example given above. XS has a script (like GPO) that it runs that edits /etc/syslog.conf so editing /etc/syslog.conf directly is pointless, like editing a windows machine registry is pointless because they will be over written by the script/GPO
The other file ALSO got overwritten.
@DustinB3403 can check and confirm or deny this, as well.
What is the other one? And did you check that there is no GUI running?
Do not know about the GUI. What would that affect?
Well a GUI obviously has to overwrite the master configuration file to do its job. The GUI is a form of a text editor to that file. So if you make a change by hand, then let the GUI control it and the GUI thinks you want something else, obviously the GUI has to change it, too. Too many cooks in the kitchen.
-
BTW, here is the directory this morning.
Apparently it CREATES the files each day, but does nothing with them.
There is still that 38M lastlog file, but I think that is a bug in XS7. (Or Linux itself.)
https://bugs.xenserver.org/browse/XSO-534From what I have read, that file maintains a DB of logins, and can be deleted.
drwxr-xr-x 2 root root 4096 Sep 7 10:10 blktap -rw-r--r-- 1 root root 38273316 Sep 7 09:36 lastlog -rw-rw-r-- 1 root utmp 44544 Sep 7 09:36 wtmp -rw------- 1 root utmp 1152 Sep 7 09:36 btmp -rw-r--r-- 1 root root 0 Sep 7 04:02 boot.log -rw------- 1 root root 0 Sep 7 04:02 cron -rw------- 1 root root 0 Sep 7 04:02 daemon.log -rw------- 1 root root 0 Sep 7 04:02 kern.log -rw------- 1 root root 0 Sep 7 04:02 maillog -rw------- 1 root root 0 Sep 7 04:02 messages -rw------- 1 root root 0 Sep 7 04:02 secure -rw------- 1 root root 0 Sep 7 04:02 SMlog -rw------- 1 root root 0 Sep 7 04:02 spooler -rw------- 1 root root 0 Sep 7 04:02 user.log -rw------- 1 root root 0 Sep 7 04:02 xcp-rrdd-plugins.log drwxr-xr-x 2 root root 4096 Sep 7 04:02 xen -rw------- 1 root root 0 Sep 7 04:02 xenstored-access.log -rw------- 1 root root 0 Sep 7 04:02 audit.log -rw-r--r-- 1 root root 0 Sep 7 04:02 interface-rename.log -rw------- 1 root root 0 Sep 7 00:40 xensource.log drwxr-xr-x 2 root root 4096 Sep 7 00:00 sa -
du says it is really 48K
-
@BRRABill said in Final Call ... XenServer Boot Media:
There is still that 38M lastlog file, but I think that is a bug in XS7. (Or Linux itself.)
https://bugs.xenserver.org/browse/XSO-534From what I have read, that file maintains a DB of logins, and can be deleted.
Yes, lastlog is a security mechanism, it is tiny and it is not part of syslogging. It's not likely 38M, that's enormous. Check that again.
-
@BRRABill said in Final Call ... XenServer Boot Media:
du says it is really 48K
Yes, it is a sparse file. Very small.
-
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
du says it is really 48K
Yes, it is a sparse file. Very small.
As I mentioned it is a bug of some sort, somewhere.
Seems like it happens across platforms.
-
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
du says it is really 48K
Yes, it is a sparse file. Very small.
As I mentioned it is a bug of some sort, somewhere.
Seems like it happens across platforms.
What is a bug?
-
lastlog is not a bug, it's exactly as intended.
-
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
du says it is really 48K
Yes, it is a sparse file. Very small.
As I mentioned it is a bug of some sort, somewhere.
Seems like it happens across platforms.
What is a bug?
" On several versions of RedHat Enterprise Linux and Fedora, corruption in this file can cause the size to be misrepresented. This has no effect on the real space used by the file, as reported by the du command."
-
@Dashrender said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
If you are trying to tell me that this ONE system that is set up like this is the norm, and all the others are incorrect or just plain dumb, well, then, fine.
General good practice (rule of thumb, NOT best practice) is to "always" use UTC for all service based systems (servers and similar devices.) End users set time for the user, not the system, so this does not normally apply to end users. But we've always set all servers to UTC since the late 1990s. It protects against time bugs from the 1990s, it makes logs way clearer, it keeps people like @Minion-Queen from causing time problems from getting confused on time zones, it lets teams in different regions work together seamlessly. Yes, UTC on everything for IT.
Huh, first time I've ever heard this - even being in SW for better than 5 years.
It is not common at all in the SMB because the SMB rarely does logging. When you get into logging it is almost always done in UTC on the backend. and the user can select their own timezone for their GUI to report if desired.
-
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
@scottalanmiller said in Final Call ... XenServer Boot Media:
@BRRABill said in Final Call ... XenServer Boot Media:
du says it is really 48K
Yes, it is a sparse file. Very small.
As I mentioned it is a bug of some sort, somewhere.
Seems like it happens across platforms.
What is a bug?
" On several versions of RedHat Enterprise Linux and Fedora, corruption in this file can cause the size to be misrepresented. This has no effect on the real space used by the file, as reported by the du command."
Who said that? That's totally wrong. It's space is not misrepresented, it's not a bug. Someone is confused. ls and du show different things, both are correct.
