Migrating Logs



  • As you guys know I'm reading a book on the linux command line before I read the cert book for RHCSA/RHCE. I try to consider use cases for everything and as such, started thinking about cat. You can combine files using cat test1.txt test2.txt > test3.txt and the file test3.txt will contain both test1 and test2.

    When you are migrating logs, are you using a cron job in conjunction with cat to migrate one big log file or do you just have a naming convention that makes using a wildcard easier?


  • Service Provider

    So the quick answer is "no", cat would not be used in this way.


  • Service Provider

    The longer answer is that if you wanted to combine your logs into a single file, which I can't think of why you would want to do that, but assuming you wanted a simple single file once a month or something with all logs to go to tape or something I suppose you could come up with a time, you would combine then with tar, not with cat because then you could retrieve them into their original logs rather than having them all merged into one, giant, unwieldy text file.



  • @scottalanmiller said in Migrating Logs:

    So the quick answer is "no", cat would not be used in this way.

    Am I correct in thinking that the naming convention is used with a wildcard to handle the migration of logs (as an example) that are migrated regularly?


  • Service Provider

    Far more often, what you will be doing is not using cat or tar at all, but instead using a tool like gzip or bzip2 to compress the individual log files to a fraction of their original size and leaving them as individual files. This is mostly better because you compress incrementally as you go reducing their size hour by hour, day by day or whatever granularity that you need. Then you can backup or ship them wherever you need whenever you need rather than waiting to bundle them all up before doing something with them. And retrieving them is just grabbing one small file rather than wading through one giant one.


  • Service Provider

    Of course you could tar up a bunch of gzipped files, but why bother? You cannot cat gzipped files, though.


  • Service Provider

    @wirestyle22 said in Migrating Logs:

    @scottalanmiller said in Migrating Logs:

    So the quick answer is "no", cat would not be used in this way.

    Am I correct in thinking that the naming convention is used with a wildcard to handle the migration of logs (as an example) that are migrated regularly?

    You would rarely use anything to grab stuff in that way with logs. A more common thing to do would either to have a script that does something complex, or a really simple command that does something like this...

    find all files over 24 hours old in the log directory whose names end in .gz and ship them to backupserver:/logarchive/



  • @scottalanmiller said in Migrating Logs:

    retrieving them is just grabbing one small file rather than wading through one giant one.

    The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.



  • @scottalanmiller said in Migrating Logs:

    @wirestyle22 said in Migrating Logs:

    @scottalanmiller said in Migrating Logs:

    So the quick answer is "no", cat would not be used in this way.

    Am I correct in thinking that the naming convention is used with a wildcard to handle the migration of logs (as an example) that are migrated regularly?

    You would rarely use anything to grab stuff in that way with logs. A more common thing to do would either to have a script that does something complex, or a really simple command that does something like this...

    find all files over 24 hours old in the log directory whose names end in .gz and ship them to backupserver:/logarchive/

    Understood. Thanks!


  • Service Provider

    @wirestyle22 said in Migrating Logs:

    @scottalanmiller said in Migrating Logs:

    retrieving them is just grabbing one small file rather than wading through one giant one.

    The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.

    If it isn't really big, what was the benefit of combining just a few?



  • @scottalanmiller said in Migrating Logs:

    @wirestyle22 said in Migrating Logs:

    @scottalanmiller said in Migrating Logs:

    retrieving them is just grabbing one small file rather than wading through one giant one.

    The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.

    If it isn't really big, what was the benefit of combining just a few?

    Yeah I guess the positive there is also the negative either way you go


  • Service Provider

    @wirestyle22 said in Migrating Logs:

    @scottalanmiller said in Migrating Logs:

    @wirestyle22 said in Migrating Logs:

    @scottalanmiller said in Migrating Logs:

    retrieving them is just grabbing one small file rather than wading through one giant one.

    The assumption here would be if it's getting done regularly it wouldn't be that big but I guess that differs depending on the situation.

    If it isn't really big, what was the benefit of combining just a few?

    Yeah I guess the positive there is also the negative either way you go

    Yes, merging just a few small files would be extra effort without benefit. If you merged enough to be beneficial in any way, you'd introduce loads of problems.


  • Service Provider

    Really, log shipping with local storage is a thing of the past as well. Not what you are looking for with your use case, but long ago people did this. Today if you want to store logs beyond what fits on the local system you look at remote log servers like syslog, rsyslog, Kiwi, Graylog, ELK, loggly, Splunk and so forth. They have more useful platforms for dealing with centralized logs, archiving and backups.



  • @scottalanmiller said in Migrating Logs:

    Really, log shipping with local storage is a thing of the past as well. Not what you are looking for with your use case, but long ago people did this. Today if you want to store logs beyond what fits on the local system you look at remote log servers like syslog, rsyslog, Kiwi, Graylog, ELK, loggly, Splunk and so forth. They have more useful platforms for dealing with centralized logs, archiving and backups.

    Then we get into why you would use each. What product benefits what situation :D


Log in to reply
 

Looks like your connection to MangoLassi was lost, please wait while we try to reconnect.