File Parsing Magic

scottalanmiller

You can process the output with uniq to make it easier to get unique results without needing to use MySQL.

Jason

Is this on windows or linux you didn't specify.

On windows use Powershell, on linux use Bash.

scottalanmiller

@Jason said in File Parsing Magic:

Is this on windows or linux you didn't specify.

On windows use Powershell, on linux use Bash.

You've got BASH on Windows now, right?

scottalanmiller

I assumed Linux since he is using MySQL which you would only run on Linux normally.

Jason

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

scottalanmiller

@Jason said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

It should still parse text, though. In theory. Maybe.

travisdh1

@scottalanmiller said in File Parsing Magic:

@Jason said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

It should still parse text, though. In theory. Maybe.

Assuming it has access to the base system and not just it's own container. From everything I've read so far it's more like a Docker container than actually BASH on Windows... if you want that you're still stuck with cygwin or the like.

Jason

@travisdh1 said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

@Jason said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

It should still parse text, though. In theory. Maybe.

Assuming it has access to the base system and not just it's own container. From everything I've read so far it's more like a Docker container than actually BASH on Windows... if you want that you're still stuck with cygwin or the like.

Parse text sure.. getting the text he wants with a script into it in the first place, not so sure.

Romo

@travisdh1 You have access to all the files in Windows from /mnt/c , so yeah you can easily parse the text with the script provided by @scottalanmiller

travisdh1

@Romo said in File Parsing Magic:

@travisdh1 You have access to all the files in Windows from /mnt/c , so yeah you can easily parse the text with the script provided by @scottalanmiller

Ok, so it's more like cygwin than Docker. Thanks for the correction/confirmation.

jyates

If windows, powershell has split and trim functions.

$this = $this.ToString().Split("name=",2)[1].Split(";",4)
$name = $this[0].split("=",2)[1]
$ip = $this[2].Trim("ip=")

anthonyh

@scottalanmiller said in File Parsing Magic:

Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

OMG SAM you are the best!

Sorry for not being clear. This is all under Linux VMs on-prem in my own environment (XenServer).

anthonyh

@scottalanmiller said in File Parsing Magic:

Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

This works 75% of the time, but it looks like some log entries show when a user is syncing an item shared by another user, which does not result in the desired output.

mailbox.log.2016-04-19:2016-04-19 01:27:53,338 INFO [qtp509886383-480009:https://10.39.6.4:443/service/soap/SyncRequest] [[email protected];[email protected];mid=14;ip=10.39.253.62;ua=ZCO/8.6.0.1320 (6.1.7601 SP1 en-US) P9b4 T1404;] soap - SyncRequest elapsed=4

What happens here is you get the following:

[email protected];14

Desired output is:

[email protected];10.39.253.62

scottalanmiller

That's because your log format changed. That second one has more fields in it.

RamblingBiped

@scottalanmiller said in File Parsing Magic:

Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

Wait, I think there is a more important question that needs to be answered now. If you echo an echo, do you get an echoed echo's echo, or do they just cancel each other out and build a strange uncomfortable silence?

RamblingBiped

@RamblingBiped said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:
Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse
Wait, I think there is a more important question that needs to be answered now. If you echo an echo, do you get an echoed echo's echo, or do they just cancel each other out and build a strange uncomfortable silence?

And to follow up, if you simultaneously echo two echos from a single echo, will your head explode or somehow magically stay intact?

anthonyh

@scottalanmiller

Understood. I need to figure out a way to parse the file so that the process finds "user=" and pulls everything after it until it hits the following ";", then finds "ip=" and pulls everything after it until it hits the following ";"

scottalanmiller

@anthonyh said in File Parsing Magic:

@scottalanmiller

Understood. I need to figure out a way to parse the file so that the process finds "user=" and pulls everything after it until it hits the following ";", then finds "ip=" and pulls everything after it until it hits the following ";"

Yes, which is basically what I did but the cut command can only use a single character delimiter.

RamblingBiped

@scottalanmiller said in File Parsing Magic:

@anthonyh said in File Parsing Magic:

@scottalanmiller

Understood. I need to figure out a way to parse the file so that the process finds "user=" and pulls everything after it until it hits the following ";", then finds "ip=" and pulls everything after it until it hits the following ";"

Yes, which is basically what I did but the cut command can only use a single character delimiter.

Could he pipe it into awk, use the "." as a delimeter and the print all fields preceding each "."?

Brett

I'm very much a Linux noob, so I don't know what command to use. But I'd just use a regular expression alone or perhaps in combination with some other command to get the desired text here. In Powershell I would use the -match operator and/or the Select-String cmdlet.