Need to parse large conf files

JaredBusch

Went and had some lunch. Getting back on this now.

Obsolesce

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

JaredBusch

@Obsolesce said in Need to parse large conf files:

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

What good would that do me when the OS is Ubuntu 12.04?

Obsolesce

@JaredBusch said in Need to parse large conf files:

@Obsolesce said in Need to parse large conf files:

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

What good would that do me when the OS is Ubuntu 12.04?

My thought was that you could grab the file and run it on something with PS.

Or is this something that needs to run on the system? Perhaps it'll work with PS6.1?

scottalanmiller

@JaredBusch said in Need to parse large conf files:

@Obsolesce said in Need to parse large conf files:

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

What good would that do me when the OS is Ubuntu 12.04?

Will PS install via AppImage in 12.04? That's so old, not sure what it can do.

1337

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

I'll write some pseudo correct php code for you here to show what I mean.

// find matching files, put in array called filename
$filename:=glob("*.ini");

// scan through all the files
foreach ($filename as $f) {

   // read all lines into lines array
   $lines:=file($f);

   // go through the file and pick out what we need
   $line:=0; // linenumber
   foreach ($lines as $line) do {
      $line++;

      // check the start of each line to look for "exten =>"
      $search:="exten =>";
      if substr($line,0,strlen($search))==$search {
         // we found the extension row
         // lets take everything after "exten =>"
         $extline:=substr($line,strlen($search));
         // lets divide it up 
         $parts:=explode($extline, ",");
         $ext:=$parts[0]; // extension number

         // lets pick out the three preceeding lines
         $comment[1]:=$lines[$line-3];
         $comment[2]:=$lines[$line-2];
         $comment[3]:=$lines[$line-1];

         // do more stuff on the line we picked out here
        
         // lets save what we might need in an array
         $extensions[$ext]:=array("comments"=>$comment, "linenumber"=>$line, "file"=>$f);
      }
      
      // search for other stuff and what not
   }
   
   // do more stuff on the file
}

// remove comment to print array for inspection/debug
//print_r($extensions);

// lets print out all extension found in all files
foreach ($extensions as $ext => $data) do {
   print "Extension: $ext\r\n";
}

JaredBusch

@Pete-S said in Need to parse large conf files:

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

I'll write some pseudo correct php code for you here to show what I mean.

// find matching files, put in array called filename
$filename:=glob("*.ini");

// scan through all the files
foreach ($filename as $f) {

   // read all lines into lines array
   $lines:=file($f);

   // go through the file and pick out what we need
   $line:=0; // linenumber
   foreach ($lines as $line) do {
      $line++;

      // check the start of each line to look for "exten =>"
      $search:="exten =>";
      if substr($line,0,strlen($search))==$search {
         // we found the extension row
         // lets take everything after "exten =>"
         $extline:=substr($line,strlen($search));
         // lets divide it up 
         $parts:=explode($extline, ",");
         $ext:=$parts[0]; // extension number

         // lets pick out the three preceeding lines
         $comment[1]:=$lines[$line-3];
         $comment[2]:=$lines[$line-2];
         $comment[3]:=$lines[$line-1];

         // do more stuff on the line we picked out here
        
         // lets save what we might need in an array
         $extensions[$ext]:=array("comments"=>$comment, "linenumber"=>$line, "file"=>$f);
      }
      
      // search for other stuff and what not
   }
   
   // do more stuff on the file
}

// remove comment to print array for inspection/debug
//print_r($extensions);

// lets print out all extension found in all files
foreach ($extensions as $ext => $data) do {
   print "Extension: $ext\r\n";
}

And does all of that work with PHP 5.3

JaredBusch

@Pete-S said in Need to parse large conf files:

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

Since when is a bash script not doing something programmatically?

matteo nunziati

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

Obsolesce

@matteo-nunziati said in Need to parse large conf files:

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

Like each line that begins with ; and exten?

matteo nunziati

@Obsolesce if schema is:
[Blabla]
; Comment line1
; Comment line 2
; Comment line 3
exten =>

Yes you can ask for:
Grep -e "\[.*\]" -A 4

Or similar: I'm in bed with my smartphone atm... Dont' remember the synthax

Grrr squared brackets have to be escaped but I can't...

1337

@JaredBusch said in Need to parse large conf files:

@Pete-S said in Need to parse large conf files:

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

Since when is a bash script not doing something programmatically?

Well, bash is programs of course a program but using grep for instance you are defining search patterns and using command line options to get it to pick out specific lines. It's not the same as writing a program that would go back and forth in the files and pick out what you need and put it together anyway you like.

1337

@JaredBusch said in Need to parse large conf files:

And does all of that work with PHP 5.3

In general yes (you just run it with php filename.php) but I'd have to have some files and debug it to know for sure. I just wrote it on the top of my head as an example just as I'm writing this.

You might favor another language. Python, java, javascript, what do I know.

JaredBusch

@matteo-nunziati said in Need to parse large conf files:

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

The is what is needed.

There are many more contexts and many dial pans per context.

They are all (generally commented).

1337

@JaredBusch said in Need to parse large conf files:

@matteo-nunziati said in Need to parse large conf files:

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

The is what is needed.

There are many more contexts and many dial pans per context.

They are all (generally commented).

So basically all extension numbers and all preceding comment lines for those extensions under each [context]?

And what should the end result look like?

JaredBusch

Switching to PHP because the shell script is having issues, i assume because some characters are breakout out of the string variables.

matteo nunziati

quite late here... sorry I've just an half backed solution:

 grep -e "^exten\|^\[\|^;;" filename

this doesn't retain any structure but extracts any line beginning with [...] or exten or ;;
unfortunately comments are shown regardless of their usefulness...

maybe pocking with awk would lead to better results... but I really need to sleep!

JaredBusch

@matteo-nunziati said in Need to parse large conf files:

quite late here... sorry I've just an half backed solution:
 grep -e "^exten\|^\[\|^;;" filename

That is the ballpark, but many lines are comments that are not needed.

I was simply going to loop down the file each time through the loop putting the previosu line back one vairable. that part was simple.

The catching the exten => STUFF,1 is the harder part. it HAS to be that full syntaxt.

1337

@JaredBusch

Jared, will this do as a test file?

;; This is a test file
[context_1]

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;   Ben's inbound 2344242342                      ;
;    This extension is not suppose to go through   ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
exten => 2344242342,2,NoOp()
same => n,Goto(main,123456,1)
same => n,Hangup()

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;   Bob's inbound 3145551212                      ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
exten => 3145551212,1,NoOp()
same => n,Goto(main,123456,1)
same => n,Hangup()

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;  Mary's inbound 4534535345                      ;
;   With some added comments                       ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
exten => 4534535345,1,NoOp()
same => n,Goto(main,123456,1)
same => n,Hangup()

[context_2]
exten => 33333333,1,NoOp()
same => n,Goto(main,123456,1)
same => n,Hangup()

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;   What is this????????????                      ;
;    Lets add a long comment section               ;
;    Line 3,,,                                     ;
;    Line 4;Let's see if that comment remains      ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
exten => 3145454,1,NoOp()
same => n,Goto(main,123456,1)
same => n,Hangup()

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;  Let's make a wonky comment line here

;   And lets add this too                         ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
exten => 232342,1,NoOp()
same => n,Hangup()

[context_3]
exten => 7777777,1
exten => 8888888,1,NoOp()
;; Let's do these without comments

JaredBusch

@Pete-S said in Need to parse large conf files:

@JaredBusch

Jared, will this do as a test file?

Close enough. I just moved your original sample code to the server as a starting point. (no PHP installed on my laptop to test locally).