Need to parse large conf files



  • I have a number of Asterisk configuration files that I need to parse in order to pull out some key highlights.

    Someone tell me if I am off base on how to do this.....

    All asterisk configurations files have headers in square brakcets [from-voipms] or [bobs-your-uncle], etc. These are called contexts.

    After each context is lines of dial plan.

    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   Bob's inbound 3145551212                      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 3145551212,1,NoOp()
    same => n,Goto(main,123456,1)
    same => n,Hangup()
    

    What I need to do is parse the first line of each dial plan in each context
    I would also like to get any comment up to 3 lines before the first line of each dial plan.
    Comments always start with a ;
    Dial plans always start with exten => + some numbers or text + ,1,

    The underlying OS is Ubuntu 12.04.

    Google tells me to make a file such as parseit.sh with this content

    #!/bin/bash
    while IFS='' read -r line || [[ -n "$line" ]]; do
        echo "Current line is: $line"
    done < "$1"
    

    Make it executable chmod +x parseit.sh
    Then call it passing the file to parse as a parameter. ./parseit.sh extensions.conf

    This is easy.

    Now the need to create the logic to do what I said above.

    I will certainly need to figure out some regex to determine the context and first line of dial plan. This part I am not strong at.

    Getting the previous three lines is easy to put in the loop like this i think.

    while
        $linem3 = $linem2;
        $linem2 = $linem1;
        $linem1 = $linem0;
        $linem0 = $line;
        if [ REGEX OF LINE MATCHES PATTERN ]
        then
            DO ALL THE THINGS (like echo to screen for testing).
        fi 
    done 
    


  • Python has configparser which handles this files, INI files.

    Maybe this would help you Jared - https://github.com/albfan/bash-ini-parser



  • @JaredBusch

    Just to clarify, what is it you want to read from the config files, is it just the extension number?
    For example from exten => 3145551212,1,NoOp() to 3145551212 ?

    What should the end result be? A new file with all the extension numbers?



  • @Romo said in Need to parse large conf files:

    Python has configparser which handles this files, INI files.

    Maybe this would help you Jared - https://github.com/albfan/bash-ini-parser

    Neither are very helpful.



  • @Pete-S said in Need to parse large conf files:

    @JaredBusch

    Just to clarify, what is it you want to read from the config files, is it just the extension number?
    For example from exten => 3145551212,1,NoOp() to 3145551212 ?

    What should the end result be? A new file with all the extension numbers?

    For now, I want the comments and the exten line.

    Once i have it in a form I can analyze, i will be writing more statements to do specific things.



  • Went and had some lunch. Getting back on this now.



  • @JaredBusch said in Need to parse large conf files:

    Went and had some lunch. Getting back on this now.

    To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.



  • @Obsolesce said in Need to parse large conf files:

    @JaredBusch said in Need to parse large conf files:

    Went and had some lunch. Getting back on this now.

    To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

    What good would that do me when the OS is Ubuntu 12.04?



  • @JaredBusch said in Need to parse large conf files:

    @Obsolesce said in Need to parse large conf files:

    @JaredBusch said in Need to parse large conf files:

    Went and had some lunch. Getting back on this now.

    To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

    What good would that do me when the OS is Ubuntu 12.04?

    My thought was that you could grab the file and run it on something with PS.

    Or is this something that needs to run on the system? Perhaps it'll work with PS6.1?



  • @JaredBusch said in Need to parse large conf files:

    @Obsolesce said in Need to parse large conf files:

    @JaredBusch said in Need to parse large conf files:

    Went and had some lunch. Getting back on this now.

    To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

    What good would that do me when the OS is Ubuntu 12.04?

    Will PS install via AppImage in 12.04? That's so old, not sure what it can do.



  • @JaredBusch

    It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

    I'll write some pseudo correct php code for you here to show what I mean.

    // find matching files, put in array called filename
    $filename:=glob("*.ini");
    
    // scan through all the files
    foreach ($filename as $f) {
    
       // read all lines into lines array
       $lines:=file($f);
    
       // go through the file and pick out what we need
       $line:=0; // linenumber
       foreach ($lines as $line) do {
          $line++;
    
          // check the start of each line to look for "exten =>"
          $search:="exten =>";
          if substr($line,0,strlen($search))==$search {
             // we found the extension row
             // lets take everything after "exten =>"
             $extline:=substr($line,strlen($search));
             // lets divide it up 
             $parts:=explode($extline, ",");
             $ext:=$parts[0]; // extension number
    
             // lets pick out the three preceeding lines
             $comment[1]:=$lines[$line-3];
             $comment[2]:=$lines[$line-2];
             $comment[3]:=$lines[$line-1];
    
             // do more stuff on the line we picked out here
            
             // lets save what we might need in an array
             $extensions[$ext]:=array("comments"=>$comment, "linenumber"=>$line, "file"=>$f);
          }
          
          // search for other stuff and what not
       }
       
       // do more stuff on the file
    }
    
    // remove comment to print array for inspection/debug
    //print_r($extensions);
    
    // lets print out all extension found in all files
    foreach ($extensions as $ext => $data) do {
       print "Extension: $ext\r\n";
    }
    
    


  • @Pete-S said in Need to parse large conf files:

    @JaredBusch

    It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

    I'll write some pseudo correct php code for you here to show what I mean.

    // find matching files, put in array called filename
    $filename:=glob("*.ini");
    
    // scan through all the files
    foreach ($filename as $f) {
    
       // read all lines into lines array
       $lines:=file($f);
    
       // go through the file and pick out what we need
       $line:=0; // linenumber
       foreach ($lines as $line) do {
          $line++;
    
          // check the start of each line to look for "exten =>"
          $search:="exten =>";
          if substr($line,0,strlen($search))==$search {
             // we found the extension row
             // lets take everything after "exten =>"
             $extline:=substr($line,strlen($search));
             // lets divide it up 
             $parts:=explode($extline, ",");
             $ext:=$parts[0]; // extension number
    
             // lets pick out the three preceeding lines
             $comment[1]:=$lines[$line-3];
             $comment[2]:=$lines[$line-2];
             $comment[3]:=$lines[$line-1];
    
             // do more stuff on the line we picked out here
            
             // lets save what we might need in an array
             $extensions[$ext]:=array("comments"=>$comment, "linenumber"=>$line, "file"=>$f);
          }
          
          // search for other stuff and what not
       }
       
       // do more stuff on the file
    }
    
    // remove comment to print array for inspection/debug
    //print_r($extensions);
    
    // lets print out all extension found in all files
    foreach ($extensions as $ext => $data) do {
       print "Extension: $ext\r\n";
    }
    
    

    And does all of that work with PHP 5.3

    9a35546f-ac40-450d-93fb-859071c04aa1-image.png



  • @Pete-S said in Need to parse large conf files:

    @JaredBusch

    It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

    Since when is a bash script not doing something programmatically?



  • @JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.



  • @matteo-nunziati said in Need to parse large conf files:

    @JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

    Like each line that begins with ; and exten?



  • @Obsolesce if schema is:
    [Blabla]
    ; Comment line1
    ; Comment line 2
    ; Comment line 3
    exten =>

    Yes you can ask for:
    Grep -e "[.*]" -A 4

    Or similar: I'm in bed with my smartphone atm... Dont' remember the synthax

    Grrr squared brackets have to be escaped but I can't...



  • @JaredBusch said in Need to parse large conf files:

    @Pete-S said in Need to parse large conf files:

    @JaredBusch

    It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

    Since when is a bash script not doing something programmatically?

    Well, bash is programs of course a program but using grep for instance you are defining search patterns and using command line options to get it to pick out specific lines. It's not the same as writing a program that would go back and forth in the files and pick out what you need and put it together anyway you like.



  • @JaredBusch said in Need to parse large conf files:

    And does all of that work with PHP 5.3

    9a35546f-ac40-450d-93fb-859071c04aa1-image.png

    In general yes (you just run it with php filename.php) but I'd have to have some files and debug it to know for sure. I just wrote it on the top of my head as an example just as I'm writing this.

    You might favor another language. Python, java, javascript, what do I know.



  • @matteo-nunziati said in Need to parse large conf files:

    @JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

    The is what is needed.

    4e10f724-9c5c-4c6e-8a2e-d85779c58b69-image.png

    There are many more contexts and many dial pans per context.

    They are all (generally commented).



  • @JaredBusch said in Need to parse large conf files:

    @matteo-nunziati said in Need to parse large conf files:

    @JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

    The is what is needed.

    4e10f724-9c5c-4c6e-8a2e-d85779c58b69-image.png

    There are many more contexts and many dial pans per context.

    They are all (generally commented).

    So basically all extension numbers and all preceding comment lines for those extensions under each [context]?

    And what should the end result look like?



  • Switching to PHP because the shell script is having issues, i assume because some characters are breakout out of the string variables.



  • quite late here... sorry I've just an half backed solution:

     grep -e "^exten\|^[\|^;;" filename
    

    this doesn't retain any structure but extracts any line beginning with [...] or exten or ;;
    unfortunately comments are shown regardless of their usefulness...

    maybe pocking with awk would lead to better results... but I really need to sleep!



  • @matteo-nunziati said in Need to parse large conf files:

    quite late here... sorry I've just an half backed solution:

     grep -e "^exten\|^[\|^;;" filename
    

    That is the ballpark, but many lines are comments that are not needed.

    I was simply going to loop down the file each time through the loop putting the previosu line back one vairable. that part was simple.

    The catching the exten => STUFF,1 is the harder part. it HAS to be that full syntaxt.



  • @JaredBusch

    Jared, will this do as a test file?

    ;; This is a test file
    [context_1]
    
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   Ben's inbound 2344242342                      ;
    ;    This extension is not suppose to go through   ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 2344242342,2,NoOp()
    same => n,Goto(main,123456,1)
    same => n,Hangup()
    
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   Bob's inbound 3145551212                      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 3145551212,1,NoOp()
    same => n,Goto(main,123456,1)
    same => n,Hangup()
    
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;  Mary's inbound 4534535345                      ;
    ;   With some added comments                       ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 4534535345,1,NoOp()
    same => n,Goto(main,123456,1)
    same => n,Hangup()
    
    [context_2]
    exten => 33333333,1,NoOp()
    same => n,Goto(main,123456,1)
    same => n,Hangup()
    
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   What is this????????????                      ;
    ;    Lets add a long comment section               ;
    ;    Line 3,,,                                     ;
    ;    Line 4;Let's see if that comment remains      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 3145454,1,NoOp()
    same => n,Goto(main,123456,1)
    same => n,Hangup()
    
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;  Let's make a wonky comment line here
    
    ;   And lets add this too                         ;;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 232342,1,NoOp()
    same => n,Hangup()
    
    [context_3]
    exten => 7777777,1
    exten => 8888888,1,NoOp()
    ;; Let's do these without comments
    


  • @Pete-S said in Need to parse large conf files:

    @JaredBusch

    Jared, will this do as a test file?

    Close enough. I just moved your original sample code to the server as a starting point. (no PHP installed on my laptop to test locally).



  • @JaredBusch said in Need to parse large conf files:

    @Pete-S said in Need to parse large conf files:

    @JaredBusch

    Jared, will this do as a test file?

    Close enough. I just moved your original sample code to the server as a starting point. (no PHP installed on my laptop to test locally).

    Let me just make some changes and test it on the test file. And I'll post it again.

    PS. And what do you want the output to look like?



  • @JaredBusch said in Need to parse large conf files:

    The catching the exten => STUFF,1 is the harder part. it HAS to be that full syntaxt.

    no it is simple with regex IF the line is:

    exten => bla,1,bla

    try:

    ^exten => .*,1,.*$
    


  • @Pete-S said in Need to parse large conf files:

    And what should the end result look like?

    more or less this

    [context 1]
    ; comment line -3 if it exists
    ; comment line -2 if it exists
    ; comment line -1 if it exists
    exten => STUFF,1,STUFF
    

    repeat for dial plans and context appropriately.



  • @matteo-nunziati said in Need to parse large conf files:

    @JaredBusch said in Need to parse large conf files:

    The catching the exten => STUFF,1 is the harder part. it HAS to be that full syntaxt.

    no it is simple with regex IF the line is:

    exten => bla,1,bla

    100% those are the only exten lines I want.



  • @JaredBusch said in Need to parse large conf files:

    @Pete-S said in Need to parse large conf files:

    And what should the end result look like?

    more or less this

    [context 1]
    ; comment line -3 if it exists
    ; comment line -2 if it exists
    ; comment line -1 if it exists
    exten => STUFF,1,STUFF
    

    repeat for dial plans and context appropriately.

    All the comment lines or just the last/first three?