Need to parse large conf files



  • @JaredBusch Could you check that you can run command line php on the server?
    Just execute php -v and you'll get the version.

    At least on debian you need to have php-cli package and I don't know if you do.



  • Just for fun I found some extensions.conf file on the net to try. It led me to add some better formatting so the output is more readable.

    PS. Just a small change like this that would have caused hours of search on regex, options for grep, awk, sed etc to try and make something work. That's why I think it's easier to just go full on programming language as soon as it is more complex than just finding a few lines that matches.

    I know as this is all for naught but whatever.

    Output from the php script:

    File: sample.conf
    
    [context_1]
    ; Bob's inbound 3145551212
    exten => 3145551212,1,NoOp()
    
    ; Mary's inbound 4534535345
    ; With some added comments
    exten => 4534535345,1,NoOp()
    
    [context_2]
    exten => 33333333,1,NoOp()
    
    ; What is this????????????
    ; Lets add a long comment section
    ; Line 3,,,
    exten => 3145454,1,NoOp()
    
    ; Let's make a wonky comment line here
    ; And lets add this too
    exten => 232342,1,NoOp()
    
    [context_3]
    exten => 7777777,1
    
    exten => 8888888,1,NoOp()
    

    PHP script v2

    <?php
       // Ver 2 - improved formatting, only showing headings that have extensions
       
       // find matching files, put in array called filename
       $filename=glob("*.conf");
       
    
       // scan through all the files
       foreach ($filename as $f) {
          // print filename
          print "File: $f\n\n";
    
          // read all lines into lines array
          $lines=file($f);
    
          $no=0; // linenumber
          $comment=[]; // comments array
          $context=''; // context text
    
          // go through the file and pick out what we need
          foreach ($lines as $line) {
             $no++;
    
             // look for [context]
             if ($line[0]=='[') {
                $context=trim($line);
             }
             
             // look for extensions
             $search="exten =>";
             if (substr($line,0,strlen($search))==$search) {
                //print $line;
    
                // we found the extension row
                // lets take everything after "exten =>"
                $extline=trim(substr($line,strlen($search)));
    
                // lets divide it up
                $parts=explode(',', $extline);
    
                // check that it's ?????,1,?????
                if ($parts[1]=='1') {
                   // valid extension, actual extension is $parts[0];
                   $ext=trim($line);
                   
                   // print context if it has not been printed
                   if ($context>'') { print "$context\n"; $context=''; }
                   // print comments
                   foreach($comment as $c) print "; $c\n";
                   // print the extension and empty line
                   print "$ext\n\n";
                }
                $comments=[]; // clear comments
             }
             
             // look for comments
             if ($line[0]==';') {
                // remove whitespace and ;
                $s=trim($line,"; \t\n\r\0\x0B");
                // add to comments if not empty but only first 3 lines
                if (($s>'') and (count($comment)<3)) $comment[]=$s;
             } else {
    
                // non empty line? => clear comments
                if (trim($line)>'') $comment=[];
    
             }
          }
       }
    
    
    ?>
    


  • @JaredBusch
    Also is it just one conf file you need to search through or many in the same directory ?



  • @Pete-S said in Need to parse large conf files:

    @JaredBusch Could you check that you can run command line php on the server?
    Just execute php -v and you'll get the version.

    At least on debian you need to have php-cli package and I don't know if you do.

    I can.



  • @Pete-S said in Need to parse large conf files:

    @JaredBusch
    Also is it just one conf file you need to search through or many in the same directory ?

    many. but feeding that into a loop is simple once I work this out.



  • @JaredBusch said in Need to parse large conf files:

    @Pete-S said in Need to parse large conf files:

    @JaredBusch
    Also is it just one conf file you need to search through or many in the same directory ?

    many. but feeding that into a loop is simple once I work this out.

    I had file scanning already in the example file but can take it out. The reason I'm asking is if you need to have the filename too in the "report" for each extension?



  • @Pete-S said in Need to parse large conf files:

    @JaredBusch said in Need to parse large conf files:

    @Pete-S said in Need to parse large conf files:

    @JaredBusch
    Also is it just one conf file you need to search through or many in the same directory ?

    many. but feeding that into a loop is simple once I work this out.

    I had file scanning already in the example file but can take it out. The reason I'm asking is if you need to have the filename too in the "report" for each extension?

    Yes. but why i dunno. this is all a hand coded mess when compared to standards.

    It is all perfectly valid syntax. and there are comments all over. but jsut wow.



  • ok my last s**t, just in case @JaredBusch doesn't know what to do waiting for @Pete-S

    grep -B3 -n -e "^exten\|^[" $YOURFILENAME | sort -n | uniq | cut -f2 -d: | cut -f2 -d- | grep -e "^[e[\;]"
    

    still not granted to get only valid exten



  • @matteo-nunziati said in Need to parse large conf files:

    ok my last s**t, just in case @JaredBusch doesn't know what to do waiting for @Pete-S

    I'm not waiting. I'm doing my thing too.



  • @matteo-nunziati said in Need to parse large conf files:

    ok my last s**t, just in case @JaredBusch doesn't know what to do waiting for @Pete-S

    grep -B3 -n -e "^exten\|^[" $YOURFILENAME | sort -n | uniq | cut -f2 -d: | cut -f2 -d- | grep -e "^[e[\;]"
    

    still not granted to get only valid exten

    this is the outcome for @Pete-S test file:

    ;; This is a test file
    [context_1]
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   Ben's inbound 2344242342                      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 2344242342,1,NoOp()
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   Bob's inbound 3145551212                      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 3145551212,1,NoOp()
    ;;  Mary's inbound 4534535345                      ;
    ;   With some added comments                       ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 4534535345,1,NoOp()
    [context_2]
    exten => 33333333,1,NoOp()
    ;    Line 3,,,                                     ;
    ;    Line 4;Let's see if that comment remains      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 3145454,1,NoOp()
    ;   And lets add this too                         ;;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 232342,1,NoOp()
    [context_3]
    exten => 7777777,1
    exten => 8888888,1,NoOp()
    


  • @matteo-nunziati said in Need to parse large conf files:

    ok my last s**t, just in case @JaredBusch doesn't know what to do waiting for @Pete-S
    still not granted to get only valid exten

    doens't work right on a live file. close. but not correct.



  • @JaredBusch said in Need to parse large conf files:

    // find matching files, put in array called filename
    $filename:=glob("*.ini");

    it doesn't like the :

    I moved a copy of the files to the FreePBX 14 box since PHP 5.6 is "current" for CentOS.

    [[email protected] newton_conf]$ sudo -u asterisk php parseit.php 
    PHP Parse error:  syntax error, unexpected ':' in /var/www/html/newton_conf/parseit.php on line 3
    

    2821a8fe-1c2b-4c43-9f87-68a5baecf3aa-image.png

    3817d8e0-37a5-4835-870f-0158ff0cd659-image.png



  • @JaredBusch

    Hang on a sec. I'm just about finished.



  • This is the output:

    File: sample.conf
    [context_1]
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   Bob's inbound 3145551212                      ;
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    exten => 3145551212,1,NoOp()
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;  Mary's inbound 4534535345                      ;
    ;   With some added comments                       ;
    exten => 4534535345,1,NoOp()
    [context_2]
    exten => 33333333,1,NoOp()
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;   What is this????????????                      ;
    ;    Lets add a long comment section               ;
    exten => 3145454,1,NoOp()
    ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    ;;  Let's make a wonky comment line here
    ;   And lets add this too                         ;;
    exten => 232342,1,NoOp()
    [context_3]
    exten => 7777777,1
    exten => 8888888,1,NoOp()
    

    Personally I'd format it a little for instance by remove all comment lines that are just ;;;;;; and all trailing and preceding ;

    Or if it's just for viewing I'd html it.



  • This is the complete php file with comments.

    It takes all *.conf files and goes through them.

    <?php
    
       // find matching files, put in array called filename
       $filename=glob("*.conf");
       
    
       // scan through all the files
       foreach ($filename as $f) {
          // print filename
          print "File: $f\n";
    
          // read all lines into lines array
          $lines=file($f);
    
          $no=0; // linenumber
          $comment=[]; // comments array
    
          // go through the file and pick out what we need
          foreach ($lines as $line) {
             $no++;
    
             // look for [context]
             if ($line[0]=='[') {
                $context=$line;
                print "$context";
             }
             
             // look for extensions
             $search="exten =>";
             if (substr($line,0,strlen($search))==$search) {
                //print $line;
    
                // we found the extension row
                // lets take everything after "exten =>"
                $extline=trim(substr($line,strlen($search)));
    
                // lets divide it up
                $parts=explode(',', $extline);
                $ext=$parts[0]; // extension number
    
                // check that it's ?????,1,?????
                if ($parts[1]=='1') {
                   // valid extension, actual extension is $parts[0];
                   $ext=$line;
                   // print comments
                   foreach($comment as $c) print "$c";
                   // print the extesion and empty line
                   print "$ext";
                }
                $comments=[]; // clear comments
             }
             
             // look for comments
             if ($line[0]==';') {
                // add to comments but only first 3 lines
                if (count($comment)<3) $comment[]=$line;
                //print_r($comment);
             } else {
    
                // non empty line? => clear comments
                if (trim($line)>'') $comment=[];
    
             }
          }
       }
    
    ?>
    


  • @JaredBusch said in Need to parse large conf files:

    @JaredBusch said in Need to parse large conf files:

    // find matching files, put in array called filename
    $filename:=glob("*.ini");

    it doesn't like the :

    I moved a copy of the files to the FreePBX 14 box since PHP 5.6 is "current" for CentOS.

    It's because I've been working with another language all day that uses := to assign a variable and not just =

    Takes me a few minutes to reboot my brain into php mode. 😊



  • Just for fun I found some extensions.conf file on the net to try. It led me to add some better formatting so the output is more readable.

    PS. Just a small change like this that would have caused hours of search on regex, options for grep, awk, sed etc to try and make something work. That's why I think it's easier to just go full on programming language as soon as it is more complex than just finding a few lines that matches.

    I know as this is all for naught but whatever.

    Output from the php script:

    File: sample.conf
    
    [context_1]
    ; Bob's inbound 3145551212
    exten => 3145551212,1,NoOp()
    
    ; Mary's inbound 4534535345
    ; With some added comments
    exten => 4534535345,1,NoOp()
    
    [context_2]
    exten => 33333333,1,NoOp()
    
    ; What is this????????????
    ; Lets add a long comment section
    ; Line 3,,,
    exten => 3145454,1,NoOp()
    
    ; Let's make a wonky comment line here
    ; And lets add this too
    exten => 232342,1,NoOp()
    
    [context_3]
    exten => 7777777,1
    
    exten => 8888888,1,NoOp()
    

    PHP script v2

    <?php
       // Ver 2 - improved formatting, only showing headings that have extensions
       
       // find matching files, put in array called filename
       $filename=glob("*.conf");
       
    
       // scan through all the files
       foreach ($filename as $f) {
          // print filename
          print "File: $f\n\n";
    
          // read all lines into lines array
          $lines=file($f);
    
          $no=0; // linenumber
          $comment=[]; // comments array
          $context=''; // context text
    
          // go through the file and pick out what we need
          foreach ($lines as $line) {
             $no++;
    
             // look for [context]
             if ($line[0]=='[') {
                $context=trim($line);
             }
             
             // look for extensions
             $search="exten =>";
             if (substr($line,0,strlen($search))==$search) {
                //print $line;
    
                // we found the extension row
                // lets take everything after "exten =>"
                $extline=trim(substr($line,strlen($search)));
    
                // lets divide it up
                $parts=explode(',', $extline);
    
                // check that it's ?????,1,?????
                if ($parts[1]=='1') {
                   // valid extension, actual extension is $parts[0];
                   $ext=trim($line);
                   
                   // print context if it has not been printed
                   if ($context>'') { print "$context\n"; $context=''; }
                   // print comments
                   foreach($comment as $c) print "; $c\n";
                   // print the extension and empty line
                   print "$ext\n\n";
                }
                $comments=[]; // clear comments
             }
             
             // look for comments
             if ($line[0]==';') {
                // remove whitespace and ;
                $s=trim($line,"; \t\n\r\0\x0B");
                // add to comments if not empty but only first 3 lines
                if (($s>'') and (count($comment)<3)) $comment[]=$s;
             } else {
    
                // non empty line? => clear comments
                if (trim($line)>'') $comment=[];
    
             }
          }
       }
    
    
    ?>
    


  • @Pete-S this all hit as I was commuting home.

    I have a copy of the files on a server I have access to from home and will be working on this tomorrow.

    Thanks.



  • @Pete-S Well tomorrow becamse the following Wednesday.. But. this works perfectly.

    Now to figure out WTF all this stuff is that I need to replicate into FreePBX.

    8c904acf-bfbc-480e-855a-b0985bd7c306-image.png

    073eba70-ab8c-4c70-b633-2b41ea9972a2-image.png



  • @JaredBusch Glad it worked as intended.