XMLStarlet - Argument List Too Long



  • Hey All,

    I'm using XMLStarlet to work with XML files we process with a partnering agency. Basically, we receive an XML file that contains a base64 encoded PDF. We consume the XML, process the document, then return the document to them.

    The process I have put together via a BASH script so far was working out great. However, I just hit a roadblock. I've been testing with the smaller documents (2 page documents). Yesterday I decided to test with one of the larger documents (18 pages). When I go to insert the processed base64 into the XML file, the command bombs.

    Here is the command:

    xmlstarlet ed -L -u "/efile:EFilingData/cse:DocumentBinary/cse:BinaryObject.Base64" -v "$stampedBase64" $workingDir/$fileName
    

    Here is the error:

    /usr/bin/xmlstarlet: Argument list too long
    

    I'm pretty confident the reason is because the $stampedBase64 of an 18 page PDF is a shit-ton long, which means I'm hitting some sort of command length limit of some sort.

    Is there any way around this limitation? Is there a way I can feed it a value from a file?

    For what is' worth the script is running on a CentOS 6 VM.



  • I think I may be on to something, or I may be going down the wrong rabbit hole...but here it goes...

    If I run "xargs --show-limits", I get:

    Your environment variables take up 1953 bytes
    POSIX upper limit on argument length (this system): 26210399
    POSIX smallest allowable upper limit on argument length (all systems): 4096
    Maximum length of command we could actually use: 26208446
    Size of command buffer we are actually using: 131072
    

    When I get the length of the command when it fails, its length is 154299. This is larger than the 131072 command buffer.

    Is there a way to bump up the command buffer size?


  • Service Provider

    Isn't the command xml not xmlstarlet?

    http://xmlstar.sourceforge.net/doc/UG/xmlstarlet-ug.html


  • Service Provider

    Any chance that your command is out of order? Move the ed bit to the next to last thing, just before the file name.



  • @scottalanmiller said in XMLStarlet - Argument List Too Long:

    Isn't the command xml not xmlstarlet?

    http://xmlstar.sourceforge.net/doc/UG/xmlstarlet-ug.html

    In all the documentation I've ready, yes. However, in my installation (which was a simple yum install xmlstarlet), "xml" does not work.



  • @scottalanmiller said in XMLStarlet - Argument List Too Long:

    Any chance that your command is out of order? Move the ed bit to the next to last thing, just before the file name.

    The command structure itself is good as far as I can tell. When my script churns through 20 or so XML files, it only bombs on the larger PDFs. The smaller PDFs (2 page docs) it works great.

    This is the loop that churns through the documents that are ready to process:

    0_1492019036887_upload-1f2d8e51-dee7-4932-91b8-1d186d33b412

    Works great on the smaller PDFs (2 page docs). On the larger PDFs (18 pages) is where line 38 bombs with "Argument list too long".

    The length of the command when processing the 2 page PDFs is in the 43000 range. The length of the command when processing the 18 page PDFs is in the 150000 range which is greater than the size of the command buffer being used.


  • Service Provider

    Oh, it is READING the PDFs into the command? I see.



  • @scottalanmiller said in XMLStarlet - Argument List Too Long:

    Oh, it is READING the PDFs into the command? I see.

    Yep. Sorry for not making that clear.

    I think I may have a solution...I haven't tried it yet though. It looks like I can append a value to an element. What I'm going to try is splitting the base64 in half and doing two operations. First, I'll update the element with the first half of the base64, then I'll append the element with the second half of the base64.

    Here goes nothing...

    0_1492021309874_upload-5a40fc5c-fd56-4b82-aaba-5479e493fc1e



  • Well that was a fail. The append function was not what I thought it was. It's for elements within the XML file, not the value of a given element. Arg. Hmm...


Log in to reply
 

Looks like your connection to MangoLassi was lost, please wait while we try to reconnect.