XMLStarlet - Argument List Too Long

[anthonyh]

Hey All,

I'm using XMLStarlet to work with XML files we process with a partnering agency. Basically, we receive an XML file that contains a base64 encoded PDF. We consume the XML, process the document, then return the document to them.

The process I have put together via a BASH script so far was working out great. However, I just hit a roadblock. I've been testing with the smaller documents (2 page documents). Yesterday I decided to test with one of the larger documents (18 pages). When I go to insert the processed base64 into the XML file, the command bombs.

Here is the command:

xmlstarlet ed -L -u "/efile:EFilingData/cse:DocumentBinary/cse:BinaryObject.Base64" -v "$stampedBase64" $workingDir/$fileName

Here is the error:

/usr/bin/xmlstarlet: Argument list too long

I'm pretty confident the reason is because the $stampedBase64 of an 18 page PDF is a shit-ton long, which means I'm hitting some sort of command length limit of some sort.

Is there any way around this limitation? Is there a way I can feed it a value from a file?

For what is' worth the script is running on a CentOS 6 VM.

[anthonyh]

I think I may be on to something, or I may be going down the wrong rabbit hole...but here it goes...

If I run "xargs --show-limits", I get:

Your environment variables take up 1953 bytes
POSIX upper limit on argument length (this system): 26210399
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 26208446
Size of command buffer we are actually using: 131072

When I get the length of the command when it fails, its length is 154299. This is larger than the 131072 command buffer.

Is there a way to bump up the command buffer size?

[scottalanmiller]

Isn't the command xml not xmlstarlet?

http://xmlstar.sourceforge.net/doc/UG/xmlstarlet-ug.html

[scottalanmiller]

Any chance that your command is out of order? Move the ed bit to the next to last thing, just before the file name.

[anthonyh]

@scottalanmiller said in XMLStarlet - Argument List Too Long:

Isn't the command xml not xmlstarlet?

http://xmlstar.sourceforge.net/doc/UG/xmlstarlet-ug.html

In all the documentation I've ready, yes. However, in my installation (which was a simple yum install xmlstarlet), "xml" does not work.

[anthonyh]

@scottalanmiller said in XMLStarlet - Argument List Too Long:

Any chance that your command is out of order? Move the ed bit to the next to last thing, just before the file name.

The command structure itself is good as far as I can tell. When my script churns through 20 or so XML files, it only bombs on the larger PDFs. The smaller PDFs (2 page docs) it works great.

This is the loop that churns through the documents that are ready to process:

Works great on the smaller PDFs (2 page docs). On the larger PDFs (18 pages) is where line 38 bombs with "Argument list too long".

The length of the command when processing the 2 page PDFs is in the 43000 range. The length of the command when processing the 18 page PDFs is in the 150000 range which is greater than the size of the command buffer being used.

[scottalanmiller]

Oh, it is READING the PDFs into the command? I see.

[anthonyh]

@scottalanmiller said in XMLStarlet - Argument List Too Long:

Oh, it is READING the PDFs into the command? I see.

Yep. Sorry for not making that clear.

I think I may have a solution...I haven't tried it yet though. It looks like I can append a value to an element. What I'm going to try is splitting the base64 in half and doing two operations. First, I'll update the element with the first half of the base64, then I'll append the element with the second half of the base64.

Here goes nothing...

[anthonyh]

Well that was a fail. The append function was not what I thought it was. It's for elements within the XML file, not the value of a given element. Arg. Hmm...