Calling Complex System Commands from Python

scottalanmiller

Working in Python 3 it is pretty easy to make basic called to the shell such as this:

from subprocess import call
call(["free", "-m"])

That works great. That allows me to execute complex single commands like "free -m". No problem. But if I want to use pipes and other forms of shell redirection, what is the best way to do that? Is there a way to leverage the underlying shell or am I stuck handling it all via Python?

An example of what I mean...

free -m | grep rs/ | cut -d' ' -f10

Is there a simple way to run that in Python 3 without using Python constructs to handle the stdin and stdout? Doing this is BASH is, obviously, simple as those are BASH constructs.

scottalanmiller

Oh, that was easier than expected. I managed to track this down very quickly.

import subprocess
ret = subprocess.getoutput('free -m | grep rs/ | cut -d" " -f10')

scottalanmiller

Found another way, just need to tell "call" that you want to leverage the shell.

subprocess.call('free -m | grep rs/', shell=True)

Alexis

Great examples! How about something more real-world?

I use the following bash command to obtain the resolution of a movie file:

/opt/local/bin/ffmpeg -i TEST.mp4 2>&1 | sed -n "s/.*, \(.*\) \[.*/\1/p"

...and it returns a string like “640x480” for the movie’s resolution. Great!

Now, trying to do this in Python 3:

cmd = ‘/opt/local/bin/ffmpeg -i TEST.mp4 2>&1 | sed -n "s/.*, \(.*\) \[.*/\1/p"’
ret = subprocess.getoutput(cmd)

Result: null string

subprocess.call(cmd, shell=True)

Result: null string

stdoutstr = subprocess.getoutput(cmd)
print("stdoutstr:” + stdoutstr.split()[0])

Result: null string

Surely, there MUST be a way!

scottalanmiller

I will have to play with that one a bit on Tuesday when I am back.

tonyshowoff

@Alexis Did you intend on putting curly quotes there:

cmd = ‘/opt/local/bin/ffmpeg -i TEST.mp4 2>&1 | sed -n "s/.*, \(.*\) \[.*/\1/p"’

Instead of:

cmd = '/opt/local/bin/ffmpeg -i TEST.mp4 2>&1 | sed -n "s/.*, \(.*\) \[.*/\1/p"'

Alexis

[My editor changed the simple quotes into “smart” quotes for the posting, sorry.]

Anyway, the code below works to obtain the movie resolution. It returns the string “640x480” for the moviespec var I’m using:

cmdstr1 = "\"\"ffmpeg -i " + moviespec + " 2>&1 | sed -n 's/.*, \(.*\) \[.*/\\1/p'\"\""
stdoutstr1 = subprocess.check_output(
    cmdstr1,
    shell=True, universal_newlines=True)

However, in translating the following bash line to obtain the frames/second:

fps=$(/opt/local/bin/ffmpeg -i $moviespec 2>&1 | sed -n "s/.*, \(.*\) fp.*/\1/p")

The above line, translated into pythonese, returns null:

cmdstr2 = "\"\"ffmpeg -i " + moviespec + " 2>&1 | sed -n 's/.*, \(.*\) fp.*/\1/p'\"\""
stdoutstr2 = subprocess.check_output(
    cmdstr2,
    shell=True, universal_newlines=True)

I can’t spot what my error might be here.

Alexis

@Alexis said:

cmdstr2 = """ffmpeg -i " + moviespec + " 2>&1 | sed -n 's/., (.) fp.*/\1/p'"""

Okay, I forgot to escape one of the backslashes in the sed string. The proper python-fu is:

cmdstr3 = "\"\"ffmpeg -i " + moviespec + " 2>&1 | sed -n 's/.*, \(.*\) fp.*/\\1/p'\"\""

...which properly returns the FPS of the movie file.

Thanks so much for getting me started on this! I'm converting my bash scripts to python and learning the language in the process!

tonyshowoff

@Alexis Another reason I like single quotes for my strings, less escape confusion