Python Pipes to Avoid Shells¶
You should take a look at the shell injection document before this one.
A lot of the time, our codebase uses
shell=True because it’s
convenient. The shell provides the ability to pipe things around
without buffering them in memory, and allows a malicious user to chain
additional commands after a legitimate command is run.
Here is a simple function that uses curl to grab a page from a website, and
pipe it directly to the
wordcount program to tell us how many
lines there are in the HTML source code.
def count_lines(website): return subprocess.check_output('curl %s | wc -l' % website, shell=True) #>>> count_lines('www.google.com') #'7\n'
(That output is correct, by the way - the google html source does have 7 lines.)
The function is insecure because it uses
shell=True, which allows
shell injection. A user to who instructs your
code to fetch the website
; rm -rf / can do terrible things to what
used to be your machine.
If we convert the function to use
shell=False, it doesn’t work.
def count_lines(website): args = ['curl', website, '|', 'wc', '-l'] return subprocess.check_output(args, shell=False) # >>> count_lines('www.google.com') # curl: (6) Could not resolve host: | # curl: (6) Could not resolve host: wc # Traceback (most recent call last): # File "<stdin>", line 3, in count_lines # File "/usr/lib/python2.7/subprocess.py", line 573, in check_output # raise CalledProcessError(retcode, cmd, output=output) # subprocess.CalledProcessError: Command # '['curl', 'www.google.com', '|', 'wc', '-l']' returned non-zero exit status 6
The pipe doesn’t mean anything special when shell=False, and so curl tries to download the website called ‘|’. This does not fix the issue, rather it causes it to be more broken than before.
If we can’t rely on pipes if we have shell=False, how should we do this?
def count_lines(website): args = ['curl', website] args2 = ['wc', '-l'] process_curl = subprocess.Popen(args, stdout=subprocess.PIPE, shell=False) process_wc = subprocess.Popen(args2, stdin=process_curl.stdout, stdout=subprocess.PIPE, shell=False) # Allow process_curl to receive a SIGPIPE if process_wc exits. process_curl.stdout.close() return process_wc.communicate() # >>> count_lines('www.google.com') # '7\n'
Rather than calling a single shell process that runs each of our
programs, we run them separately and connect stdout from curl to stdin
for wc. We specify
stdout=subprocess.PIPE, which tells subprocess
to send that output to the respective file handler.
Treat pipes like file descriptors (you can actually use FDs if you want)
they may block on reading and writing if nothing is connected to the
other end. That’s why we use
communicate(), which reads until EOF
on the output and then waits for the process to terminate. You should
generally avoid reading and writing to pipes directly unless you
really know what you’re doing - it’s easy to work yourself into a situation
that can deadlock.
communicate() buffers the result in memory - if that’s not
what you want, use a file descriptor for
stdout to pipe that output
into a file.
Using pipes helps you avoid shell injection in more complex situations
It’s possible to deadlock things with pipes (in Python or in shell)