ti1

READ 3 Ways to Write a List to a File in Python

Bestseller Complete Python Bootcamp: Go from zero to hero in Python 3Learn Python like a Professional! Start from the basics and go all the way to creating your own applications and games! 4.5€194.99€10.99

Learn Python Programming MasterclassThis Python For Beginners Course Teaches You The Python Language Fast. Includes Python Online Training With Python 3 4.5€144.99€10.99

READ 3 Ways to Write a List to a File in Python

The official documentation recommends the subprocess module over the alternative os.system():

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function [os.system()].

The "Replacing Older Functions with the subprocess Module" section in the subprocess documentation may have some helpful recipes.

Older versions of Python use call:

import subprocess
subprocess.call(["ls", "-l"])

Is there a way to use variable substitution? IE I tried to do echo $PATH by using call(["echo", "$PATH"]), but it just echoed the literal string $PATH instead of doing any substitution. I know I could get the PATH environment variable, but I’m wondering if there is an easy way to have the command behave exactly as if I had executed it in bash. – Kevin Wheeler Sep 1 ’15 at 23:17
@KevinWheeler You’ll have to use shell=True for that to work. – SethMMorton Sep 2 ’15 at 20:38
@KevinWheeler You should NOT use shell=True, for this purpose Python comes with os.path.expandvars. In your case you can write: os.path.expandvars("$PATH"). @SethMMorton please reconsider your comment -> Why not to use shell=True – Murmel Nov 11 ’15 at 20:24
To simplify at least conceptually:\n call("ls -l".split()) – slehar Jun 16 ’18 at 17:15
you forgot to say it needs python 3.5 at least. It doesn’t work on python 3.4.3 for example, which is default for Ubuntu 14.04 LTS – pulse Feb 2 at 22:50

Here’s a summary of the ways to call external programs and the advantages and disadvantages of each:

os.system("some_command with args") passes the command and arguments to your system’s shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example:
```
os.system("some_command < input_file | another_command > output_file")  
```
However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs. See the documentation.
stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don’t need to worry about escaping anything. See the documentation.
The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you’d say:
```
print subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE).stdout.read()
```
instead of:
```
print os.popen("echo Hello World").read()
```
but it is nice to have all of the options there in one unified class instead of 4 different popen functions. See the documentation.
The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply waits until the command completes and gives you the return code. For example:
```
return_code = subprocess.call("echo Hello World", shell=True)  
```
See the documentation.
If you’re on Python 3.5 or later, you can use the new subprocess.run function, which is a lot like the above but even more flexible and returns a CompletedProcess object when the command finishes executing.
The os module also has all of the fork/exec/spawn functions that you’d have in a C program, but I don’t recommend using them directly.

The subprocess module should probably be what you use.

Finally please be aware that for all methods where you pass the final command to be executed by the shell as a string and you are responsible for escaping it. There are serious security implications if any part of the string that you pass can not be fully trusted. For example, if a user is entering some/any part of the string. If you are unsure, only use these methods with constants. To give you a hint of the implications consider this code:

print subprocess.Popen("echo %s " % user_input, stdout=PIPE).stdout.read()

and imagine that the user enters something "my mama didnt love me && rm -rf /" which could erase the whole filesystem.

Nice answer/explanation. How is this answer justifying Python’s motto as described in this article ? fastcompany.com/3026446/… "Stylistically, Perl and Python have different philosophies. Perl’s best known mottos is " There’s More Than One Way to Do It". Python is designed to have one obvious way to do it" Seem like it should be the other way! In Perl I know only two ways to execute a command - using back-tick or open. – Jean May 26 ’15 at 21:16
If using Python 3.5+, use subprocess.run(). docs.python.org/3.5/library/subprocess.html#subprocess.run – phoenix Oct 7 ’15 at 16:37
What one typically needs to know is what is done with the child process’s STDOUT and STDERR, because if they are ignored, under some (quite common) conditions, eventually the child process will issue a system call to write to STDOUT (STDERR too?) that would exceed the output buffer provided for the process by the OS, and the OS will cause it to block until some process reads from that buffer. So, with the currently recommended ways, subprocess.run(..), what exactly does "This does not capture stdout or stderr by default." imply? What about subprocess.check_output(..) and STDERR? – Evgeni Sergeev Jun 1 ’16 at 10:44
@Pitto yes, but that is not what gets executed by the example. Notice the echo in front of the string passed to Popen? So the full command will be echo my mama didnt love me && rm -rf /. – Chris Arndt Sep 10 ’18 at 16:38
This is arguably the wrong way around. Most people only need subprocess.run() or its older siblings subprocess.check_call() et al. For cases where these do not suffice, see subprocess.Popen(). os.popen() should perhaps not be mentioned at all, or come even after "hack your own fork/exec/spawn code". – tripleee Dec 3 ’18 at 6:00

I typically use:

import subprocess

p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
    print line,
retval = p.wait()

You are free to do what you want with the stdout data in the pipe. In fact, you can simply omit those parameters (stdout= and stderr=) and it’ll behave like os.system().

share improve this answer

.readlines() reads all lines at once i.e., it blocks until the subprocess exits (closes its end of the pipe). To read in real time (if there is no buffering issues) you could: for line in iter(p.stdout.readline, ''): print line, – jfs Nov 16 ’12 at 14:12
Could you elaborate on what you mean by "if there is no buffering issues"? If the process blocks definitely, the subprocess call also blocks. The same could happen with my original example as well. What else could happen with respect to buffering? – EmmEff Nov 17 ’12 at 13:25
the child process may use block-buffering in non-interactive mode instead of line-buffering so p.stdout.readline() (note: no s at the end) won’t see any data until the child fills its buffer. If the child doesn’t produce much data then the output won’t be in real time. See the second reason in Q: Why not just use a pipe (popen())?. Some workarounds are provided in this answer (pexpect, pty, stdbuf) – jfs Nov 17 ’12 at 13:51
the buffering issue only matters if you want output in real time and doesn’t apply to your code that doesn’t print anything until all data is received – jfs Nov 17 ’12 at 13:53
This answer was fine for its time, but we should no longer recommend Popen for simple tasks. This also needlessly specifies shell=True. Try one of the subprocess.run() answers. – tripleee Dec 3 ’18 at 5:39

Some hints on detaching the child process from the calling one (starting the child process in background).

Suppose you want to start a long task from a CGI-script, that is the child process should live longer than the CGI-script execution process.

The classical example from the subprocess module docs is:

import subprocess
import sys

# some code here

pid = subprocess.Popen([sys.executable, "longtask.py"]) # call subprocess

# some more code here

The idea here is that you do not want to wait in the line ‘call subprocess’ until the longtask.py is finished. But it is not clear what happens after the line ‘some more code here’ from the example.

My target platform was freebsd, but the development was on windows, so I faced the problem on windows first.

On windows (win xp), the parent process will not finish until the longtask.py has finished its work. It is not what you want in CGI-script. The problem is not specific to Python, in PHP community the problems are the same.

The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in win API. If you happen to have installed pywin32 you can import the flag from the win32process module, otherwise you should define it yourself:

DETACHED_PROCESS = 0x00000008

pid = subprocess.Popen([sys.executable, "longtask.py"],
                       creationflags=DETACHED_PROCESS).pid

/* UPD 2015.10.27 @eryksun in a comment below notes, that the semantically correct flag is CREATE_NEW_CONSOLE (0×00000010) */

On freebsd we have another problem: when the parent process is finished, it finishes the child processes as well. And that is not what you want in CGI-script either. Some experiments showed that the problem seemed to be in sharing sys.stdout. And the working solution was the following:

pid = subprocess.Popen([sys.executable, "longtask.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)

I have not checked the code on other platforms and do not know the reasons of the behaviour on freebsd. If anyone knows, please share your ideas. Googling on starting background processes in Python does not shed any light yet.

share improve this answer

i noticed a possible "quirk" with developing py2exe apps in pydev+eclipse. i was able to tell that the main script was not detached because eclipse’s output window was not terminating; even if the script executes to completion it is still waiting for returns. but, when i tried compiling to a py2exe executable, the expected behavior occurs (runs the processes as detached, then quits). i am not sure, but the executable name is not in the process list anymore. this works for all approaches (os.system("start *"), os.spawnl with os.P_DETACH, subprocs, etc.) – maranas Apr 9 ’10 at 8:09

you might also need CREATE_NEW_PROCESS_GROUP flag. See Popen waiting for child process even when the immediate child has terminated
The following is incorrect: "[o]n windows (win xp), the parent process will not finish until the longtask.py has finished its work". The parent will exit normally, but the console window (conhost.exe instance) only closes when the last attached process exits, and the child may have inherited the parent’s console. Setting DETACHED_PROCESS in creationflags avoids this by preventing the child from inheriting or creating a console. If you instead want a new console, use CREATE_NEW_CONSOLE (0×00000010).
1

I didn’t mean that executing as a detached process is incorrect. That said, you may need to set the standard handles to files, pipes, or os.devnull because some console programs exit with an error otherwise. Create a new console when you want the child process to interact with the user concurrently with the parent process. It would be confusing to try to do both in a single window.
1

is there not an OS-agnostic way to have the process run in the background?

I’d recommend using the subprocess module instead of os.system because it does shell escaping for you and is therefore much safer: http://docs.python.org/library/subprocess.html

subprocess.call(['ping', 'localhost'])

If you want to create a list out of a command with parameters, a list which can be used with subprocess when shell=False, then use shlex.split for an easy way to do this docs.python.org/2/library/shlex.html#shlex.split (it’s the recommended way according to the docs docs.python.org/2/library/subprocess.html#popen-constructor) – Daniel F Sep 20 ’18 at 18:07This is incorrect: "it does shell escaping for you and is therefore much safer". subprocess doesn’t do shell escaping, subprocess doesn’t pass your command through the shell, so there’s no need to shell escape. – Lie Ryan Dec 4 ’18 at 8:36

import os
cmd = 'ls -al'
os.system(cmd)

If you want to return the results of the command, you can use os.popen. However, this is deprecated since version 2.6 in favor of the subprocess module, which other answers have covered well.

popen is deprecated in favor of subprocess. – Fox Wilson Aug 8 ’14 at 0:22
You can also save your result with the os.system call, since it works like the UNIX shell itself, like for example os.system(‘ls -l > test2.txt’) – Stefan Gruenwald Nov 7 ’17 at 23:19

import os
os.system("your command")

Note that this is dangerous, since the command isn’t cleaned. I leave it up to you to google for the relevant documentation on the ‘os’ and ‘sys’ modules. There are a bunch of functions (exec* and spawn*) that will do similar things.

—

ti1█

plus un memo qu’un tuto

PYTHON > exécuter des commandes système

Python System Command

Python os.system() function

Python subprocess.call() Function

Python subprocess.check_output() function

Execute Shell command in Python with os module

Execute shell command in Python with subprocess module

Aucun commentaire

Laissez un commentaire

Tutos

Travaux

De la même plume :