Troubleshooting and diagnosing issues with a subprocess.Popen function call

In my previous experiences, I've had success using subprocess.Popen to wrap binaries with a Python script and customize arguments. However, when developing another wrapper recently, I encountered an issue.

Here's the snippet of code I'm working with:

print command
p = subprocess.Popen(command, shell=True)
result = p.communicate()[0]
print vars(p)
return result

After executing this code, the output is as follows:

/usr/bin/sh /tmp/run/launch.sh
{'_child_created': True, 'returncode': 0, 'stdout': None, 'stdin': None, 'pid': 21650, 'stderr': None, 'universal_newlines': False}

The objective here is to create a shell script that sets up everything required and then runs it. While I'd prefer to use actual Python code, the launch.sh file calls third-party shell scripts that I don't want to replicate (despite my attempts to get a Python API for them for over a year).

The issue at hand includes:

  • The shell script doesn't execute as expected (it should spawn a process and output something)
  • No Python exception is raised
  • There are no indications of errors in the p object

I also tried using check_call without any luck.

I'm stuck on what steps to take next, and would appreciate any guidance on where I may have made mistakes or how to resolve the situation.

EDIT:

  • I'm running this on Linux using sh
  • The shell is necessary for variable substitution in the invoked scripts

EDIT 2:

Following a suggestion from badp, I made some adjustments to the code and included

subprocess.Popen('ps', shell=True).communicate()

Right after the line where the process is created, here's the output:

/usr/bin/sh /tmp/run/launch.sh
  PID TTY          TIME CMD
29978 pts/0    00:00:01 zsh
 1178 pts/0    00:00:01 python
 1180 pts/0    00:00:00 sh <defunct>
 1181 pts/0    00:00:00 ps
None

It appears that the process is launched (even though it shows <defunct>), and there seems to be an issue with passing parameters in as well.

Thank you.

Answer №1

Give this a shot:

process = subprocess.Popen(command,
                     shell = True, #consider removing
                     stdin = subprocess.PIPE,
                     stdout = subprocess.PIPE,
                   # stderr = subprocess.STDOUT #uncomment if necessary
                    )

This has been successfully tested on Windows using the ping command. It allows for communication, which could help identify why the script isn't initiating in the first place :)

Answer №2

I've at last discovered the solution to my inquiry, all thanks to badp and his valuable debugging suggestions.

As mentioned on the python page about the subprocess module:

The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable. The only reason you would need to specify shell=True on Windows is where the command you wish to execute is actually built in to the shell, eg dir, copy. You don’t need shell=True to run a batch file, nor to run a console-based executable.

Given that I am utilizing Linux and using shell=True, my command is essentially a list of arguments to be executed by executable, which defaults to /bin/sh. Therefore, the complete command executed was:

/bin/sh /usr/bin/sh /tmp/run/launch.sh
... but unfortunately, it did not yield the desired outcome.

I should have either used:

subprocess.Popen('/tmp/run/launch.sh', shell=True)

or

subprocess.Popen('/tmp/run/launch.sh', executable = '/usr/bin/sh', shell=True)

It's interesting how shell=True would alter the default executable value specifically on Linux...

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Error occurs when using ragged tensors within the TensorFlow data pipeline

When attempting to create a tf.data pipeline using RaggedTensors in TensorFlow, I'm consistently encountering a TypeError message: "Cannot convert the argument `type_value`: RaggedTensorSpec(TensorShape(\[None, 3\]), tf.int16, 1, tf.int ...

What is the best way to display integers in a single string with spaces in between using Python?

I have created a Python function that takes two arguments - "Genome" and "Pattern" as strings. The function determines when the pattern matches the genome, it saves the starting index of the match in a list. However, I need to modify the function so that i ...

Navigating through Matrices in Python

My journey with Python has just begun, and I've encountered a roadblock when working with matrices. Here's the matrix I'm currently dealing with: A = [1 0 0 2; 3 3 3 2; 3 3 0 2; 3 4 4 4] My goal is to set all elements in the matrix to zer ...

Having difficulty printing a browser.find_element_by_xpath using Selenium in Python

I am trying to use the following code: print browser.find_element_by_xpath('/html/body/div[2]/div/div[3]/div/div/div[1]/div[2]/ul/li[4]/ul/li/span[3]/span[3]').text However, I keep getting an unexpected token error on the "browser.find_element_ ...

Using Python's Selenium library to scroll through a specific web element

I am currently attempting to scrape job offerings from a company's LinkedIn page. In order to do so, I need to scroll through a specific section of the page that has an inner scrollbar. Here is an approximation of what I have tried: 1. scroll_active ...

Having difficulty examining Celery with SQS Backend

I have been utilizing Celery for my asynchronous task processing, along with SQS on Amazon as my messaging server. All of a sudden, the tasks have ceased processing and upon inspecting the Celery queue using the following code: from celery.task.control im ...

Tips for implementing an Item loader in my Scrapy spider script?

After working on a Scrapy spider to extract news articles and data from a website, I encountered an issue with excessive whitespace in one of the items. Seeking a solution, I came across recommendations for using an Item Loader in the Scrapy documentation ...

Creating a unique Deterministic or Stochastic model in pymc3 using theano.op: A step-by-step guide

Currently working with pymc3 and interested in creating custom Stochastics. However, I'm finding limited documentation on how to accomplish this task. While I am familiar with using the as_op method, it appears that doing so renders the NUTS sampler i ...

Encountering an Issue with Python Selenium when Trying to Retrieve the 'href' Attribute

Could someone please assist me with extracting the href from a link? Here is my code: url = 'http://money.finance.sina.com.cn/bond/notice/sz149412.html' link = driver.find_element_by_xpath("//div[@class='blk01'])//ul//li[3]//a[contains( ...

Ensure that you correctly format a callable function that returns a Generator expression

Here is a code snippet that I have been working on: from contextlib import _GeneratorContextManager, contextmanager GoodWrapperType = Callable[[int, str], _GeneratorContextManager[None]] BadWrapperType = Callable[[int, str], Generator[None, None, None]] ...

Displaying the users who have seen the post

I'm in the process of developing a BlogApp and have implemented a feature that tracks the number of viewers for each post. However, I'm struggling to identify the users who have viewed a specific Post. views.py queryset = BlogPost.objects.annota ...

Steps for executing a singular GETNEXT request in PySNMP

I am currently attempting to execute a basic SNMP GETNEXT query in order to retrieve the next item of a specific OID within the tree hierarchy. Essentially, what I am looking for is: When sending a GETNEXT request with the OID 1.3.6.1.2.1.1 (iso.org.dod. ...

Unable to find element to click on "Join Now" button in Google Meet using Selenium and Python for browser automation

I am working on creating a web automation tool that can automatically join Google Meet using Selenium with the Firefox driver in Python. So far, I have successfully signed up, muted the microphone, and turned off the camera without any issues. However, I a ...

Managing Errors in Loops: Addressing the issue of loops not terminating when conditions are met using Selenium and

Developing a custom script to extract user details from AD and transfer that information into an automation tool for creating user accounts on a website. I've successfully navigated through the entire process, but encountering an issue with handling u ...

Unable to construct wheels for dependency-injector, a prerequisite for the installation of pyproject.toml-dependent projects

Seeking help with an error that keeps occurring while I attempt to install the "dependency-injector" package for my project, (env) PS C:\Multi-Participants_Survey_Project-main-2\djangosurveybackend> pip install dependency-injector Collecting d ...

I am looking for assistance with a Python web-scraping issue. I need help with scraping URLs from a webpage that has partially hidden pagination numbers. Can you lend a hand?

Looking to extract the URLs of each page from the pagination on a specific site located at . The challenge lies in the fact that not all page URLs are displayed simultaneously. When attempting to scrape using the provided link, only page-1, page-14, page- ...

Initiating the personalized profile in Chrome to run test version

When attempting to execute the code below, I keep encountering a SessionNotCreatedException error related to Driver. Does anyone have any insights on how to indicate a ChromeProfile while utilizing Chrome for testing? Error Code: Exception has occurred: ...

Writing data tables to Excel files using Python's Selenium module

I am trying to create a data table in an Excel file. The table consists of 7 rows and 1 column, with each row containing a unique value. However, when I view the Excel file, only the last row is displayed. This is how the data should appear in Excel: FT ...

"Using Python Regex in Expresso works perfectly, but unfortunately it does not work in Iron

Exploring HTML and diving into learning RegEx, even though I am aware of other approaches. Embracing challenges makes the process interesting... The regular expression I am working with is: publisher.php\?c=.*?\">(.*?)</a>(?:.*?)<br ...

"Error message: Invalid operation type(s) when subtracting a string and an integer," during the development of an age conversion tool

Can someone help figure out why this isn't working? (Click on the link for a screenshot) http://prntscr.com/5tbfos I would really appreciate it if someone could assist me in resolving this issue! Thanks so much, Ben Johnson ...