Questions tagged [celery]
Celery stands out as a renowned Python framework designed for handling distributed task queues. This versatile tool excels in enabling asynchronous and parallel execution, showcasing its efficiency in various applications.
Scenario This question revolves around my usage of celery and rabbitmq to develop a distributed messaging application using an HDFStore that transfers pandas DataFrames to distributed processes (then writes to the HDFStore). Given that json is one of the ...
Two of my Python applications are utilizing Celery and connected to the same broker. Instance A contains all of my @tasks, but I need to run these tasks from Instance B. Unfortunately, I cannot perform standard imports as the tasks do not exist on Instanc ...
My Flask Celery app instantiates the celery instance. I'm aware that I can add a normal Flask route to the same .py file, but would need to run the code twice: To run the worker: % celery worker -A app.celery ... To run the code as a normal ...
I have been utilizing Celery for my asynchronous task processing, along with SQS on Amazon as my messaging server. All of a sudden, the tasks have ceased processing and upon inspecting the Celery queue using the following code: from celery.task.control im ...
I am currently working with a Django 1.11.5 application and Celery 4.1.0, but I keep encountering the following error: kombu.exceptions.EncodeError: <User: testuser> is not JSON serializable Here are my settings in settings.py: CELERY_BROKER_URL = ...
Check out the Github repository for reproducing this issue Run localhost:3000 to replicate the problem. In my setup with Meteor 1.4.4.1, I am utilizing the node-celery npm packages on the server side. Upon Meteor initialization, the client automatically i ...
I'm currently tackling a new project where each developer has their own settings file to work with. When it comes to running Django, I have to execute the following command: python manage.py runserver --settings="databank_web.settings.dqs.dev_houman ...
For my Django project, I needed to run long tasks so I decided to use Celery with Redis as the broker. After installing Redis, it runs smoothly: The server is now ready to accept connections on port 6379 Next, I installed django-celery and configured i ...
I am facing an issue with a celery task where the soft limit is set at 10 and the hard limit at 32: from celery.exceptions import SoftTimeLimitExceeded from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings @app.ta ...
New to Python Django and Celery, I'm looking to set up Celery locally. Right now, my focus is on configuring error emails for failed tasks. Here's what I've done so far: Added the following code to setting.py CELERY_SEND_TASK_ERROR_EMAILS ...
There is a dilemma I'm facing with my small script that enqueues tasks for processing. It performs numerous database queries to obtain the items that need to be enqueued. The problem arises when the celery workers immediately start picking up the tasks as ...
I am in need of a specific workflow that involves running a ParentTask first, followed by spawning N instances of ChildTask to run in parallel. The ParentTask then waits for the ChildTasks to finish, collects and processes the results before completing the ...
In my Django + Celery 3.1.25 application, I have imported a Python module with C++ code using pybind11. This C++ code contains assertions that may trigger within a Celery worker and result in a WorkerLostError. Despite attempting to enclose the calls to t ...
Can a list of tasks be accessed by only using a group identifier? from celery import group def f1(): task_group = group(task1, task2) return task_group().id def f2(group_id): pass # TODO: retrieve task1.id and task2.id GroupResult(id=f1( ...
I'm having trouble figuring out how to create a file in Python and then pass a reference to the file for converting it into different image resolutions. I am using Celery to asynchronously generate these various image resolutions, but passing the entire im ...
Currently, I am utilizing selenium-python in combination with PhantomJS. The code structure is as follows: from selenium.webdriver import PhantomJS from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from se ...
Recently, I created a Flask web application that utilizes Celery for task handling. Within this app, one of the tasks involves scraping approximately 200 pages using a custom Class derived from a selenium chrome driver. @celery_app.task def scrape_async(): ...