I am facing an issue with a celery task where the soft limit is set at 10 and the hard limit at 32:
from celery.exceptions import SoftTimeLimitExceeded
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
@app.task(bind=True, acks_late=False, time_limit=32, soft_time_limit=10)
def my_task(self, **kwargs):
try:
if 'twisted.internet.reactor' in sys.modules:
del sys.modules['twisted.internet.reactor']
settings = get_project_settings()
process = CrawlerProcess(settings)
process.crawl(**kwargs)
process.start()
except SoftTimeLimitExceeded as te:
print('Time Exceeded...')
The code mentioned above runs correctly. However, when the crawl operation exceeds the soft limit, no exception is triggered. The crawl operation continues until the hard limit is reached, causing this error to be displayed:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/billiard/pool.py", line 684, in on_hard_timeout
raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(32,)
I have tried catching this error within the task but was unsuccessful. To test, I replaced the process.start() command with time.sleep(50) to create a delay without starting any crawl operation:
@app.task(bind=True, acks_late=False, time_limit=32, soft_time_limit=10)
def my_task(self, **kwargs):
try:
if 'twisted.internet.reactor' in sys.modules:
del sys.modules['twisted.internet.reactor']
settings = get_project_settings()
process = CrawlerProcess(settings)
process.crawl(**kwargs)
time.sleep(50)
except SoftTimeLimitExceeded as te:
print('Time Exceeded...')
The catch occurs for SoftTimeLimitExceeded. What could be the reason for this?
Versions
celery==5.2.7
Scrapy==2.6.1