Background jobs for Python
Web applications produce slow operations that should not run in the request/response cycle: email sending, report generation, batch processing, external API calls with timeout. Standard pattern: task queue — the app enqueues a task, a separate worker executes it asynchronously.
Celery, created by Ask Solem Hoel (Norway) in 2009, is Python’s standard distributed task queue. Version 3.0 (July 2012) consolidates production maturity. BSD licence.
Architecture
[Django/Flask app] → [Broker (RabbitMQ/Redis)] → [Celery workers]
↓
[Result backend]- Broker — external message queue (RabbitMQ most solid, Redis most used, SQS for AWS)
- Workers — Python processes consuming tasks
- Result backend — optional, task result store (Redis, PostgreSQL, Cassandra)
- Beat — scheduler for periodic tasks
Task definition
from celery import Celery
app = Celery('myapp', broker='redis://localhost:6379/0')
@app.task
def send_email(to, subject, body):
# ... SMTP sending ...
return True
# Usage
send_email.delay('user@example.com', 'Welcome', 'Hi there')
# or async with parameters
result = send_email.apply_async(args=['...'], countdown=60, retry=True)Canvas (workflow primitives)
Celery provides workflow primitives:
- chain(task1, task2, task3) — sequential
- group(task1, task2, task3) — parallel
- chord(group, callback) — parallel + callback on completion
- chunks, map, starmap
Enables complex orchestration without a separate workflow engine.
Practical use
- Email/notification dispatch
- Report/export generation
- ETL/data processing
- Media transcoding
- ML inference batch
- Periodic scheduled tasks (cron-like with Celery Beat)
Alternatives
- RQ (Redis Queue) — simpler, Redis-only
- Dramatiq — modern alternative with similar API
- Huey — lightweight
- TaskIQ — asyncio-native
- Apache Airflow — heavy-duty orchestration (DAG)
In the Italian context
Celery is in the backend of almost every non-trivial Italian corporate Python application — e-commerce, PA portals, SaaS.
References: Celery 3.0 (July 2012). Ask Solem Hoel. BSD licence. Brokers: RabbitMQ, Redis, SQS. Canvas primitives: chain, group, chord. Celery Beat scheduler.