This page provides a high-level introduction to Celery, a distributed task queue system for Python. It covers the fundamental concepts, system architecture, and main components that form the Celery ecosystem. For installation instructions, see Installation and Setup. For detailed information about the core architecture components, see Core Architecture.
Celery is a distributed task queue system written in Python that enables asynchronous task execution across multiple workers and machines. It uses message-passing via a broker to distribute work units (tasks) to dedicated worker processes.
Key characteristics:
Sources: README.rst1-119 docs/includes/introduction.txt10-36
A task queue is a mechanism to distribute work across threads or machines. The input is a unit of work called a task, and dedicated worker processes constantly monitor queues for new work to perform README.rst94-98
Tasks are the units of work in Celery. They are Python functions decorated with @app.task that can be executed asynchronously. Tasks are defined in application code and registered with a Celery application instance.
Sources: README.rst186-196 celery/__init__.py29-34
Brokers mediate message passing between clients (task producers) and workers (task consumers). When a client initiates a task, it puts a message on a queue; the broker then delivers that message to a worker README.rst100-102
Supported brokers:
Sources: README.rst145-148 README.rst219-221
Workers are processes that execute tasks. They continuously monitor queues for new work, execute the task code, and optionally store results. Workers support multiple concurrency models: prefork (multiprocessing), eventlet, gevent, thread, and solo README.rst224-225
Result backends store the return values and state of executed tasks. Celery supports various backends including Redis, databases (SQLAlchemy, Django ORM), AMQP, Memcached, and cloud storage services README.rst228-234
Sources: README.rst216-234
Celery System Architecture: This diagram shows the main components of a Celery system and how they interact. Tasks are defined in the client application, sent through a message broker, processed by worker pools, with results stored in a backend. Beat independently schedules periodic tasks.
Sources: Diagram 1 from provided context, celery/app/base.py230-265 celery/__init__.py1
The Celery class (celery/app/base.py230-265) is the central entry point for using Celery. It coordinates all parts of the system including configuration, task registration, broker connections, and worker management.
Key responsibilities:
app.conf (celery/app/base.py377-382)@app.task decorator (celery/app/base.py489-559)For detailed information about the Celery application, see The Celery Application.
Sources: celery/app/base.py230-432 celery/__init__.py29-34
Tasks are callable units of work defined using the @app.task decorator. The Task class (celery/app/task.py164-171) provides methods for execution (run), retries (retry), and asynchronous invocation (apply_async, delay).
Canvas primitives enable workflow composition:
signature: Serializable task invocation (celery/canvas.py232-287)chain: Sequential execution (A → B → C)group: Parallel executionchord: Group with callback (parallel tasks → single callback)chunks: Splits tasks into chunksFor comprehensive task documentation, see Tasks. For workflow composition, see Canvas Workflows.
Sources: celery/app/task.py164-627 celery/canvas.py1-40
The AMQP layer (celery/app/amqp.py219-220) handles message serialization, routing, and broker communication. It creates task messages using the v1 or v2 protocol and manages producer connection pools.
Key classes:
AMQP: AMQP API for the app (celery/app/amqp.py219-220)Queues: Queue name to declaration mapping (celery/app/amqp.py41-58)Router: Routes tasks to queues based on configurationTask Message Creation Flow: Shows how task invocation flows through the AMQP layer to create and send messages to the broker.
For broker configuration details, see Message Brokers and Routing. For the AMQP protocol, see AMQP Layer and Message Protocol.
Sources: celery/app/amqp.py1-21 celery/app/amqp.py219-320
Workers are processes that consume and execute tasks. The WorkController (celery/worker/__init__.py) orchestrates components using the bootstep pattern, where components have dependencies and initialize in order.
Worker component hierarchy:
Timer: Internal schedulingHub: Event loop (kombu.asynchronous)Pool: Task execution (prefork/eventlet/gevent/thread/solo)Consumer: Message consumption from brokerBeat: Optional embedded schedulerStateDB: Persistent state storageConcurrency models (README.rst224-225):
For worker architecture details, see Workers.
Sources: celery/apps/worker.py celery/worker/__init__.py README.rst224-225
Result backends store task states and return values. The BaseBackend (celery/backends/base.py) defines the interface, with implementations for various storage systems.
Backend types:
The AsyncResult class (celery/result.py) provides the client-side API to retrieve task results:
For backend configuration and usage, see Result Backends.
Sources: celery/backends/base.py celery/result.py README.rst228-234
Celery Beat (celery/beat.py) is a separate process that sends tasks to the broker on a schedule. It runs independently of workers.
Schedule types:
schedule: Fixed time intervals (celery/schedules.py)crontab: Cron-like expressions (minute/hour/day/month patterns)solar: Astronomical events (sunrise/sunset)Storage backends:
PersistentScheduler: Python shelve database (default)django-celery-beat: Django database with admin UIFor detailed Beat documentation, see Periodic Tasks (Beat).
Sources: celery/beat.py celery/schedules.py celery/apps/beat.py
The celery command (celery/bin/celery.py) provides a comprehensive CLI built with Click for managing workers, monitoring, and administrative tasks.
Key commands:
celery worker: Start worker processescelery beat: Start periodic task schedulercelery inspect: Query worker state (active tasks, stats, registered tasks)celery control: Send control commands (rate limits, revoke tasks, shutdown)celery events: Monitor event streamcelery purge: Clear all queuesFor CLI documentation, see Command Line Interface.
Sources: celery/bin/celery.py README.rst454-487
Task Execution Flow: This sequence diagram shows the complete lifecycle of a task from submission by the client through execution by a worker to result storage and retrieval.
Task states (celery/app/task.py celery/states.py):
PENDING: Task is waiting to be executedSTARTED: Task has been started (if track_started=True)SUCCESS: Task completed successfullyFAILURE: Task raised an exceptionRETRY: Task is being retriedREVOKED: Task was cancelled/revokedSources: celery/app/task.py446-627 celery/app/amqp.py320-412 celery/result.py
Celery applications are configured through the app.conf object, which can load settings from:
Example configuration:
Configuration sources (celery/app/base.py677-722):
app.config_from_object('module.path'): Load from Python moduleapp.config_from_envvar('CELERY_CONFIG_MODULE'): Load from environment variableFor installation and initial setup, see Installation and Setup. For application configuration details, see Application Initialization and Configuration.
Sources: celery/app/base.py677-722 celery/app/defaults.py
Current stable version: 5.6.0 (recovery) celery/__init__.py18-20
Python version requirements (README.rst123-126):
Minimum versions for older Python:
Sources: celery/__init__.py18-20 README.rst123-140
Celery is designed for:
Sources: README.rst1-11 docs/getting-started/introduction.rst10-33
Celery integrates with popular web frameworks README.rst248-268:
| Framework | Integration | Notes |
|---|---|---|
| Django | Built-in | No extra package needed |
| Flask | Built-in | No extra package needed |
| FastAPI | Built-in | No extra package needed |
| Pyramid | pyramid_celery | Integration package |
| Tornado | tornado-celery | Integration package |
Integration packages provide framework-specific features like database connection management at fork and configuration loading.
Sources: README.rst248-285
Development happens on GitHub with contributions welcome. See the Contributing to Celery guide in the documentation.
Sources: README.rst478-537 docs/includes/resources.txt1-56
Note: This overview provides a high-level introduction to Celery. For detailed information on specific components, refer to the linked sections in the table of contents. The architecture is highly modular and configurable, allowing customization of almost every aspect of the system.
Refresh this wiki
This wiki was recently refreshed. Please wait 2 days to refresh again.