Skip to content

Releases: celery/celery

v5.6.0

30 Nov 17:40
cca1116

Choose a tag to compare

Celery v5.6.0 is now available.

Key Highlights

See What's new in Celery 5.6 for a complete overview or read the main highlights below.

Python 3.9 Minimum Version

Celery 5.6.0 drops support for Python 3.8 (EOL). The minimum required Python version is now 3.9. Users still on Python 3.8 must upgrade their Python version before upgrading to Celery 5.6.0.

Additionally, this release includes initial support for Python 3.14.

SQS: Reverted to pycurl from urllib3

The switch from pycurl to urllib3 for the SQS transport (introduced in Celery 5.5.0 via Kombu) has been reverted due to critical issues affecting SQS users.

Contributed by @auvipy in #9620.

Security Fix: Broker Credential Leak Prevention

Fixed a security issue where broker URLs containing passwords were being logged in plaintext by the delayed delivery mechanism. Broker credentials are now properly sanitized in all log output.

Contributed by @giancarloromeo in #9997.

Memory Leak Fixes

Two significant memory leaks have been fixed in this release:

Exception Handling Memory Leak: Fixed a critical memory leak in task exception handling that was particularly severe on Python 3.11+ due to enhanced traceback data. The fix properly breaks reference cycles in tracebacks to allow garbage collection.

Contributed by @jaiganeshs21 in #9799.

Pending Result Memory Leak: Fixed a memory leak where AsyncResult subscriptions were not being cleaned up when results were forgotten.

Contributed by @tsoos99dev in #9806.

ETA Task Memory Limit

New configuration option worker_eta_task_limit to prevent out-of-memory crashes when workers fetch large numbers of ETA or countdown tasks. Previously, workers could exhaust available memory when the broker contained many scheduled tasks.

Example usage:

app.conf.worker_eta_task_limit = 1000

Contributed by @sashu2310 in #9853.

Queue Type Selection for Auto-created Queues

New configuration options allow specifying the queue type and exchange type when Celery auto-creates missing queues. This is particularly useful for RabbitMQ users who want to use quorum queues with auto-created queues.

Configuration options:

  • task_create_missing_queue_type: Sets the queue type for auto-created queues (e.g., quorum, classic)
  • task_create_missing_queue_exchange_type: Sets the exchange type for auto-created queues

Example usage:

app.conf.task_create_missing_queue_type = 'quorum'

Contributed by @ghirailghiro in #9815.

What's Changed

Full Changelog: v5.6.0rc2...v5.6.0

v5.6.0rc2

22 Nov 17:54
b446910

Choose a tag to compare

v5.6.0rc2 Pre-release
Pre-release

Celery v5.6.0 Release Candidate 2 is now available for testing.
Please help us test this version and report any issues.

What's Changed

  • Remove Python 4.0 version condition for pytest dependencies by @ulgens in #9993
  • Sanitize broker URL in delayed delivery logs (avoid leaking credentials) by @giancarloromeo in #9997
  • Don't fail task on timeout during cold shutdown by @Nusnus in #9678
  • Add Py39-314t to CI by @Nusnus in #9999
  • asynpool: Don't return from inside a finally block by @torarvid in #10000
  • Prepare for (pre) release: v5.6.0rc2 by @Nusnus in #10005

New Contributors

Full Changelog: v5.6.0rc1...v5.6.0rc2

v5.6.0rc1

01 Nov 23:53
5fd0b3e

Choose a tag to compare

v5.6.0rc1 Pre-release
Pre-release

Celery v5.6.0 Release Candidate 1 is now available for testing.
Please help us test this version and report any issues.

What's Changed

  • Add support for Django Connection pool by @petrprikryl in #9953
  • Pin tblib to ==3.1.0 by @kernelshard in #9967
  • fix(worker): continue to attempt to bind other queues after a native delayed delivery binding failure has occurred by @Izzette in #9959
  • Handle UnpicklingError in persistent scheduler initialization by @pakallis in #9952
  • Bug Fix: Nested Chords Fail When Using django-celery-results with a Redis Backend by @kumuthu53 in #9950
  • Add support pymongo 4.12 by @kidoz in #9665
  • Make tests compatible with pymongo >= 4.14 by @cjwatson in #9968
  • tblib updated from 3.1.0 to 3.2.0 by @MehrazRumman in #9970
  • Fix remaining function typing and docstring by @wcislo-saleor in #9971
  • Fix regex pattern in version parsing and remove duplicate entry in all by @sagar1343 in #9978
  • Bump Kombu to v5.6.0 and removed <5.7 limit on kombu by @Nusnus in #9981
  • Prepare for (pre) release: v5.6.0rc1 by @Nusnus in #9982

New Contributors

Full Changelog: v5.6.0b2...v5.6.0rc1

v5.6.0b2

20 Oct 08:49
2731860

Choose a tag to compare

v5.6.0b2 Pre-release
Pre-release

Celery v5.6.0 Beta 2 is now available for testing.
Please help us test this version and report any issues.

What's Changed

New Contributors

Full Changelog: v5.6.0b1...v5.6.0b2

v5.6.0b1

15 Sep 00:46
7c75fa7

Choose a tag to compare

v5.6.0b1 Pre-release
Pre-release

Celery v5.6.0 Beta 1 is now available for testing.
Please help us test this version and report any issues.

What's Changed

  • docs: mention of json serializer recursive reference message size blowup (#5000) by @chrisbarber in #9743
  • docs: typo in canvas.rst by @chrisbarber in #9744
  • Makes _on_retry return a float as required to be used as errback on retry_over_time by @maccinza in #9741
  • Update canvas.rst doc calculation order for callback by @auvipy in #9758
  • Updated Blacksmith logo by @Nusnus in #9763
  • Made the Sponsors logos link to their website by @Nusnus in #9764
  • add missing cloudamqp logo by @auvipy in #9767
  • Improve sponsor visibility by @auvipy in #9768
  • fix: (#9773) task_id must not be empty with chain as body of a chord by @dfm88 in #9774
  • Update setup.py to fix deprecation warning by @auvipy in #9771
  • Adds integration test for chord_unlock bug when routed to quorum/topic queue by @maccinza in #9766
  • Add xfail test for default queue/exchange fallback ignoring task_default_* settings by @maccinza in #9765
  • Add xfail test for RabbitMQ quorum queue global QoS race condition by @maccinza in #9770
  • fix: (#8786) time out when chord header fails with group body by @dfm88 in #9788
  • Fix #9738 : Add root_id and parent_id to .apply() by @dfm88 in #9784
  • Replace DelayedDelivery connection creation to use context manger by @bityob in #9793
  • Fix #9794: Pydantic integration fails with future.annotations. by @davidt in #9795
  • add go and rust implementation in docs by @darixsamani in #9800
  • Fix memory leak in exception handling (Issue #8882) by @jaiganeshs21 in #9799
  • Fix handlers docs (Issue #9787) by @jaiganeshs21 in #9804
  • Remove importlib_metadata leftovers by @cjwatson in #9791
  • Update timeout minutes for smoke tests CI by @auvipy in #9807
  • Revert "Remove dependency on pycurl" by @auvipy in #9620
  • Add Blacksmith Docker layer caching to all Docker builds by @Nusnus in #9840
  • Bump Kombu to v5.6.0b1 by @Nusnus in #9839
  • Disable pytest-xdist for smoke tests and increase retries (CI ONLY) by @Nusnus in #9842
  • Fix Python 3.13 compatibility in events dumper by @Nusnus in #9826
  • Dockerfile Build Optimizations by @Nusnus in #9733
  • Migrated from useblacksmith/build-push-action@v1 to useblacksmith/setup-docker-builder@v1 in the CI by @Nusnus in #9846
  • Remove incorrect example by @YuppY in #9854
  • Revert "Use Django DB max age connection setting" by @danlamanna in #9824
  • Fix pending_result memory leak by @tsoos99dev in #9806
  • Update python-package.yml by @auvipy in #9856
  • Bump actions/checkout from 4 to 5 by @dependabot[bot] in #9857
  • Bump Kombu to v5.6.0b2 by @Nusnus in #9858
  • Refactor integration and smoke tests CI by @Nusnus in #9855
  • Fix AsyncResult.forget() with couchdb backend method raises TypeError: a bytes-like object is required, not 'str' by @temaxuck in #9865
  • Improve Docs for SQS Authentication by @BlaiseMuhirwa in #9868
  • Added .github/copilot-instructions.md for GitHub Copilot by @Nusnus in #9874
  • misc: credit removal by @0asys in #9877
  • Choose queue type and exchange type when creating missing queues (fix #9671) by @ghirailghiro in #9815
  • fix: prevent celery from hanging due to spawned greenlet errors in greenlet drainers by @linusphan in #9371
  • Feature/disable prefetch fixes by @Dkhodos in #9863
  • Add worker_eta_task_limit configuration to manage ETA task memory usage by @sashu2310 in #9853
  • Update runner version in Docker workflow by @auvipy in #9884
  • Prepare for (pre) release: v5.6.0b1 by @Nusnus in #9890

New Contributors

Full Changelog: v5.5.3...v5.6.0b1

v5.5.3

01 Jun 11:08
088c39c

Choose a tag to compare

What's Changed

  • make the tests run on python 3.13 for gcs backend by @auvipy in #9677
  • Added DeepWiki to README by @Nusnus in #9683
  • Limit redis to <=v5.2.1 to match Kombu by @Nusnus in #9693
  • Use EX_OK instead of literal zero by @Nusnus in #9684
  • Make wheel metadata reproducible by @cjwatson in #9687
  • let celery install from kombu dependencies for better align by @auvipy in #9696
  • Fix stamping documentation to clarify stamped_headers key is optional in visitor methods by @Nusnus in #9697
  • Support apply_async without queue argument on quorum queues by @bityob in #9686
  • Updated rabbitmq doc about using quorum queues with task routes by @bityob in #9707
  • Add: Dumper Unit Test by @yiyu0x in #9711
  • Add unit test for event.group_from by @rogerforlife in #9709
  • refactor: add beat_cron_starting_deadline documentation warning by @alirafiei75 in #9712
  • fix: resolve issue #9569 by supporting distinct broker transport options for workers by @yiyu0x in #9695
  • Fixes issue with retry callback arguments in DelayedDelivery by @maccinza in #9708
  • get_exchange-unit-test by @rogerforlife in #9710
  • ISSUE-9704: Update documentation of result_expires, filesystem backend is supported by @MattiasDC in #9716
  • update to blacksmith ubuntu 24.04 by @auvipy in #9717
  • Added unit tests for celery.utils.iso8601 by @jaiganeshs21 in #9725
  • Update introduction.rst docs by @auvipy in #9728
  • Prepare for release: v5.5.3 by @Nusnus in #9732

New Contributors

Full Changelog: v5.5.2...v5.5.3

v5.5.2

25 Apr 20:10
e8adf7f

Choose a tag to compare

What's Changed

New Contributors

Full Changelog: v5.5.1...v5.5.2

v5.5.1

07 Apr 22:22
d643c7c

Choose a tag to compare

What's Changed

  • Fixed "AttributeError: list object has no attribute strip" with quorum queues and failover brokers by @Nusnus in #9657
  • Prepare for release: v5.5.1 by @Nusnus in #9660

Full Changelog: v5.5.0...v5.5.1

v5.5.0

31 Mar 20:25
d1c35bb

Choose a tag to compare

Celery v5.5.0 is now available.

Key Highlights

See What's new in Celery 5.5 for a complete overview or read the main highlights below.

Redis Broker Stability Improvements

Long-standing disconnection issues with the Redis broker have been identified and resolved in Kombu 5.5.0, which is included with this release. These improvements significantly enhance stability when using Redis as a broker.

Additionally, the Redis backend now has better exception handling with the new exception_safe_to_retry feature, which improves resilience during temporary Redis connection issues. See Redis backend settings for complete documentation.

Contributed by @drienkop in #9614.

pycurl replaced with urllib3

Replaced the pycurl dependency with urllib3.

We're monitoring the performance impact of this change and welcome feedback from users who notice any significant differences in their environments.

Contributed by @spawn-guy in Kombu #2134 and integrated in Celery via #9526.

RabbitMQ Quorum Queues Support

Added support for RabbitMQ's new Quorum Queues feature, including compatibility with ETA tasks. This implementation has some limitations compared to classic queues, so please refer to the documentation for details.

Native Delayed Delivery is automatically enabled when quorum queues are detected to implement the ETA mechanism.

See Using Quorum Queues for complete documentation.

Configuration options:

  • broker_native_delayed_delivery_queue_type: Specifies the queue type for delayed delivery (default: quorum)
  • task_default_queue_type: Sets the default queue type for tasks (default: classic)
  • worker_detect_quorum_queues: Controls automatic detection of quorum queues (default: True)

Contributed in #9207, #9121, and #9599.

For details regarding the 404 errors, see New Year's Security Incident.

Soft Shutdown Mechanism

Soft shutdown is a time limited warm shutdown, initiated just before the cold shutdown. The worker will allow worker_soft_shutdown_timeout seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks.

This feature is particularly valuable when using brokers with visibility timeout mechanisms, such as Redis or SQS. It allows the worker enough time to re-queue tasks that were not completed before exiting, preventing task loss during worker shutdown.

See Stopping the worker for complete documentation on worker shutdown types.

Configuration options:

  • worker_soft_shutdown_timeout: Sets the duration in seconds for the soft shutdown period (default: 0.0, disabled)
  • worker_enable_soft_shutdown_on_idle: Controls whether soft shutdown should be enabled even when the worker is idle (default: False)

Contributed by @Nusnus in #9213, #9231, and #9238.

Pydantic Support

New native support for Pydantic models in tasks. This integration allows you to leverage Pydantic's powerful data validation and serialization capabilities directly in your Celery tasks.

Example usage:

from pydantic import BaseModel
from celery import Celery

app = Celery('tasks')

class ArgModel(BaseModel):
    value: int

class ReturnModel(BaseModel):
    value: str

@app.task(pydantic=True)
def x(arg: ArgModel) -> ReturnModel:
    # args/kwargs type hinted as Pydantic model will be converted
    assert isinstance(arg, ArgModel)

    # The returned model will be converted to a dict automatically
    return ReturnModel(value=f"example: {arg.value}")

See Argument validation with Pydantic for complete documentation.

Configuration options:

  • pydantic=True: Enables Pydantic integration for the task
  • pydantic_strict=True/False: Controls whether strict validation is enabled (default: False)
  • pydantic_context={...}: Provides additional context for validation
  • pydantic_dump_kwargs={...}: Customizes serialization behavior

Contributed by @mathiasertl in #9023, #9319, and #9393.

Google Pub/Sub Transport

New support for Google Cloud Pub/Sub as a message transport, expanding Celery's cloud integration options.

See Google Pub/Sub Transport for complete documentation.

For the Google Pub/Sub support you have to install additional dependencies:

$ pip install "celery[gcpubsub]"

Then configure your Celery application to use the Google Pub/Sub transport:

broker_url = 'gcpubsub://projects/project-id'

Contributed by @haimjether in #9351.

Python 3.13 Support

Official support for Python 3.13. All core dependencies have been updated to ensure compatibility, including Kombu and py-amqp.

This release maintains compatibility with Python 3.8 through 3.13, as well as PyPy 3.10+.

Contributed by @Nusnus in #9309 and #9350.

REMAP_SIGTERM Support

The "REMAP_SIGTERM" feature, previously undocumented, has been tested, documented, and is now officially supported. This feature allows you to remap the SIGTERM signal to SIGQUIT, enabling you to initiate a soft or cold shutdown using TERM instead of QUIT.

This is particularly useful in containerized environments where SIGTERM is the standard signal for graceful termination.

See Cold Shutdown documentation for more info.

To enable this feature, set the environment variable:

export REMAP_SIGTERM="SIGQUIT"

Contributed by @Nusnus in #9461.

Database Backend Improvements

New create_tables_at_setup option for the database backend. This option controls when database tables are created, allowing for non-lazy table creation.

By default (create_tables_at_setup=True), tables are created during backend initialization. Setting this to False defers table creation until they are actually needed, which can be useful in certain deployment scenarios where you want more control over database schema management.

See Database backend settings for complete documentation.

Configuration:

app.conf.result_backend = 'db+sqlite:///results.db'
app.conf.database_create_tables_at_setup = False

Contributed by @MarcBresson in #9228.

What's Changed

Read more

v5.5.0rc5

25 Feb 16:45
e73b71e

Choose a tag to compare

v5.5.0rc5 Pre-release
Pre-release

Celery v5.5.0 Release Candidate 5 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

See What’s new in Celery 5.5 (Immunity) or read the main highlights below.

Using Kombu 5.5.0rc3

The minimum required Kombu version has been bumped to 5.5.0.
Kombu is currently at 5.5.0rc3.

Complete Quorum Queues Support

A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues.

After upgrading to this version, please share your feedback on the quorum queues support.

Relevant Issues: #9207, #6067

New support for Google Pub/Sub transport

After upgrading to this version, please share your feedback on the Google Pub/Sub transport support.

Relevant Issues: #9351

Python 3.13 Improved Support

Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp.

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

  • Bump mypy from 1.13.0 to 1.14.0 by @dependabot in #9476
  • Fix cassandra backend port settings not working by @kairi003 in #9465
  • Unroll group when a group with a single item is chained using the | operator by @thedrow in #9456
  • fix(django): catch the right error when trying to close db connection by @Lotram in #9392
  • Replacing a task with a chain which contains a group now returns a result instead of hanging by @thedrow in #9484
  • Avoid using a group of one as it is now unrolled into a chain by @thedrow in #9510
  • Link to the correct IRC network by @yigitsever in #9509
  • Bump pytest-github-actions-annotate-failures from 0.2.0 to 0.3.0 by @dependabot in #9504
  • Update canvas.rst to fix output result from chain object by @kamalfarahani in #9502
  • Unauthorized Changes Cleanup by @Nusnus in #9528
  • [RE-APPROVED] fix(django): catch the right error when trying to close db connection by @Nusnus in #9529
  • [RE-APPROVED] Link to the correct IRC network by @Nusnus in #9531
  • [RE-APPROVED] Update canvas.rst to fix output result from chain object by @Nusnus in #9532
  • Update test-ci-base.txt by @auvipy in #9539
  • Update install-pyenv.sh by @auvipy in #9540
  • Update elasticsearch requirement from <=8.17.0 to <=8.17.1 by @dependabot in #9518
  • Bump google-cloud-firestore from 2.19.0 to 2.20.0 by @dependabot in #9493
  • Bump mypy from 1.14.0 to 1.14.1 by @dependabot in #9483
  • Update elastic-transport requirement from <=8.15.1 to <=8.17.0 by @dependabot in #9490
  • Update Dockerfile by adding missing Python version 3.13 by @auvipy in #9549
  • Fix typo for default of sig by @daveisfera in #9495
  • fix(crontab): resolve constructor type conflicts by @pcrock-thmdo in #9551
  • worker_max_memory_per_child: kilobyte is 1024 bytes by @mksm in #9553
  • Fix formatting in quorum queue docs by @HenrikOssipoff in #9555
  • Bump cryptography from 44.0.0 to 44.0.1 by @dependabot in #9556
  • Fix the send_task method when detecting if the native delayed delivery approach is available by @mikhaillazko in #9552
  • Reverted PR #7814 & minor code improvement by @MehrazRumman in #9494
  • Improved donation and sponsorship visibility by @Nusnus in #9558
  • Updated the Getting Help section, replacing deprecated with new resources by @Nusnus in #9559
  • Fixed django example by @Nusnus in #9562
  • Bump Kombu to v5.5.0rc3 by @Nusnus in #9564
  • Bump ephem from 4.1.6 to 4.2 by @dependabot in #9565
  • Bump pytest-celery to v1.2.0 by @Nusnus in #9568
  • Remove dependency on pycurl by @jmsmkn in #9526
  • Set TestWorkController.test by @WilliamDEdwards in #9574
  • Fixed bug when revoking by stamped headers a stamp that does not exist by @Nusnus in #9575
  • Canvas Stamping Doc Fixes by @Nusnus in #9578
  • Bugfix: Chord with a chord in header doesn't invoke error callback on inner chord header failure (default config) by @Nusnus in #9580
  • Prepare for (pre) release: v5.5.0rc5 by @Nusnus in #9582

New Contributors

  • @kairi003 made their first contribution in ...
Read more