Open Collective
Open Collective
Loading
Celery
Contribute
Budget
Connect
About
New Official Pytest plugin for Celery: pytest-celery v1.0.0 is now released!
Published on April 12, 2024 by Tomer Nosrati
v1.0.0 Official Release

Install with:
pip install -U "pytest-celery[all]"

Key Highlights

Simple

The plugin provides a single entry point to the test case and makes sure everything is configured according to the selected architecture and requirements.

By default, all of the supported architecture components are added to a matrix of all possible combinations. Pytest will generate a test case for each combination, and will run it in an isolated environment.

This allows separation of concerns, and makes it simple to access different architectures in a single test case, for example:

def test_hello_world(celery_setup: CeleryTestSetup): assert celery_setup.ready()

This code will generate test cases for all possible combinations of the supported brokers and backends, using the latest version of Celery. Under the context of the test, each combination will be available as a celery_setup fixture, with access to all of the required components, and will run in an isolated environment.

Flexible

The plugin is highly configurable, and can be used to test a wide range of Celery architectures. It can be configured to use a specific version of Celery, or to use a specific version of a broker or backend. It can also be configured to use a custom broker or backend, or to use a custom Celery application.

For basic usage, the plugin provides default components that can be configured and extended.

For more advanced use cases, the plugin uses the pytest fixtures mechanism to allow injecting custom components into the environment and build a custom Celery architecture for your project.

For example, see the rabbitmq-management example, which demonstrates how to replace the default broker matrix with a single RabbitMQ Management broker.

Fast

The plugin is designed to run tests in parallel using isolated environments. It supports the pytest-xdist plugin to run tests in parallel and scales well with available resources to improve the overall test suite performance.

Annotated

The codebase is fully annotated with type hints and is tested with mypy to ensure type safety across the board, allowing for a better development experience.

Supports
  • Workers
    • Latest Celery version.
    • Custom worker.
  • Brokers
    • RabbitMQ.
    • Redis.
    • Custom broker.
  • Backends
    • Redis.
    • Memcached.
    • Custom backend.
  • Clusters
    • Worker clusters.
    • Broker clusters.
    • Backend clusters.

Features

Architecture Injection

By default, a set of predefined components is used to build the Celery architecture for each test. Each built-in component can be either configured or completely replaced with a custom implementation.

Architecture Injection can be done at different layers, and can be used to replace only specific elements of the architecture pipeline, or to replace the entire pipeline altogether.

Docker Based

The plugin uses docker containers to build the Celery architecture for each test. This means that the plugin is not limited to specific versions and can be used to test potentially any Celery setup.

It uses the pytest-docker-tools plugin to manage the docker containers which is useful for accessing the docker containers in the test case during the test run and assert on their state with high granularity.

Batteries Included

The plugin provides a set of built-in components that can be used to test ideas quickly. You can start with the default settings and gradually modify the configurations to fine-tune the test environment. By focusing on the test case, you can quickly iterate and test ideas, without wasting time on the overhead of setting up different environments manually.

Code Generation

One of the challenges in testing production Celery applications is the need to inject testing infrastructure into the Celery worker container at runtime. The plugin provides a Code Generation mechanism that can be used to inject code into the Celery worker container at runtime according to the test case. This opens the door to a wide range of testing scenarios, and allows higher level of control over the tested Celery application.

Isolated Environments

Each test case is executed in an isolated environment. This means that each test case has its own Celery architecture, and is not affected by other test cases. Tests may run in parallel and take care of tearing down themselves when they are done, regardless of the test result.

Tests as First-Class Citizens

The plugin is designed to enhance testing capabilities by treating tests as first-class citizens. It uses advanced mechanisms to encapsulate the complexity of setting up a Celery environment, thus allowing the developer to focus on the test case itself and leave the hard lifting to the plugin.

Extensible

The plugin is designed to be extensible to fit a wide range of use cases and provides a set of built-in components that can be extended to fit more advanced use cases.

It's based on the S.O.L.I.D principles and provides APIs for developing high-quality test suites. It combines the sophistication of the pytest fixtures mechanism with OOP principles to create separation of concerns between each layer of the infrastructure and its elements, which allow a higher level of granularity and control when extending the plugin.