Truthhunter346

Run tests in a module. Pytest testmod.py. Run tests in a directory. Pytest testing/ Run tests by keyword expressions. Pytest -k 'MyClass and not method' This will run tests which contain names that match the given string expression, which can include Python operators that use filenames, class names and function names as variables. Run tests with Tox. Azure Pipelines can run parallel Tox test jobs to split up the work. On a development computer, you have to run your test environments in series. This sample uses tox -e py to run whichever version of Python is active for the current job. In this Django project tutorial, we learned how to create a project in Django and how to run the test server. These processes are just the basic aspects that you need to understand while working with Django. However, we will also learn the other aspects related to the Django framework for creating an application in upcoming articles.

If a long-running process is part of your application's workflow, rather than blocking the response, you should handle it in the background, outside the normal request/response flow.

Perhaps your web application requires users to submit a thumbnail (which will probably need to be re-sized) and confirm their email when they register. If your application processed the image and sent a confirmation email directly in the request handler, then the end user would have to wait unnecessarily for them both to finish processing before the page loads or updates. Instead, you'll want to pass these processes off to a task queue and let a separate worker process deal with it, so you can immediately send a response back to the client. The end user can then do other things on the client-side while the processing takes place. Your application is also free to respond to requests from other users and clients.

To achieve this, we'll walk you through the process of setting up and configuring Celery and Redis for handling long-running processes in a Django app. We'll also use Docker and Docker Compose to tie everything together. Finally, we'll look at how to test the Celery tasks with unit and integration tests.

Django + Celery Series:

1. Asynchronous Tasks with Django and Celery (this article!)

Contents

Objectives

By the end of this tutorial you will be able to:

1. Integrate Celery into a Django app and create tasks
2. Containerize Django, Celery, and Redis with Docker
3. Run processes in the background with a separate worker process
4. Save Celery logs to a file
5. Set up Flower to monitor and administer Celery jobs and workers
6. Test a Celery task with both unit and integration tests

Background Tasks

Again, to improve user experience, long-running processes should be run outside the normal HTTP request/response flow, in a background process.

Examples:

1. Running machine learning models
2. Sending confirmation emails
3. Scraping and crawling
4. Analyzing data
5. Processing images
6. Generating reports

As you're building out an app, try to distinguish tasks that should run during the request/response lifecycle, like CRUD operations, from those that should run in the background.

Workflow

Our goal is to develop a Django application that works in conjunction with Celery to handle long-running processes outside the normal request/response cycle.

1. The end user kicks off a new task via a POST request to the server-side.
2. Within the view, a task is added to the queue and the task id is sent back to the client-side.
3. Using AJAX, the client continues to poll the server to check the status of the task while the task itself is running in the background.

Project Setup

Clone down the base project from the django-celery repo, and then check out the v1 tag to the master branch:

Since we'll need to manage three processes in total (Django, Redis, worker), we'll use Docker to simplify our workflow by wiring them up so that they can all be run from one terminal window with a single command.

From the project root, create the images and spin up the Docker containers:

Once the build is complete, navigate to http://localhost:1337:

Make sure the tests pass as well:

Take a quick look at the project structure before moving on:

Want to learn how to build this project? Check out the Dockerizing Django with Postgres, Gunicorn, and Nginx blog post.

Trigger a Task

An event handler in project/static/main.js is set up that listens for a button click. On click, an AJAX POST request is sent to the server with the appropriate task type: 1, 2, or 3.

On the server-side, a view is already configured to handle the request in project/tasks/views.py:

Now comes the fun part: wiring up Celery!

Celery Setup

Start by adding both Celery and Redis to the project/requirements.txt file:

Celery uses a message broker -- RabbitMQ, Redis, or AWS Simple Queue Service (SQS) -- to facilitate communication between the Celery worker and the web application. Messages are added to the broker, which are then processed by the worker(s). Once done, the results are added to the backend.

Redis will be used as both the broker and backend. Add both Redis and a Celery worker to the docker-compose.yml file like so:

Take note of celery worker --app=core --loglevel=info:

1. celery worker is used to start a Celery worker
2. --app=core runs the core Celery Application (which we'll define shortly)
3. --loglevel=info sets the logging level to info

Within the project's settings module, add the following at the bottom to tell Celery to use Redis as the broker and backend:

Next, create a new file called sample_tasks.py in 'project/tasks':

Here, using the shared_task decorator, we defined a new Celery task function called create_task.

Keep in mind that the task itself will not be executed from the Django process; it will be executed by the Celery worker.

Now, add a celery.py file to 'project/core':

What's happening here?

1. First, we set a default value for the DJANGO_SETTINGS_MODULE environment variable so that Celery will know how to find the Django project.
2. Next, we created a new Celery instance, with the name core, and assigned the value to a variable called app.
3. We then loaded the celery configuration values from the settings object from django.conf. We used namespace='CELERY' to prevent clashes with other Django settings. All config settings for Celery must be prefixed with CELERY_, in other words.
4. Finally, app.autodiscover_tasks() tells Celery to look for Celery tasks from applications defined in settings.INSTALLED_APPS.

Update project/core/__init__.py so that the Celery app is automatically imported when Django starts:

Trigger a Task

Update the view to kick off the task and respond with the id:

Don't forget to import the task:

Build the images and spin up the new containers:

To trigger a new task, run:

You should see something like:

Task Status

Turn back to the event handler on the client-side:

When the response comes back from the original AJAX request, we then continue to call getStatus() with the task id every second:

If the response is successful, a new row is added to the table on the DOM.

Update the get_status view to return the status:

Import AsyncResult:

Update the containers:

Trigger a new task:

Then, grab the task_id from the response and call the updated endpoint to view the status:

Test it out in the browser as well:

Celery Logs

Run Django Tests Positive

Update the celery service, in docker-compose.yml, so that Celery logs are dumped to a log file:

Add a new directory to 'project' called 'logs. Then, add a new file called celery.log to that newly created directory.

Update:

You should see the log file fill up locally since we set up a volume:

Flower Dashboard

Flower is a lightweight, real-time, web-based monitoring tool for Celery. You can monitor currently running tasks, increase or decrease the worker pool, view graphs and a number of statistics, to name a few.

Add it to requirements.txt:

Then, add a new service to docker-compose.yml:

Test it out:

Navigate to http://localhost:5555 to view the dashboard. You should see one worker ready to go:

Kick off a few more tasks to fully test the dashboard:

Try adding a few more workers to see how that affects things:

Run Django Tests In Pycharm

Tests

Let's start with the most basic test:

Add the above test case to project/tests/test_tasks.py, and then add the following import:

Run that test individually:

It should take about one minute to run:

It's worth noting that in the above asserts, we used the .run method (rather than .delay) to run the task directly without a Celery worker.

Want to mock the .run method to speed things up?

Import:

Test:

Much quicker!

How about a full integration test?

Keep in mind that this test uses the same broker and backend used in development. You may want to instantiate a new Celery app for testing:

Add the import:

Ensure the test passes.

Conclusion

This has been a basic guide on how to configure Celery to run long-running tasks in a Django app. You should let the queue handle any processes that could block or slow down the user-facing code.

Celery can also be used to execute repeatable tasks and break up complex, resource-intensive tasks so that the computational workload can be distributed across a number of machines to reduce (1) the time to completion and (2) the load on the machine handling client requests.

Finally, if you're curious about how to use Websockets (via Django Channels) to check the status of a Celery task, instead of using AJAX polling, check out the The Definitive Guide to Celery and Django course.

Grab the code from the repo.

Django + Celery Series:

1. Asynchronous Tasks with Django and Celery (this article!)

Testing your code is very important.

Getting used to writing testing code and running this code in parallel is nowconsidered a good habit. Used wisely, this method helps to define yourcode’s intent more precisely and have a more decoupled architecture.

Some general rules of testing:

• A testing unit should focus on one tiny bit of functionality and prove itcorrect.
• Each test unit must be fully independent. Each test must be able to runalone, and also within the test suite, regardless of the order that they arecalled. The implication of this rule is that each test must be loaded witha fresh dataset and may have to do some cleanup afterwards. This isusually handled by setUp() and tearDown() methods.
• Try hard to make tests that run fast. If one single test needs more than afew milliseconds to run, development will be slowed down or the tests willnot be run as often as is desirable. In some cases, tests can’t be fastbecause they need a complex data structure to work on, and this data structuremust be loaded every time the test runs. Keep these heavier tests in aseparate test suite that is run by some scheduled task, and run all othertests as often as needed.
• Learn your tools and learn how to run a single test or a test case. Then,when developing a function inside a module, run this function’s testsfrequently, ideally automatically when you save the code.
• Always run the full test suite before a coding session, and run it againafter. This will give you more confidence that you did not break anythingin the rest of the code.
• It is a good idea to implement a hook that runs all tests before pushingcode to a shared repository.
• If you are in the middle of a development session and have to interruptyour work, it is a good idea to write a broken unit test about what youwant to develop next. When coming back to work, you will have a pointerto where you were and get back on track faster.
• The first step when you are debugging your code is to write a new testpinpointing the bug. While it is not always possible to do, those bugcatching tests are among the most valuable pieces of code in your project.
• Use long and descriptive names for testing functions. The style guide hereis slightly different than that of running code, where short names areoften preferred. The reason is testing functions are never called explicitly.square() or even sqr() is ok in running code, but in testing code youwould have names such as test_square_of_number_2(),test_square_negative_number(). These function names are displayed whena test fails, and should be as descriptive as possible.
• When something goes wrong or has to be changed, and if your code has agood set of tests, you or other maintainers will rely largely on thetesting suite to fix the problem or modify a given behavior. Thereforethe testing code will be read as much as or even more than the runningcode. A unit test whose purpose is unclear is not very helpful in thiscase.
• Another use of the testing code is as an introduction to new developers. Whensomeone will have to work on the code base, running and reading the relatedtesting code is often the best thing that they can do to start. They willor should discover the hot spots, where most difficulties arise, and thecorner cases. If they have to add some functionality, the first step shouldbe to add a test to ensure that the new functionality is not already aworking path that has not been plugged into the interface.

The Basics¶

unittest¶

unittest is the batteries-included test module in the Python standardlibrary. Its API will be familiar to anyone who has used any of theJUnit/nUnit/CppUnit series of tools.

Creating test cases is accomplished by subclassing unittest.TestCase.

As of Python 2.7 unittest also includes its own test discovery mechanisms.

Doctest¶

The doctest module searches for pieces of text that look like interactivePython sessions in docstrings, and then executes those sessions to verify thatthey work exactly as shown.

Doctests have a different use case than proper unit tests: they are usuallyless detailed and don’t catch special cases or obscure regression bugs. Theyare useful as an expressive documentation of the main use cases of a module andits components. However, doctests should run automatically each time the fulltest suite runs.

A simple doctest in a function:

When running this module from the command line as in pythonmodule.py, thedoctests will run and complain if anything is not behaving as described in thedocstrings.

Tools¶

py.test¶

py.test is a no-boilerplate alternative to Python’s standard unittest module.

Despite being a fully-featured and extensible test tool, it boasts a simplesyntax. Creating a test suite is as easy as writing a module with a couple offunctions:

and then running the py.test command:

is far less work than would be required for the equivalent functionality withthe unittest module!

Hypothesis¶

Hypothesis is a library which lets you write tests that are parameterized bya source of examples. It then generates simple and comprehensible examplesthat make your tests fail, letting you find more bugs with less work.

For example, testing lists of floats will try many examples, but report theminimal example of each bug (distinguished exception type and location):

Hypothesis is practical as well as very powerful and will often find bugsthat escaped all other forms of testing. It integrates well with py.test,and has a strong focus on usability in both simple and advanced scenarios.

tox¶

tox is a tool for automating test environment management and testing againstmultiple interpreter configurations.

tox allows you to configure complicated multi-parameter test matrices via asimple INI-style configuration file.

mock¶

unittest.mock is a library for testing in Python. As of Python 3.3, it isavailable in thestandard library.

For older versions of Python:

It allows you to replace parts of your system under test with mock objects andmake assertions about how they have been used.

For example, you can monkey-patch a method:

To mock classes or objects in a module under test, use the patch decorator.In the example below, an external search system is replaced with a mock thatalways returns the same result (but only for the duration of the test).

Mock has many other ways with which you can configure and control its behaviour.