Exploring Python Task Queues: Celery and rq

Using task queues and background workers can greatly speed up your web applications by offloading work to a different process. This prevents your web app from being blocked from responding to user requests while it does the work.

I’ve been looking at a couple of options for setting up a task queue and background worker, and it’s something that a lot of readers and students have asked for.

The two options I’m looking at are rq and celery, and having tried both I can recommend rq for most applications. It’s a bit simpler, scales well, and it does most of what you might need for a task queue.

On the other hand, celery is more feature-rich but also more complicated (both to learn and use). I found that I could use rq after just a short learning curve, whereas celery took substantially longer. Since developer time is one of our most valuable assets, that’s definitely something to have in mind!

How to set up and use rq

Set up: Redis

To use rq (which stands for “Redis Queue”), you need Redis. You can install using Docker (docker pull redis), or install it locally (I don’t recommend it), or use a free cloud provider (for testing I like Render.com’s offering, since it’s free).

I set up a Redis database with Render.com and enabled incoming connections from anywhere, and it just took a couple minutes to get started.

By contrast, celery recommends RabbitMQ, which you can also set up with Docker relatively painlessly, but it’s more difficult to find a free cloud provider for RabbitMQ.

Set up: tasks

To run a background task, you first need to define what a “task” is. In the rq world, that’s a function plus a set of arguments (which have to be pickle-able). I’d start by creating a file in your project called tasks.py, and put whatever imports and code you need in there so that you can define a function that will run independently of your application.

For example, here’s a tasks.py I wrote that defines a function that sends an email:

import os
import requests
from dotenv import load_dotenv

load_dotenv()

DOMAIN = os.getenv("MAILGUN_DOMAIN")


def send_simple_message(to, subject, body):
    return requests.post(
        f"https://api.mailgun.net/v3/{DOMAIN}/messages",
        auth=("api", os.getenv("MAILGUN_API_KEY")),
        data={
            "from": f"Your Name <mailgun@{DOMAIN}>",
            "to": [to],
            "subject": subject,
            "text": body,
        },
    )


def send_user_registration_email(email, username):
    return send_simple_message(
        email,
        "Successfully signed up",
        f"Hi {username}! You have successfully signed up to the Stores REST API.",
    )

You can see that there are two functions: one is my task, which I’ll give to rq, and that’s send_user_registration_email. It takes in two strings, and calls the other function to actually send the email.

That’s something to remember with task queues and background workers: they are part of your project, and they can have access to your database, environment variables, models, and everything else. The background worker imports the function from tasks.py and runs it with the given arguments, so think about what things tasks.py is importing and what will be available globally.

For example, in Flask terms, you won’t have an application or request context when processing a task, so you can’t use things like current_app or g.

Add tasks to the queue

Adding tasks to the queue is done where you would normally just do what the task does. In my case I was sending emails when users registered, so I would do this in my API endpoint.

Below is a fragment of my Flask-Smorest API that:

1. Sets up a connection to Redis using the redis package.
2. Creates an rq Queue object called "emails" using that connection.
3. At the appropriate point in the API endpoint, enqueues a job to send the user an email using the background worker:

connection = redis.from_url(
    os.getenv("REDIS_URL")
)  # Get this from Render.com or run in Docker
queue = Queue("emails", connection=connection)


class UserRegister(MethodView):
    @blp.arguments(UserRegisterSchema)
    def post(self, user_data):
        # Validate data and create the user object

        # Enqueue the job
        queue.enqueue(send_user_registration_email, user.email, user.username)

Run the background worker

To run the background worker you can use the command line:

rq worker -u <redis_url_here> <queue_name>

To do this in Render.com, you’d use their “Background worker” service type, connect your GitHub repository, and set up the command to run. That command would be the following:

rq worker -c settings

And instead of passing the Redis URL yourself, you’d create a settings.py file like this one:

import os
from dotenv import load_dotenv

load_dotenv()

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379")
QUEUES = ["emails", "default"]

And you’d define an environment variable in Render.com which then would be used.

I’m working on a full section of my REST APIs with Flask and Python that covers task queues with rq in detail and how to send emails from a Flask application (including nice looking HTML emails). If you’d like to learn more about that, consider enrolling in the course!