Spokane Python User Group

Background Task Processing:

Celery vs. Dramatiq

Joseph Riddle

Raffle

Feedback for free JetBrains

https://forms.office.com/r/EFnAfpu3Ue

Outline

  • Definitions
  • What is Celery
  • What is Dramatiq
  • Celery vs. Dramatiq
  • Demo

Message

A unit of communcation between computer processes.

Task

An asynchronous unit of work.

Message Queue

Provides fast asynchronous communication between software components.

Temporarily stores messages, with endpoints for software services to connect to the queue.

https://better.engineering/message-queues/

https://better.engineering/message-queues/

Message Broker

An architectural pattern for message validation, transformation, and routing.

Decoupled communication.

Popular message brokers include RabbitMQ, Apache Kafka, and more...

Cloud provider options include Amazon MSK, Azure Event Hubs, Google Pub/Sub, and more...

RabbitMQ

RabbitMQ is the most widely deployed open source message broker.

https://www.rabbitmq.com/

Supports AMQP and others

Official Docker image

AMQP

The Advanced Message Queuing Protocol (AMQP) is an open standard for passing business messages between applications or organizations.

AMQP Primer

ISO 19464

Redis

An open-source, networked, in-memory, key-value data store with optional durability.

Can be used as a message broker.

Official Docker image

Celery

Celery is an open source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real time.

The de facto Python task queue library.

GitHub repo created in 2009

17.7K⭐

Dramatiq

Dramatiq is a background task processing library for Python with a focus on simplicity, reliability and performance. Created by a user of Celery.

Pronounced the same as "dramatic".

GitHub repo created in 2017

2.7K⭐

Use Cases

  • IO bound tasks
    • Background tasks for web API
    • Sending emails
    • Connecting to other APIs
    • Long running tasks
  • CPU/GPU bound tasks
    • Distributed machine learning batch job
    • Can distribute to multiple workers
  • Scheduling tasks

Celery vs. Dramatiq

Celery Dramatiq
Broker Support RabbitMQ
Redis
Amazon SQS
more...		 RabbitMQ
Redis
In-memory
more...
Result Store Support Redis
Memcached
MongoDB
File system
more...		 Redis
Memcached

Celery vs. Dramatiq

Celery Dramatiq
Scheduling Celery Beat Accomplished via APScheduler
Monitoring Flower
 Yes *
Chaining Yes Yes

https://dramatiq.io/motivation.html

Celery Code

from celery import Celery

app = Celery('tasks', broker='pyamqp://rabbitmq//')

@app.task
def count_to(n):
    for i in range(n):
        print(i)

Dramatiq Code

import dramatiq
from dramatiq.brokers.rabbitmq import RabbitmqBroker

rabbitmq_broker = RabbitmqBroker(url='amqp://guest:guest@rabbitmq')
dramatiq.set_broker(rabbitmq_broker)

@dramatiq.actor
def count_to(n):
    for i in range(n):
        print(i)

Best Practices

  • Pass small amounts of data as messages. Prefer passing IDs instead of large objects, when possible.
  • Don't use result backends if you don't have to.
  • Make tasks idempotent

Other Python Message Queue projects

See https://www.fullstackpython.com/task-queues.html

Questions?

Raffle

Feedback for free JetBrains

https://forms.office.com/r/EFnAfpu3Ue

References

More on Message Queues

Dramatiq with docker-compose

Celery with Kubernetes

A message broker is software that enables applications, systems, and services to communicate with each other and exchange information.

Dramatiq monitoring has a few options, including "dramatiq tasks with monitoring" and "dramatiq dashbord", but neither are as good as Flower

idempotent - easily retryable, can run multiple times and get the same result