Will & Skill Developers

Will & Skill Developers


Thoughts, snippets and ideas from the team at Will & Skill AB, Stockholm.

Faisal Mahmud
Author

“The mind is not a vessel to be filled, but a fire to be kindled.” ― Plutarch

Share


Using Celery with Django and Redis on Ubuntu 14.04

Faisal MahmudFaisal Mahmud

Celery is a great tool for queuing tasks. In this tutorial I will install Celery from scratch in a Django project and set it up as a daemon.

Installing Redis

sudo apt-get install redis-server

Installing Celery

pip install django-celery

Changes to settings.py

INSTALLED_APPS = (
    ...,
    djcelery,
)

# CELERY SETTINGS
CELERY_RESULT_BACKEND = "redis://localhost:6379/2"
BROKER_URL = 'redis://localhost:6379/2'

import djcelery
djcelery.setup_loader()

Check the docs if you want to use a different broker/backend.

Run the migrations

python manage.py migrate djcelery

Setting it up as a daemon

NOTE: You will have to have root access!

/etc/init.d/celeryd

See http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#generic-init-scripts for more info in order to customize it.

A sample file of the init.d script is available here https://github.com/celery/celery/tree/3.1/extra/generic-init.d/

Copy the file to your /etc/init.d/celeryd and run the command below

chmod +x /etc/init.d/celeryd

/etc/default/celeryd

The /etc/default/celeryd is used for configuring the daemon.

# Name of nodes to start, here we have a single node
CELERYD_NODES="worker1"

# Where to chdir at start.
CELERYD_CHDIR="/home/django/gitrepo/djangoproject"

# How to call "manage.py celery worker"
CELERYD_MULTI="/home/django/env/bin/python $CELERYD_CHDIR/manage.py celery multi"

# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=4 --settings=djangoproject.settings"

# Name of the celery config module.
CELERY_CONFIG_MODULE="djangoproject.settings"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/home/celery/logs/celery-%n.log"
CELERYD_PID_FILE="/home/celery/logs/celery-run-%n.log"

# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="djangoproject.settings"

Create the celery user and create the logs folder

sudo useradd -m celery -d /home/celery -b /bin/bash
sudo su celery
mkdir /home/celery/logs

Try firing it up!

sudo /etc/init.d/celeryd start

You should see something like below on your terminal

celery init v10.1.
Using config script: /etc/default/celeryd
celery multi v3.1.17 (Cipater)
> Starting nodes...
    > worker1@djangoproject: OK

That is all!

Faisal Mahmud
Author

Faisal Mahmud

“The mind is not a vessel to be filled, but a fire to be kindled.” ― Plutarch

Comments