Celery is a great tool for queuing tasks. In this tutorial I will install Celery from scratch in a Django project and set it up as a daemon.
sudo apt-get install redis-server
pip install django-celery
Changes to settings.py
INSTALLED_APPS = ( ..., djcelery, ) # CELERY SETTINGS CELERY_RESULT_BACKEND = "redis://localhost:6379/2" BROKER_URL = 'redis://localhost:6379/2' import djcelery djcelery.setup_loader()
Check the docs if you want to use a different broker/backend.
Run the migrations
python manage.py migrate djcelery
Setting it up as a daemon
NOTE: You will have to have root access!
See http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#generic-init-scripts for more info in order to customize it.
A sample file of the init.d script is available here https://github.com/celery/celery/tree/3.1/extra/generic-init.d/
Copy the file to your /etc/init.d/celeryd and run the command below
chmod +x /etc/init.d/celeryd
The /etc/default/celeryd is used for configuring the daemon.
# Name of nodes to start, here we have a single node CELERYD_NODES="worker1" # Where to chdir at start. CELERYD_CHDIR="/home/django/gitrepo/djangoproject" # How to call "manage.py celery worker" CELERYD_MULTI="/home/django/env/bin/python $CELERYD_CHDIR/manage.py celery multi" # Extra arguments to celeryd CELERYD_OPTS="--time-limit=300 --concurrency=4 --settings=djangoproject.settings" # Name of the celery config module. CELERY_CONFIG_MODULE="djangoproject.settings" # %n will be replaced with the nodename. CELERYD_LOG_FILE="/home/celery/logs/celery-%n.log" CELERYD_PID_FILE="/home/celery/logs/celery-run-%n.log" # Workers should run as an unprivileged user. CELERYD_USER="celery" CELERYD_GROUP="celery" # Name of the projects settings module. export DJANGO_SETTINGS_MODULE="djangoproject.settings"
Create the celery user and create the logs folder
sudo useradd -m celery -d /home/celery -b /bin/bash sudo su celery mkdir /home/celery/logs
Try firing it up!
sudo /etc/init.d/celeryd start
You should see something like below on your terminal
celery init v10.1. Using config script: /etc/default/celeryd celery multi v3.1.17 (Cipater) > Starting nodes... > worker1@djangoproject: OK
That is all!