Django笔记 使用Celery来进行定时Batch任务

環境

unix

Python2.7

Django1.8

Celery3.1

django-celery3.1

需要安装

pip install celery
pip install django-celery

代码

代码的框架

djangotest
└-apptest
└-tasks.py
└-

└-djangotest
└-celery.py
└-settings.py

代码

__init__.py
from __future__ import absolute_import

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app  # noqa

djangotest/setting.py

# 30秒进行一次处理
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
'add-every-30-seconds': {
'task': 'apptest.tasks.test_celery',
'schedule': timedelta(seconds=30),
'args': (16, 16)
},
# Executes every Monday morning at 7:30 A.M
'add-every-1-minute': {
'task': 'apptest.tasks.test_celery3',
'schedule': crontab(minute='*/1'),
'args': ('test_celery3',),
},
}

CELERY_TIMEZONE = 'UTC'

BROKER_URL = 'django://'

djangotest/celery.py

from __future__ import absolute_import

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djangotest.settings')

from django.conf import settings

app = Celery('djangotest')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)

app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.cache:CacheBackend',
)

@app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))

apptest/tasks.py

from __future__ import absolute_import

import logging
from celery import task
from celery.utils.log import get_task_logger
from appmanager import models

@task
def test_celery(x, y):
logger = get_task_logger(__name__)
logger.info('func start  ----------------->')
logger.info('application:%s', "TEST_APP")
logger.info('func end -------------------->')
return x + y

在setting.py中添加app和同步数据库

INSTALLED_APPS = [
。。。
'djcelery',
'kombu.transport.django',
'apptask',
]

执行

python manage.py migrate

启动celery服务

1.启动工作进程 work

celery -A djangotest worker -B -l info

#error ImportError: No module named
celery -A djangotest worker --app=djangotest.celery:app -l info -c 1

python manage.py celeryd -B 只启动一个进程

2.启动工作触发进程 celerybeat

python manage.py celerybeat

然后就会循环显示执行的结果

func start  ----------------->
application:TEST_APP
func end -------------------->

在生产环境中，使用django作为容器，并不安定

1.安装redis和启动

2.在setting文件中，设置容器地址。

BROKER_URL = 'redis://localhost:6379/0'

3.在celery setting文件中，设

CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

参照

http://www.weiguda.com/blog/73/

http://www.tuicool.com/articles/aIvYbie

http://qiita.com/shinno21/items/8c57a66536c0f38b3fbb

http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/

http://www.ibm.com/developerworks/cn/opensource/os-cn-celery-web-service/index.html?ca=drs-&utm_source=tuicool&utm_medium=referral

djcelery入门：实现运行定时任务 http://my.oschina.net/kinegratii/blog/292395#OSC_h2_6

使用celery之了解celery - 小明明’ http://www.dongwm.com/archives/shi-yong-celeryzhi-liao-jie-celery/?utm_source=tuicool&utm_medium=referral

使用django+celery+RabbitMQ实现异步执行 http://charlee.li/django-celery-rabbitmq-intro.html

官方文档：http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

cron时间格式：http://www.nncron.ru/help/EN/working/cron-format.htm

celery最佳实践：http://my.oschina.net/siddontang/blog/284107

使用django+celery+RabbitMQ实现异步执行：http://blog.charlee.li/django-celery-rabbitmq-intro/

RabbitMQ文档： http://www.rabbitmq.com/documentation.html

Django/Celery Quickstart (or, how I l
b849
earned to stop using cron and love celery) http://chase-seibert.github.io/blog/2010/07/09/djangocelery-quickstart-or-how-i-learned-to-stop-using-cron-and-love-celery.html

Using Celery to handle asynchronous tasks in Django – Sebastian Dahlgren http://sebastiandahlgren.se/2012/11/13/using-celery-for-asynchronous-messages-in-django/

Django-celery配置及使用指南 http://zhujinliang.cn/django/2013/07/18/Django-celery%E9%85%8D%E7%BD%AE%E5%8F%8A%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97/

Introducing Celery for Python+Django - LINUX For You http://www.opensourceforu.com/2013/12/introducing-celery-pythondjango/

Django中如何使用django-celery完成异步任务 (1) | 上海味股达信息科技有限公司 http://www.weiguda.com/blog/73/

Django中如何使用django-celery完成异步任务 (2) http://www.weiguda.com/blog/74/

Django Celery Architecture | langoor.mobi Blog http://blog.langoor.mobi/django-celery-redis-vs-rabbitmq-message-broker/django_celery_architecture/

Queueing Messages using Celery with RabbitMQ Message Broker Server - 2014 http://www.bogotobogo.com/python/RabbitMQ_Celery/python_Queueing%20using_Celery_with_RabbitMQ_Message_Broker_Server.php

AMQP, RabbitMQ and Celery - A Visual Guide For Dummies | Abhishek Tiwari http://abhishek-tiwari.com/post/amqp-rabbitmq-and-celery-a-visual-guide-for-dummies

Tracing Celery Performance For Web Applications - 推酷 http://www.tuicool.com/articles/ZbyUnu

http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

redis

在Ubuntu中安装Redis http://blog.fens.me/linux-redis-install/

http://dim5.net/windows-server/install-redis.html

http://www.cnblogs.com/happyframework/archive/2013/07/18/3197392.html