任务队列 Task queues
比如，一个web应用汇每10分钟pull一次GitHub API，去收集前100个标星的仓库。一个任务队列用来处理这个工作，去调用GitHub API，并把结果存储到持久的数据库里。
- International Space Station notifications with Python and Redis Queue (RQ) shows how to combine the RQ task queue library with Flask to send text message notifications every time a condition is met – in this blog post’s case that the ISS is currently flying over your location on Earth.
- Evaluating persistent, replicated message queues is a detailed comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and Kafka’s designs and performance.
- Queues.io is a collection of task queue systems with short summaries for each one. The task queues are not all compatible with Python but ones that work with it are tagged with the “Python” keyword.
- Why Task Queues is a presentation for what task queues are and why they are needed.
- Asynchronous Processing in Web Applications Part One and Part Two are great reads for understanding the difference between a task queue and why you shouldn’t use your database as one.
- Flask by Example Implementing a Redis Task Queue provides a detailed walkthrough of setting up workers to use RQ with Redis.
- Heroku has a clear walkthrough for using RQ for background tasks.
- How to use Celery with RabbitMQ is a detailed walkthrough for using these tools on an Ubuntu VPS.
- Celery – Best Practices explains things you should not do with Celery and shows some underused features for making task queues easier to work with.
- Celery in Production on the Caktus Group blog contains good practices from their experience using Celery with RabbitMQ, monitoring tools and other aspects not often discussed in existing documentation.
- A 4 Minute Intro to Celery is a short introductory task queue screencast.
- Heroku wrote about how to secure Celery when tasks are otherwise sent over unencrypted networks.
- Miguel Grinberg wrote a nice post on using the task queue Celery with Flask. He gives an overview of Celery followed by specific code to set up the task queue and integrate it with Flask.
- 3 Gotchas for Working with Celery are things to keep in mind when you’re new to the Celery task queue implementation.
- Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis is a video along with code that shows how to set up Celery with Redis as the broker in a Django application.
- Setting up an asynchronous task queue for Django using Celery and Redis is a straightforward tutorial for setting up the Celery task queue for Django web applications using the Redis broker on the back end.
- Three quick tips from two years with Celery provides some solid advice on retry delays, the -Ofair flag and global task timeouts for Celery.
- Pick a slow function in your project that is called during an HTTP request.
- Determine if you can precompute the results on a fixed interval instead of during the HTTP request. If so, create a separate function you can call from elsewhere then store the precomputed value in the database.
- Read the Celery documentation and the links in the resources section below to understand how the project works.
- Install a message broker such as RabbitMQ or Redis and then add Celery to your project. Configure Celery to work with the installed message broker.
- Use Celery to invoke the function from step one on a regular basis.
- Have the HTTP request function use the precomputed value instead of the slow running code it originally relied upon.