Opnieuw: A simple and intuitive retrying library for Python

栏目: IT技术 · 发布时间: 5年前

内容简介:In this post, we want to introduce our new, easy to use, intuitive and simple retry package:Opnieuw is a general-purpose retrying library that simplifies the task of adding retry behavior to both synchronous as well as asynchronous tasks.Channable is a fee

In this post, we want to introduce our new, easy to use, intuitive and simple retry package: Opnieuw , which is responsible for all the network retries inChannable’s Python code.

Opnieuw is a general-purpose retrying library that simplifies the task of adding retry behavior to both synchronous as well as asynchronous tasks.

Channable is a feed processing tool that imports products from webshops and exports those to marketplaces, price comparison sites, and advertisement platforms. At Channable we consume a lot of external APIs for our daily operations. If you make enough API calls, even the most robust API will occasionally fail. By retrying our request we can usually solve this problem. Therefore, having a good retry mechanism is important for making our operations run smoothly.

Why we wrote Opnieuw

Before writing our own retry package, we tried several other retry packages. Unfortunately we fell into multiple pitfalls:

  • Due to too many knobs, it is easy to configure a retry in a way that does not make sense.For example, we had multiple places in our codebase where we used a combination of exponential backoff with a minimum and maximum delay, such that the effective delay would always be the minimum or the maximum; there was no exponential backoff going on. We solve this by providing just two parameters, where every combination of parameters is valid.

  • Due to a modular approach, doing the right thing is hard.Most places in our codebase were not using jitter, simply because adding jitter is a separate step that was often forgotten. We solve this by providing only a single retry strategy: exponential backoff with jitter.

  • Time units were unclear.When you see a line that reads

    @retry(wait_exponential_multiplier=10)

    does that mean that the base delay is 10 milliseconds, or 10 seconds? Due to the nature of the APIs we interact with, reasonable wait times span several orders of magnitude, from milliseconds to minutes. We solve the unit problem by putting the unit in the parameter name. When we upgraded our codebase to our own retry decorator, we fixed multiple bugs where the original code confused seconds with milliseconds.

  • Parameters were unintuitive.All retry libraries that we used, required configuring the base delay for exponential backoff. To determine the base delay, we always found ourselves working backwards from a total wait time and a maximum number of calls. In Opnieuw, we accept those more intuitive parameters directly, and we solve for the base delay.

  • The meaning of parameters was unclear.When we checked the documentation of the libraries we used, we would find explanations like “Wait 2 n * base_delay between each retry.” It is unclear here if that n starts counting at 0 or 1. With exponential growth, an off-by-one can make minutes difference after a few attempts. We solve this by providing more intuitive parameters, and by giving them a verbose but unambiguous name.

Opnieuw is our take at avoiding those pitfalls, to make a retry library that is easy to use. Let’s look at an example in order to demonstrate the power of Opnieuw. Suppose we want to parse https://tech.channable.com/atom.xml , and we want to retry on ConnectionError and HTTPError . We can add Opnieuw to our network handler as follows:

import requests 
from requests.exceptions import ConnectionError, HTTPError

from opnieuw import retry


@retry(
    retry_on_exceptions=(ConnectionError, HTTPError),
    max_calls_total=4,
    retry_window_after_first_call_in_seconds=60, 
) 
def get_page() -> str: 
    response = requests.get('https://tech.channable.com/atom.xml') 
    return response.text 

As is clear from the above example, by decorating the function we add retry behavior. You should not need to read any documentation to understand the snippet:

  • retry_on_exceptions specifies which exceptions we want to retry on.

  • max_calls_total is the maximum number of times that the decorated function gets called. By avoiding terms like “number of retries” and referring to calls instead, it is clear that this includes the initial call.

  • retry_window_after_first_call_in_seconds is the maximum number of seconds after the first call, where we would still do a new attempt. By avoiding terms like “stop after delay”, it is clear that this is not about the total time spent waiting, or the duration of the final delay, but the time elapsed since the first call was initiated.

In the above example, we would call get_page() at least once and at most 4 times, and the last call would be at most 60 seconds after the first call. On a ConnectionError or HTTPError we would retry, and the wait time after a failed attempt grows exponentially. Other exceptions would bubble up up and not be retried.

Opnieuw is a simple and easy way to add exponential backoff retries with jitter to a call. In addition to retry() , Opnieuw features retry_async() for retrying asynchronous tasks. All the parameters remain the same, the only difference is that it can be used with async functions.

About the name

“Opnieuw” means “again” in Dutch. We had an internal naming competition and Opnieuw won out over Perseverance and channable-retry .

Conclusion

At Channable, we saw an opportunity to improve upon existing retry packages. Unclear APIs were a source of bugs and counter-intuitive behavior. By prioritizing the parameters that users care about, Opnieuw makes it easy to reason about retry behavior of network operations.

Today we are releasing Opnieuw as open source.

Discuss this post on Reddit oron Hacker News


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

互联网+秋叶课堂:我的网课创业

互联网+秋叶课堂:我的网课创业

秋叶 / 北京:机械工业出版社 / 2016-3-1 / 39.0

在线教育被很多人视为 “互联网+”创业热潮中的下一个风口,越来越多的老师和创业者选择在线教育创业。本书作者秋叶老师2013年双11正式带一帮小伙伴开始在线教育创业,在短短两年内“从0到1”,累计做到了超500万元的销售业绩,成为国内Office领域在线教育运营最成功的团队之一。在这本书中秋叶老师结合自己的实战经历,向各位完整剖析了两年创业的真实复盘思考,是一本值得在线教育创业同行入手的必读书。 ......一起来看看 《互联网+秋叶课堂:我的网课创业》 这本书的介绍吧!

JSON 在线解析
JSON 在线解析

在线 JSON 格式化工具

随机密码生成器
随机密码生成器

多种字符组合密码

html转js在线工具
html转js在线工具

html转js在线工具