Detailed tutorial on decorators closures and functools in Python
- 2020-05-05 11:24:46
- OfStack
decorator (Decorators)
Decorator is a design pattern that can be used if a class wants to add some functionality from other classes, without inheriting or directly modifying the source code. Simply put, an decorator in Python is a function or other callable object that takes a function or class as an optional input parameter and returns a function or class. This new feature, added in Python 2.6, can be used to implement the decorator design pattern.
By the way, if you are not familiar with the concept of closures in Python (Closure) before moving on, see the appendix at the end of this article. Without the concept of closures, it is difficult to understand decorators in Python properly.
In Python, decorators are used for functions or classes that use the @ syntax for sugar rhetoric. Now let's use a simple decorator example to demonstrate how to make a function call logger. In this example, the decorator takes the time format as an input parameter and prints the time of the function call when the function decorated by the decorator is called. This decorator is useful when you need to manually compare the efficiency of two different algorithms or implementations.
def logged(time_format):
def decorator(func):
def decorated_func(*args, **kwargs):
print "- Running '%s' on %s " % (
func.__name__,
time.strftime(time_format)
)
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print "- Finished '%s', execution time = %0.3fs " % (
func.__name__,
end_time - start_time
)
return result
decorated_func.__name__ = func.__name__
return decorated_func
return decorator
Consider an example where the add1 and add2 functions are decorated with logged, and an example of the output is given below. Note here that the time format parameter is stored in the decorator function that is returned (decorated_func). This is why understanding closures is important to understanding decorators. Also notice how the name of the return function is replaced by the original function name, just in case it is used again, to prevent confusion. Python doesn't do this by default.
@logged("%b %d %Y - %H:%M:%S")
def add1(x, y):
time.sleep(1)
return x + y
@logged("%b %d %Y - %H:%M:%S")
def add2(x, y):
time.sleep(2)
return x + y
print add1(1, 2)
print add2(1, 2)
# Output:
- Running 'add1' on Jul 24 2013 - 13:40:47
- Finished 'add1', execution time = 1.001s
3
- Running 'add2' on Jul 24 2013 - 13:40:48
- Finished 'add2', execution time = 2.001s
3
If you are careful, you may notice that we have a special treatment with the name of the return function, but not with the other injections, with either s/s s/s s/s/s/s/s/s. So if, in this case, the add function has an doc string, it will be discarded. So how do you deal with that? Of course we can treat all the fields as with s 50en__, but it would be too cumbersome to do this with each decorator. This is why the functools module provides a decorator called wraps, which is designed to handle this situation. It can be confusing to understand an decorator, but when you think of an decorator as a receiving function name as an input parameter and a function as a return, it makes sense. We will use the wraps decorator in the next example instead of handling the s 54en__ or other attributes manually.
The next example is a little more complicated. Our task is to cache the return result of a function call for a period of time, and the input parameter determines the cache time. The input parameter passed to the function must be a hash object, because we use tuple, which contains the call input parameter, as the first parameter, and frozenset, which contains the keyword kwargs, as cache key. Each function has a unique cache dictionary stored in the function's closure.
set and frozenset are two built-in sets of Python. The former is a mutable object (mutable) whose elements can be changed using add() or remove(), while the latter is an immutable object (imutable) whose elements are hashed (hashable), which can be used as an key of a dictionary or as an element of another set.
import time
from functools import wraps
def cached(timeout, logged=False):
"""Decorator to cache the result of a function call.
Cache expires after timeout seconds.
"""
def decorator(func):
if logged:
print "-- Initializing cache for", func.__name__
cache = {}
@wraps(func)
def decorated_function(*args, **kwargs):
if logged:
print "-- Called function", func.__name__
key = (args, frozenset(kwargs.items()))
result = None
if key in cache:
if logged:
print "-- Cache hit for", func.__name__, key
(cache_hit, expiry) = cache[key]
if time.time() - expiry < timeout:
result = cache_hit
elif logged:
print "-- Cache expired for", func.__name__, key
elif logged:
print "-- Cache miss for", func.__name__, key
# No cache hit, or expired
if result is None:
result = func(*args, **kwargs)
cache[key] = (result, time.time())
return result
return decorated_function
return decorator
Let's see how it works. We use decorators to decorate a very basic fipolacci number generator. this cache The decorator will use the memo mode on the code (Memoize Pattern) . Please note that fib How are function closures stored cache A dictionary, a pointer fib Function references, logged The value of the parameter and timeout Of the last value of the argument. dump_closure Will be defined at the end of the document.
>>> @cached(10, True)
... def fib(n):
... """Returns the n'th Fibonacci number."""
... if n == 0 or n == 1:
... return 1
... return fib(n - 1) + fib(n - 2)
...
-- Initializing cache for fib
>>> dump_closure(fib)
1. Dumping function closure for fib:
-- cell 0 = {}
-- cell 1 =
-- cell 2 = True
-- cell 3 = 10
>>>
>>> print "Testing - F(4) = %d" % fib(4)
-- Called function fib
-- Cache miss for fib ((4,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((3,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((2,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((1,), frozenset([]))
-- Called function fib
-- Cache miss for fib ((0,), frozenset([]))
-- Called function fib
-- Cache hit for fib ((1,), frozenset([]))
-- Called function fib
-- Cache hit for fib ((2,), frozenset([]))
Testing - F(4) = 5
Class Decorators
In the previous section, we looked at some function decorators and some tips to use, then we'll look at class decorators. The class decorator takes an class as an input parameter (a type object in Python) and returns a modified class.
The first example is a simple mathematical problem. When given an ordered set P, we define Pd as P(x,y) < of the reverse order set P
-
>
Pd(x,y), that is, the order of the elements of two ordered sets is opposite to each other. How can this be realized in Python? Suppose a class defines arbitration 103en__ and arbitration 104en__ or some other method to implement order. We can then replace these methods by writing a class decorator.
def make_dual(relation):
@wraps(relation, ['__name__', '__doc__'])
def dual(x, y):
return relation(y, x)
return dual
def dual_ordering(cls):
"""Class decorator that reverses all the orderings"""
for func in ['__lt__', '__gt__', '__ge__', '__le__']:
if hasattr(cls, func):
setattr(cls, func, make_dual(getattr(cls, func)))
return cls
The following is an example of using this decorator of type str to create a new class named rstr in reverse lexicographical order (opposite lexicographic).
@dual_ordering
class rstr(str):
pass
x = rstr("1")
y = rstr("2")
print x < y
print x <= y
print x > y
print x >= y
# Output:
False
False
True
True
Let's do a more complicated example. Suppose we want the logged decorator mentioned earlier to be used for all methods of a class. One solution is to add decorators to each class method. Another option is to write a class decorator that does this automatically. Before getting started, I'll take out the logged decorator from the previous example and make some minor improvements. First, it USES the wraps decorator provided by functools to complete the work of fixing s 129en__. Second, a _logged_decorator attribute (a Boolean variable set to True) is introduced to indicate whether the method has been decorated by the decorator, since the class may be inherited and subclasses may continue to use the decorator. Finally, the name_prefix parameter is added to set the printed log information.
def logged(time_format, name_prefix=""):
def decorator(func):
if hasattr(func, '_logged_decorator') and func._logged_decorator:
return func
@wraps(func)
def decorated_func(*args, **kwargs):
start_time = time.time()
print "- Running '%s' on %s " % (
name_prefix + func.__name__,
time.strftime(time_format)
)
result = func(*args, **kwargs)
end_time = time.time()
print "- Finished '%s', execution time = %0.3fs " % (
name_prefix + func.__name__,
end_time - start_time
)
return result
decorated_func._logged_decorator = True
return decorated_func
return decorator
Okay, let's start with the class decorator:
def log_method_calls(time_format):
def decorator(cls):
for o in dir(cls):
if o.startswith('__'):
continue
a = getattr(cls, o)
if hasattr(a, '__call__'):
decorated_a = logged(time_format, cls.__name__ + ".")(a)
setattr(cls, o, decorated_a)
return cls
return decorator
The following is the use method, notice how the inherited or overridden method is handled.
@log_method_calls("%b %d %Y - %H:%M:%S")
class A(object):
def test1(self):
print "test1"
@log_method_calls("%b %d %Y - %H:%M:%S")
class B(A):
def test1(self):
super(B, self).test1()
print "child test1"
def test2(self):
print "test2"
b = B()
b.test1()
b.test2()
# Output:
- Running 'B.test1' on Jul 24 2013 - 14:15:03
- Running 'A.test1' on Jul 24 2013 - 14:15:03
test1
- Finished 'A.test1', execution time = 0.000s
child test1
- Finished 'B.test1', execution time = 1.001s
- Running 'B.test2' on Jul 24 2013 - 14:15:04
test2
- Finished 'B.test2', execution time = 2.001s
Our first example of a class decorator is the antiorder method of a class. Can a similar decorator, which can be quite useful, implement one of these classes, s/s 162en__, s/s 163en__, s/s 164en__, s/s 165en__ and s/s 166en__, s/s fully sort classes? This is what the functools.total_ordering decorator does. See the reference documentation for details.
Some examples from Flask
Let's take a look at some of the interesting decorators used in Flask.
Suppose you want some function to print a warning at a particular call time, for example, only in debug mode. You don't want to add control code to every function, so you can use decorators. Here is how the decorator defined in Flask app py works.
def setupmethod(f):
"""Wraps a method so that it performs a check in debug mode if the
first request was already handled.
"""
def wrapper_func(self, *args, **kwargs):
if self.debug and self._got_first_request:
raise AssertionError('A setup function was called after the '
'first request was handled. This usually indicates a bug '
'in the application where a module was not imported '
'and decorators or other functionality was called too late.\n'
'To fix this make sure to import all your view modules, '
'database models and everything related at a central place '
'before the application starts serving requests.')
return f(self, *args, **kwargs)
return update_wrapper(wrapper_func, f)
A more interesting example is Flask's route decorator, defined in the Flask class. Notice that the decorator can be a method in a class, with self as the first argument. The complete code is in app.py. Note that the decorator simply registers the decorated function as an URL handle by calling the add_url_rule function.
def route(self, rule, **options):
"""A decorator that is used to register a view function for a
given URL rule. This does the same thing as :meth:`add_url_rule`
but is intended for decorator usage::
@app.route('/')
def index():
return 'Hello World'
For more information refer to :ref:`url-route-registrations`.
:param rule: the URL rule as string
:param endpoint: the endpoint for the registered URL rule. Flask
itself assumes the name of the view function as
endpoint
:param options: the options to be forwarded to the underlying
:class:`~werkzeug.routing.Rule` object. A change
to Werkzeug is handling of method options. methods
is a list of methods this rule should be limited
to (`GET`, `POST` etc.). By default a rule
just listens for `GET` (and implicitly `HEAD`).
Starting with Flask 0.6, `OPTIONS` is implicitly
added and handled by the standard request handling.
"""
def decorator(f):
endpoint = options.pop('endpoint', None)
self.add_url_rule(rule, endpoint, f, **options)
return f
return decorator
extension read
1. official Python Wiki
2. metaprogramming in Python 3
appendix: closure
A function closure is a combination of a function and a set of references to variables in the scope in which the function is defined. The latter typically points to a reference environment (referencing environment), which enables the function to be executed outside the region where it is defined. In Python, this reference environment is stored in tuple of cell. You can access it by using the s 239en_closure or s 242en__ attribute of s 241en 3. One thing to keep in mind is that references are references, not deep copies of objects. Of course, this is not a problem for immutable objects, but for mutable objects (list) it is important to be aware of this, and an example will follow. Note that the function also has the s 244en__ field at the defined location to store the global reference environment.
Consider a simple example:
@logged("%b %d %Y - %H:%M:%S")
def add1(x, y):
time.sleep(1)
return x + y
@logged("%b %d %Y - %H:%M:%S")
def add2(x, y):
time.sleep(2)
return x + y
print add1(1, 2)
print add2(1, 2)
# Output:
- Running 'add1' on Jul 24 2013 - 13:40:47
- Finished 'add1', execution time = 1.001s
3
- Running 'add2' on Jul 24 2013 - 13:40:48
- Finished 'add2', execution time = 2.001s
3
A slightly more complicated example. Make sure you understand why you're doing this.
@logged("%b %d %Y - %H:%M:%S")
def add1(x, y):
time.sleep(1)
return x + y
@logged("%b %d %Y - %H:%M:%S")
def add2(x, y):
time.sleep(2)
return x + y
print add1(1, 2)
print add2(1, 2)
# Output:
- Running 'add1' on Jul 24 2013 - 13:40:47
- Finished 'add1', execution time = 1.001s
3
- Running 'add2' on Jul 24 2013 - 13:40:48
- Finished 'add2', execution time = 2.001s
3
When z.append(3), the internal reference to g() and z still refer to a variable, and after z=[1], they no longer refer to a variable.
Finally, let's look at the definition of the dump_closure method used in the code.
@logged("%b %d %Y - %H:%M:%S")
def add1(x, y):
time.sleep(1)
return x + y
@logged("%b %d %Y - %H:%M:%S")
def add2(x, y):
time.sleep(2)
return x + y
print add1(1, 2)
print add2(1, 2)
# Output:
- Running 'add1' on Jul 24 2013 - 13:40:47
- Finished 'add1', execution time = 1.001s
3
- Running 'add2' on Jul 24 2013 - 13:40:48
- Finished 'add2', execution time = 2.001s
3