Functional programming of Python decorator
- 2020-04-02 14:35:09
- OfStack
Python decorators are called decorators in English, and when you see them, you might confuse them with decorators in Design Pattern, which are two very different things. Although like, they are very similar to do - for an existing module are all want to do some "decoration", the so-called touch-up work to existing module and a few small adornment (a few small function, these small function may be a lot of module is used), but don't let the small ornament (small) intrusion into the code contained in the original module. But OO Decorator is a nightmare, do not believe you look at the wikipedia entry (the Decorator Pattern) in the UML diagram and the code, this is me in the look from the object-oriented design Pattern software design "" dessert" section, OO encouraged - "thick glue and complex layers", is "so understanding of object-oriented programming of" OO enthusiasts are very afraid of processing data ", The code in the Decorator Pattern is an OO tutorial in reverse.
Python's Decorator is similar to Java/C#'s Annotation in that it decorates the method name with an @xxx Annotation. But the Java/C# Annotation thing is also daunting, it's so fucking complicated, you're going to play with it, you're going to have to know a bunch of class library documents for annotations, it's going to feel like you're learning another language.
Python, on the other hand, USES a very elegant approach to Decorator patterns and annotations that doesn't require you to master complex OO models or the various library conventions of annotations, but is entirely language-level: a functional programming technique. If you've ever seen functional programming on this site, you're sure to be happy with the way that functional programming describes what you want to do, not how you're going to do it. (if you don't know anything about functional programming, take a step back to "functional programming" before you read this article.) well, for a little intuition, let's look at the Hello World code for a Python decorator.
Hello World
Here is the code:
File name: hello.py
def hello(fn):
def wrapper():
print "hello, %s" % fn.__name__
fn()
print "goodby, %s" % fn.__name__
return wrapper
@hello
def foo():
print "i am foo"
foo()
When you run the code, you will see the following output:
[chenaho@chenhao-air]$ python hello.py
hello, foo
i am foo
goodby, foo
You can see something like this:
1) the function foo has an @hello "annotation" in front of it
2) in the hello function, it needs an argument of fn (this is the function used for callback)
3) the hello function returns an inner wrapper that calls back the incoming fn and adds two statements before and after the callback.
The essence of the Decorator
For Python's @annotation sugar-syntactic Sugar, when you use an @decorator to decorate a function func, it looks like this:
@decorator
def func():
pass
The interpreter will interpret it as follows:
func = decorator(func)
Nim, isn't that just taking a function and passing it to another function and then calling it back? Yes, but we need to note that there is also an assignment statement that assigns the return value of the decorator function back to the original func. As defined in first class functions in functional programming, you can use the function as a variable, so the decorator must return a function to func, which is called higher order function, otherwise it will fail when func() is called. In the case of the hello.py example above,
@hello
def foo():
print "i am foo"
It was interpreted as:
foo = hello(foo)
Yes, this is a statement and it is executed. If you don't believe me, you can write a program like this:
def fuck(fn):
print "fuck %s!" % fn.__name__[::-1].upper()
@fuck
def wfg():
pass
No, on the above code, there is no statement to call WFG (), you will find that the fuck function was called, but also very NB to output our voices!
Going back to our example of hello.py, we can see that hello(foo) returns the wrapper() function, so foo actually becomes a variable in the wrapper, and then foo() executes as wrapper().
Knowing this, you won't be scared when you see multiple decorators or decorators with parameters.
For example: multiple decorators
@decorator_one
@decorator_two
def func():
pass
Is equivalent to:
func = decorator_one(decorator_two(func))
For example: decorator with parameters:
@decorator(arg1, arg2)
def func():
pass
Is equivalent to:
func = decorator(arg1,arg2)(func)
This means that the decorator(arg1, arg2) function needs to return a "real decorator".
With parameters and multiple decrorators
Let's take a look at an example that makes sense:
HTML. Py
def makeHtmlTag(tag, *args, **kwds):
def real_decorator(fn):
css_class = " class='{0}'".format(kwds["css_class"])
if "css_class" in kwds else ""
def wrapped(*args, **kwds):
return "<"+tag+css_class+">" + fn(*args, **kwds) + "</"+tag+">"
return wrapped
return real_decorator
@makeHtmlTag(tag="b", css_class="bold_css")
@makeHtmlTag(tag="i", css_class="italic_css")
def hello():
return "hello world"
print hello()
# Output:
# <b class='bold_css'><i class='italic_css'>hello world</i></b>
In the above example, we can see that makeHtmlTag takes two arguments. So, in order for hello = makeHtmlTag(arg1, arg2)(hello) to succeed, the makeHtmlTag must return a decorator(which is why we added real_decorator() to makeHtmlTag) so that we can get into the decorator's logic -- the decorator must return a wrapper with a callback to hello inside the wrapper. It may seem like that makeHtmlTag() is written in layers, but, given the nature of it, it feels natural.
You see, Python Decorator is so simple, no complicated things, you don't need to understand things too much, use rise is so natural, considerate, dry and breathable, uniquely available concave lines and perfect absorption path, let you no longer feel anxiety and discomfort for the days each month, plus close wings design, how don't have to be careful. Sorry, I'm naughty.
What, you think the above Decorator function with arguments is too nested for you. Okay, that's fine. Let's take a look at the following.
The Decorator class type
First, let's talk about the decorator's class style, but let's look at an example:
class myDecorator(object):
def __init__(self, fn):
print "inside myDecorator.__init__()"
self.fn = fn
def __call__(self):
self.fn()
print "inside myDecorator.__call__()"
@myDecorator
def aFunction():
print "inside aFunction()"
print "Finished decorating aFunction()"
aFunction()
# Output:
# inside myDecorator.__init__()
# Finished decorating aFunction()
# inside aFunction()
# inside myDecorator.__call__()
The above example shows declaring a decorator as a class. We can see that there are two members in this class:
1) one is s), this method is called when we give a function decorator, so there needs to be an argument of fn, which is the function being decorator.
2) one is s calling (), this method is called when we call the decorator function.
The output above shows the order in which the entire program is executed.
This seems a bit easier to read than the "functional" way.
Next, let's look at the code to override the above html.py in a class way:
HTML. Py
class makeHtmlTagClass(object):
def __init__(self, tag, css_class=""):
self._tag = tag
self._css_class = " class='{0}'".format(css_class)
if css_class !="" else ""
def __call__(self, fn):
def wrapped(*args, **kwargs):
return "<" + self._tag + self._css_class+">"
+ fn(*args, **kwargs) + "</" + self._tag + ">"
return wrapped
@makeHtmlTagClass(tag="b", css_class="bold_css")
@makeHtmlTagClass(tag="i", css_class="italic_css")
def hello(name):
return "Hello, {}".format(name)
print hello("Hao Chen")
In the above code, we need to pay attention to the following:
1) if the decorator has arguments, then the s/s () member cannot be passed in with the f/n, which is then passed in with the s/s.
2) the code also shows wrapped(*args, **kwargs) as a way to pass arguments to the decorator function. (where: args is a parameter list and kwargs is a parameter dict. For details, please refer to the Python documentation or the question of StackOverflow.
Use the Decorator to set the call parameters of the function
There are three ways you can do this:
First, by **kwargs, this method decorator injects parameters into kwargs.
def decorate_A(function):
def wrap_function(*args, **kwargs):
kwargs['str'] = 'Hello!'
return function(*args, **kwargs)
return wrap_function
@decorate_A
def print_message_A(*args, **kwargs):
print(kwargs['str'])
print_message_A()
The second, the agreement of good parameters, directly modify the parameters
def decorate_B(function):
def wrap_function(*args, **kwargs):
str = 'Hello!'
return function(str, *args, **kwargs)
return wrap_function
@decorate_B
def print_message_B(str, *args, **kwargs):
print(str)
print_message_B()
Third, through *args injection
def decorate_C(function):
def wrap_function(*args, **kwargs):
str = 'Hello!'
#args.insert(1, str)
args = args +(str,)
return function(*args, **kwargs)
return wrap_function
class Printer:
@decorate_C
def print_message(self, str, *args, **kwargs):
print(str)
p = Printer()
p.print_message()
Side effects of decorators
By this point, I'm sure you know how the whole Python decorator works.
I'm sure you'll find that the function being decorated is already a different function. For the previous hello. Py example, if you look up foo. s. So, a decorator called wrap is provided in Python's functool package to eliminate this side effect. Here's our new version of hello.py.
File name: hello.py
from functools import wraps
def hello(fn):
@wraps(fn)
def wrapper():
print "hello, %s" % fn.__name__
fn()
print "goodby, %s" % fn.__name__
return wrapper
@hello
def foo():
'''foo help doc'''
print "i am foo"
pass
foo()
print foo.__name__ # The output foo
print foo.__doc__ # The output foo help doc
Of course, even if you take wraps with functools, that doesn't completely eliminate this side effect.
Consider the following example:
from inspect import getmembers, getargspec
from functools import wraps
def wraps_decorator(f):
@wraps(f)
def wraps_wrapper(*args, **kwargs):
return f(*args, **kwargs)
return wraps_wrapper
class SomeClass(object):
@wraps_decorator
def method(self, x, y):
pass
obj = SomeClass()
for name, func in getmembers(obj, predicate=inspect.ismethod):
print "Member Name: %s" % name
print "Func Name: %s" % func.func_name
print "Args: %s" % getargspec(func)[0]
# Output:
# Member Name: method
# Func Name: method
# Args: []
You'll notice that even if you take wraps of functools, when you take getargspec, the argument is gone.
To fix this problem, we'll have to solve it with Python reflection. Here's the code:
def get_true_argspec(method):
argspec = inspect.getargspec(method)
args = argspec[0]
if args and args[0] == 'self':
return argspec
if hasattr(method, '__func__'):
method = method.__func__
if not hasattr(method, 'func_closure') or method.func_closure is None:
raise Exception("No closure for method.")
method = method.func_closure[0].cell_contents
return get_true_argspec(method)
Of course, I'm sure most people's programs don't go to getargspec. So, wraps with functools should be enough.
Some examples of decorators
Okay, now let's look at some examples of decorators:
Cache function calls
This example is so classic that it's used all over the web as a classic example of a decorator, and because it's so classic, this article is no exception.
from functools import wraps
def memo(fn):
cache = {}
miss = object()
@wraps(fn)
def wrapper(*args):
result = cache.get(args, miss)
if result is miss:
result = fn(*args)
cache[args] = result
return result
return wrapper
@memo
def fib(n):
if n < 2:
return n
return fib(n - 1) + fib(n - 2)
In this example, we have a recursive algorithm for fipolacci Numbers. We know that this recursion is pretty inefficient because it repeats. For example, we want to calculate fib(5), so it is decomposed into fib(4) +fib(3), and fib(4) is decomposed into fib(3)+fib(2), and fib(3) is decomposed into fib(2)+fib(1)... And you can see, basically, that fib of 3, fib of 2, and fib of 1 are called twice in the whole recursion.
Instead, we use the decorator, which queries the cache before calling the function, and returns the value from the cache if there is none. All of a sudden, this recursion goes from being a binary tree recursion to being a linear recursion.
An example of Profiler
There's nothing fancy about this example, it's just practical.
import cProfile, pstats, StringIO
def profiler(func):
def wrapper(*args, **kwargs):
datafn = func.__name__ + ".profile" # Name the data file
prof = cProfile.Profile()
retval = prof.runcall(func, *args, **kwargs)
#prof.dump_stats(datafn)
s = StringIO.StringIO()
sortby = 'cumulative'
ps = pstats.Stats(prof, stream=s).sort_stats(sortby)
ps.print_stats()
print s.getvalue()
return retval
return wrapper
Register the callback function
The following example shows an example of a function that invokes the relevant registration via the URL path:
class MyApp():
def __init__(self):
self.func_map = {}
def register(self, name):
def func_wrapper(func):
self.func_map[name] = func
return func
return func_wrapper
def call_method(self, name=None):
func = self.func_map.get(name, None)
if func is None:
raise Exception("No function registered against - " + str(name))
return func()
app = MyApp()
@app.register('/')
def main_page_func():
return "This is the main page."
@app.register('/next_page')
def next_page_func():
return "This is the next page."
print app.call_method('/')
print app.call_method('/next_page')
Note:
1) in the above example, use an instance of the class to do the decorator.
2) there is no s/s/s () in the decorator class, but the wrapper returns the function. So, nothing happened to the function.
Log the function
The following example demonstrates a decorator for logger that prints the function name, parameters, return value, and run time.
from functools import wraps
def logger(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
ts = time.time()
result = fn(*args, **kwargs)
te = time.time()
print "function = {0}".format(fn.__name__)
print " arguments = {0} {1}".format(args, kwargs)
print " return = {0}".format(result)
print " time = %.6f sec" % (te-ts)
return result
return wrapper
@logger
def multipy(x, y):
return x * y
@logger
def sum_num(n):
s = 0
for i in xrange(n+1):
s += i
return s
print multipy(2, 10)
print sum_num(100)
print sum_num(10000000)
The log above is still a little rough, let's look at a better one (with log level parameters) :
import inspect
def get_line_number():
return inspect.currentframe().f_back.f_back.f_lineno
def logger(loglevel):
def log_decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
ts = time.time()
result = fn(*args, **kwargs)
te = time.time()
print "function = " + fn.__name__,
print " arguments = {0} {1}".format(args, kwargs)
print " return = {0}".format(result)
print " time = %.6f sec" % (te-ts)
if (loglevel == 'debug'):
print " called_from_line : " + str(get_line_number())
return result
return wrapper
return log_decorator
However, there are two things wrong with this one with the log level argument,
1) when the loglevel is not debug, you still need to calculate the time of function call.
2) different levels should be written together, not easy to read.
Let's improve it:
import inspect
def advance_logger(loglevel):
def get_line_number():
return inspect.currentframe().f_back.f_back.f_lineno
def _basic_log(fn, result, *args, **kwargs):
print "function = " + fn.__name__,
print " arguments = {0} {1}".format(args, kwargs)
print " return = {0}".format(result)
def info_log_decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
result = fn(*args, **kwargs)
_basic_log(fn, result, args, kwargs)
return wrapper
def debug_log_decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
ts = time.time()
result = fn(*args, **kwargs)
te = time.time()
_basic_log(fn, result, args, kwargs)
print " time = %.6f sec" % (te-ts)
print " called_from_line : " + str(get_line_number())
return wrapper
if loglevel is "debug":
return debug_log_decorator
else:
return info_log_decorator
You can see two things,
1) we divided two log levels, one for info and one for debug, and then we returned different decorators on the outer tail according to different parameters.
2) we extracted the same code from info and debug into a function called _basic_log, the DRY principle.
A Decorator for MySQL
The following decorator is the code I use in my work. I've simplified it a bit by removing the DB connection pool code to make it easier to read.
import umysql
from functools import wraps
class Configuraion:
def __init__(self, env):
if env == "Prod":
self.host = "coolshell.cn"
self.port = 3306
self.db = "coolshell"
self.user = "coolshell"
self.passwd = "fuckgfw"
elif env == "Test":
self.host = 'localhost'
self.port = 3300
self.user = 'coolshell'
self.db = 'coolshell'
self.passwd = 'fuckgfw'
def mysql(sql):
_conf = Configuraion(env="Prod")
def on_sql_error(err):
print err
sys.exit(-1)
def handle_sql_result(rs):
if rs.rows > 0:
fieldnames = [f[0] for f in rs.fields]
return [dict(zip(fieldnames, r)) for r in rs.rows]
else:
return []
def decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
mysqlconn = umysql.Connection()
mysqlconn.settimeout(5)
mysqlconn.connect(_conf.host, _conf.port, _conf.user,
_conf.passwd, _conf.db, True, 'utf8')
try:
rs = mysqlconn.query(sql, {})
except umysql.Error as e:
on_sql_error(e)
data = handle_sql_result(rs)
kwargs["data"] = data
result = fn(*args, **kwargs)
mysqlconn.close()
return result
return wrapper
return decorator
@mysql(sql = "select * from coolshell" )
def get_coolshell(data):
... ...
... ..
Thread asynchronous
Here is a very simple decorator for asynchronous execution. Note that asynchronous processing is not easy. Here is an example.
from threading import Thread
from functools import wraps
def async(func):
@wraps(func)
def async_func(*args, **kwargs):
func_hl = Thread(target = func, args = args, kwargs = kwargs)
func_hl.start()
return func_hl
return async_func
if __name__ == '__main__':
from time import sleep
@async
def print_somedata():
print 'starting print_somedata'
sleep(2)
print 'print_somedata: 2 sec passed'
sleep(2)
print 'print_somedata: 2 sec passed'
sleep(2)
print 'finished print_somedata'
def main():
print_somedata()
print 'back in main'
print_somedata()
print 'back in main'
main()
Although this article is very long, but is very practical, very basic knowledge, I hope you can be patient to finish.