Deep understanding of magical methods in Python

I have been exposed to Python for a long time, and I have been exposed to a lot of frameworks and modules related to Python. I hope I can share with you the good design and implementation I have come across. Therefore, I chose a small standard of Charming Python, which is a start for me. :)

The from flask import request

Flask is a popular Python Web framework, and I've written about it for a number of projects, large and small. One of the features of Flask that I really like is that if you want to get the current request object anywhere, you can simply:

from flask import request
#  From the current request Access to content 

request.args

request.forms

request.cookies

... ...

Very simple to remember and very friendly to use. However, the implementation behind the simplicity is slightly more complex. Follow my article to find out why.

Two questions?

Before we move on, let's ask two questions:

Question 1: request looks just like a static class instance. Why can we directly use such an expression as request-args to get the args property of the current request, instead of using such an expression as:

from flask import get_request
#  Gets the current request

request = get_request()

get_request().args

How about this? How does flask correspond the request to the current request object?

Question2: in a real production environment, there might be many threads (or coroutae) under the same worker process. As I just said, how does an instance of the class request work in such an environment?

To know the secret, we have to start with the source code for flask.

Source code, source code, source code

First of all, open the flask's source code and see how the request comes out, starting with the original flask.

# File: flask/__init__.py

from .globals import current_app, g, request, session, _request_ctx_stack


# File: flask/globals.py

from functools import partial

from werkzeug.local import LocalStack, LocalProxy


def _lookup_req_object(name):

    top = _request_ctx_stack.top

    if top is None:

        raise RuntimeError('working outside of request context')

    return getattr(top, name)
# context locals

_request_ctx_stack = LocalStack()

request = LocalProxy(partial(_lookup_req_object, 'request'))

Flask's request is introduced from globals.py, and the code for a request is request = LocalProxy(partial(_lookup_req_object, 'request')).

However, we can simply understand partial(func, 'request') as using 'request' as the first default parameter of func to generate another function.

So partial(_lookup_req_object, 'request') can be interpreted as:

Generate a callable function, which mainly gets the first RequestContext object at the top of the stack from the LocalStack object _request_ctx_stack, and then returns the request attribute of this object.

The LocalProxy under werkzeug caught our attention, so let's take a look at what it is:

@implements_bool

class LocalProxy(object):

    """Acts as a proxy for a werkzeug local.  Forwards all operations to

    a proxied object.  The only operations not supported for forwarding

    are right handed operands and any kind of assignment.

    ... ...

As the name suggests, LocalProxy is mainly a Proxy, a Proxy that serves werkzeug's Local object. He "forwards" all of his actions to the object it represents.

So how is the Proxy implemented in Python? The answer is in the source code:

#  I've made some cuts and changes to the code for illustration purposes 
@implements_bool

class LocalProxy(object):

    __slots__ = ('__local', '__dict__', '__name__')
    def __init__(self, local, name=None):

        #  There's one point here that needs to be noted, and it's passed __setattr__ Method, self the 

        # "_LocalProxy__local"  Property is set to local You may wonder 

        #  Why is the name of this property so strange Python No real support 

        # Private member , please refer to the official documents: 

        # http://docs.python.org/2/tutorial/classes.html#private-variables-and-class-local-references

        #  Here you just have to treat it as  self.__local = local  It is ok  :)

        object.__setattr__(self, '_LocalProxy__local', local)

        object.__setattr__(self, '__name__', name)
    def _get_current_object(self):

        """

         Get the real object that is currently being proffered. This method is not normally invoked unless you are 

         For some performance reason you need to get the real object that is being proxied, or you need to use it for something else 

         Place. 

        """

        #  The main purpose here is to determine whether the object of the proxy is a werkzeug the Local Object in our analysis request

        #  I'm not going to use this logic. 

        if not hasattr(self.__local, '__release_local__'):

            #  from LocalProxy(partial(_lookup_req_object, 'request')) It seems 

            #  By calling the self.__local() Method, we get  partial(_lookup_req_object, 'request')()

            #  That is  ``_request_ctx_stack.top.request``

            return self.__local()

        try:

            return getattr(self.__local, self.__name__)

        except AttributeError:

            raise RuntimeError('no object bound to %s' % self.__name__)
    #  And then there are long paragraphs Python The magic method, Local Proxy Reload the ( almost )? all Python

    #  Built - in magic method to make all about himself operations All point to _get_current_object()

    #  The returned object is the real proxied object. 
    ... ...

    __setattr__ = lambda x, n, v: setattr(x._get_current_object(), n, v)

    __delattr__ = lambda x, n: delattr(x._get_current_object(), n)

    __str__ = lambda x: str(x._get_current_object())

    __lt__ = lambda x, o: x._get_current_object() < o

    __le__ = lambda x, o: x._get_current_object() <= o

    __eq__ = lambda x, o: x._get_current_object() == o

    __ne__ = lambda x, o: x._get_current_object() != o

    __gt__ = lambda x, o: x._get_current_object() > o

    __ge__ = lambda x, o: x._get_current_object() >= o

    ... ...

This is where our second question at the beginning of the article is answered, and it's all thanks to LocalProxy that we don't need a method call like get_request() to get the current request object.

LocalProxy ACTS as a proxy, using custom magic methods. Proxy all of our operations on the request to point to the real request object.

So, now that you know request-args is not as simple as it looks.

Now, let's look at the second question, how does request work in a multi-threaded environment? Let's go back to globals.py:

from functools import partial

from werkzeug.local import LocalStack, LocalProxy


def _lookup_req_object(name):

    top = _request_ctx_stack.top

    if top is None:

        raise RuntimeError('working outside of request context')

    return getattr(top, name)
# context locals

_request_ctx_stack = LocalStack()

request = LocalProxy(partial(_lookup_req_object, 'request'))

The key to the problem is the _request_ctx_stack object, let's find the source of LocalStack:

class LocalStack(object):
    def __init__(self):

        #  Actually, LocalStack I used the other one mainly Local class 

        #  Some of its key methods are also proxied to this Local On the class 

        #  Relative to the Local Class, it implements a little bit more and stack. Stack "Related methods, for example push , pop Such as 

        #  So, we just have to look Local The code will do 

        self._local = Local()
    ... ...
    @property

    def top(self):

        """

         Returns the object at the top of the stack 

        """

        try:

            return self._local.stack[-1]

        except (AttributeError, IndexError):

            return None


#  So, when we call _request_ctx_stack.top Is actually called  _request_ctx_stack._local.stack[-1]

#  Let's see Local How is the class implemented, but before we do that we have to look at the following get_ident methods 
#  First try to follow greenlet The import getcurrent Method, because if flask Run in the like gevent When you're in this container 

#  All the requests are greenlet As the smallest unit, instead of thread Threads. 

try:

    from greenlet import getcurrent as get_ident

except ImportError:

    try:

        from thread import get_ident

    except ImportError:

        from _thread import get_ident
#  Anyway, this get_ident Method will return the current coroutine / thread ID , which is unique for each request 


class Local(object):

    __slots__ = ('__storage__', '__ident_func__')
    def __init__(self):

        object.__setattr__(self, '__storage__', {})

        object.__setattr__(self, '__ident_func__', get_ident)
    ... ...
    #  The point is this Local The class reloading __getattr__ and __setattr__ These two magic methods 
    def __getattr__(self, name):

        try:

            #  Here we return the call self.__ident_func__() Which is the only one at the moment ID

            #  As a __storage__ the key

            return self.__storage__[self.__ident_func__()][name]

        except KeyError:

            raise AttributeError(name)
    def __setattr__(self, name, value):

        ident = self.__ident_func__()

        storage = self.__storage__

        try:

            storage[ident][name] = value

        except KeyError:

            storage[ident] = {name: value}
    ... ...
    #  After overloading the two magic methods 
    # Local().some_value  It's not as easy as it looks :

    #  So let's call it first get_ident Method to get the currently running thread / coroutines ID

    #  And then get this ID Under the space of some_value Properties, like this: 

    #

    #   Local().some_value -> Local()[current_thread_id()].some_value

    #

    #  The same goes for setting properties 

Through this analysis, it is believed that query 2 has also been resolved. Flask has implemented its own stack object that is used by different worker threads by using the current thread/coroutine ID and overloading some magic methods. This ensures that the request works.

So much for this article. As you can see, there's a lot of extra work that goes into being a developer of frameworks and tools for the convenience of the user, and sometimes it's inevitable to use some of the magic of the language, and Python has great support for that.

All we need to do is learn the magic parts of Python and use the magic to make our code cleaner and easier to use.

But remember, magic is cool, don't abuse it.