Python metaclass Details of Dark Magic

Directory 1. What is metaclass 2. What problems can metaclass solve? 3. Understand metaclass 4 through an example. How does the underlying language design level of Python realize metaclass? 1. All Python user-defined classes are instances of type. 2. User-defined class, just the __call__ operator overload of type class 3, "Beyond deformation" normal class 4. Risk of using metaclass

About Python Dark magic metaclass Two extreme views of:

This feature is too cow force, is the omnipotent Aladdin magic lamp, must find opportunities to use to show their own Python Strength. This feature is too dangerous, will be demagogic to abuse, 1 once open will release demons, so that the code is difficult to maintain.

Let's take a look today. metaclass Is it Aladdin's magic lamp or Pandora's box?

1. What is metaclass

Many books will be translated into metaclass. Literally, meta is indeed metaclass and origin, and there is nothing wrong with translation. But when we understand it, we should understand meta as transcendental data describing data. In fact, metaclass meta originated from Greek vocabulary meta Contains two meanings:

" Beyond "For example, technical vocabulary metadata It means describing the transcendental data of data. " Change "For example, technical vocabulary metamorphosis It means a changed form.

Therefore, it can be understood as metaclass In order to describe the superclass of a class, the form of subclasses can be changed at the same time. You may ask, is this similar to the definition of metadata, and what is the use of this feature in programming?

It's very useful. In no metaclass The child class inherits the parent class, and the parent class cannot perform operations on the child class, but with the metaclass 2 You can manipulate subclasses, just like a decorator can dynamically customize and modify decorated classes, metaclass Subclasses that inherit it can be dynamically customized or modified.

2. What problems can metaclass solve?

You already know metaclass You can customize and modify subclasses that inherit it like a decorator, so here's what practical problems it can solve. For example, in a large-scale project of intelligent voice assistant, we have 10,000 voice conversation scenes, and each scene is developed by different teams. As a core team member of intelligent voice assistant, it is impossible for you to know the implementation details of each sub-scenario.

When dynamically configuring different experimental scenes, it is often necessary to experiment with the configuration of A and B today, and experiment with the configuration of B and C tomorrow. The optical configuration file has tens of thousands of lines, and the workload is not small. With this dynamic configuration concept, I can let the engine dynamically load the required ones according to my text configuration file Python Class.

If you're not quite sure, then metaclass 6 You should know that it is a well-known Python tool that can easily serialize and deserialize data. metaclass 7 You can have any of its subclasses support serialization and deserialization ( metaclass 8 ).

Serialization and deserialization:

Serialization: When the program runs, all variables or objects are stored in memory. Once the program is called, the memory occupied by these variables or objects will be reclaimed. In order to realize the persistent storage of variables and objects to disk or transmission on the network, we need to convert variables or objects into binary streams. The process of converting it into binary stream is serialization. Deserialization: Deserialization means that when the program runs, it cannot be read from the disk. It needs to transfer the serialized objects or variables from the disk to the memory, and at the same time, it will convert the binary stream into the original data format. We call this 1 process deserialization.

Now you have 10,000 YAML profiles in different formats. You would have written 10,000 classes to load these profiles, and you have metaclass 2 You only need to implement 1 metaclass Superclass, and then implement a subclass to inherit this metaclass 2 You can automatically pull different classes according to different configuration files, which greatly improves efficiency.

3. Understanding metaclass with an example

Please manually enter the Python 2 Make code, look at each step of the output of what, so that you can thoroughly understand the class creation and instantiation steps.

In[15]: class Mymeta(type):
   ...:     def __init__(self, name, bases, dic):
   ...:         super().__init__(name, bases, dic)
   ...:         print('===>Mymeta.__init__')
   ...:         print(self.__name__)
   ...:         print(dic)
   ...:         print(self.yaml_tag)
   ...: 
   ...:     def __new__(cls, *args, **kwargs):
   ...:         print('===>Mymeta.__new__')
   ...:         print(cls.__name__)
   ...:         return type.__new__(cls, *args, **kwargs)
   ...: 
   ...:     def __call__(cls, *args, **kwargs):
   ...:         print('===>Mymeta.__call__')
   ...:         obj = cls.__new__(cls)
   ...:         cls.__init__(cls, *args, **kwargs)
   ...:         return obj
   ...: 
In[16]: 
In[16]: 
In[16]: class Foo(metaclass=Mymeta):
   ...:     yaml_tag = '!Foo'
   ...: 
   ...:     def __init__(self, name):
   ...:         print('Foo.__init__')
   ...:         self.name = name
   ...: 
   ...:     def __new__(cls, *args, **kwargs):
   ...:         print('Foo.__new__')
   ...:         return object.__new__(cls)
   ...:     
===>Mymeta.__new__
Mymeta
===>Mymeta.__init__
Foo
{'__module__': '__main__', '__qualname__': 'Foo', 'yaml_tag': '!Foo', '__init__': <function Foo.__init__ at 0x0000000007EF3828>, '__new__': <function Foo.__new__ at 0x0000000007EF3558>}
!Foo

In[17]: foo = Foo('foo')
===>Mymeta.__call__
Foo.__new__
Foo.__init__

In[18]:

From the above running results, we can find that in the definition Python 3 When defined, the Python 4 Adj. Python 5 And Python 6 Method construction Python 7 Class, and then call the Python 8 When an instance object of the class is created, the Python 9 Method to call the metaclass 0 Class metaclass 1 And metaclass 2 Law.

After running the above example, we will understand a lot. Normally, we can't operate on the attributes of subclasses in the parent class, but metaclasses can. To understand in a different way: 元类、装饰器、类装饰器都可以归为元编程。

4. How does the underlying language design level of Python implement metaclass?

To understand metaclass You need to understand the underlying principles of Python Type model. Below, it will be explained in three points.

1. All user-defined classes of Python are instances of the class type.

It may surprise you that the class itself is, in fact, nothing more than an instance of a class named type. In the type world of Python, the class type is the God of creation. This can be verified in code:

In [2]: # Python 3 And Python 2 Similar 
   ...: class MyClass:
   ...:   pass
   ...:
   ...: instance = MyClass()
   ...:
in [3]: type(instance)
   ...:
Out[2]: __main__.MyClass
In [4]: type(MyClass)
   ...:
Out[4]: type
In [5]:

As you can see, instance Yes MyClass And MyClass is just "God" type Gets or sets an instance of the.

2. User-defined class, just __call__ operator overload of type class

What really happens when our statement defining a class ends is that Python calls type __call__ Operator. Simply put, when you define a class, write it as follows:

class MyClass:
    data = 1

What Python really executes is the following code:

class = type(classname, superclasses, attributedict)

The one to the right of the equal sign here type(classname, superclasses, attributedict)， Is the type __call__ Operator, it calls in one step:

type.__new__(typeclass, classname, superclasses, attributedict)
type.__init__(class, classname, superclasses, attributedict)

Of course, these 1 cuts can be verified by code, such as

In [5]: class MyClass:
   ...:     data = 1
   ...:
   ...: instance = MyClass()
   ...:

In [6]: MyClass, instance
   ...:
Out[6]: (__main__.MyClass, <__main__.MyClass at 0x4ef5188>)

In [7]: instance.data
   ...:
Out[7]: 1

In [8]: MyClass = type('MyClass', (), {'data': 1})
   ...: instance = MyClass()
   ...:

In [9]: MyClass, instance
   ...:
Out[9]: (__main__.MyClass, <__main__.MyClass at 0x4f40748>)

In [10]: instance.data
    ...:
Out[10]: 1

In [11]:

It can be seen that normal MyClass Definition, and you manually call it type The result of the operator is exactly 1-like.

3. "Beyond deformation" is a normal class

metaclass is a subclass of type by replacing type's __call__ Operator overloading mechanism, "beyond deformation" normal classes

In fact, after understanding the above points, we will understand that it is the class creation mechanism of Python that gives metaclass the opportunity to show its talents.

1 Once you put 1 type MyClass Adj. metaclass Set to MyMeta, MyClass is no longer created by the native type, but is called Python 4 Adj. __call__ Operator overload.

class = type(classname, superclasses, attributedict) 
#  Become 
class = MyMeta(classname, superclasses, attributedict)

4. Risks of using metaclass

However, everything has its advantages and disadvantages, especially metaclass The existence of such "rebellion". As you can see, metaclass Will "distort" the normal Python type model. Therefore, if used carelessly, the risk to the whole code base is immeasurable.

In other words, metaclass Just for a small part Python Developers, at the development framework level Python That is used when using the. In the application layer, metaclass Often not a good choice.

Summary:

This article creates the process from the Python class to help you understand metaclass The role of.

metaclass It's dark magic. If it's used properly, it's heaven, and if it's used properly, it's hell.