An in depth understanding of the metaclass of metaclass in Python

  • 2020-04-02 14:36:33
  • OfStack

This is a hot post on Stack overflow. The questioner claimed to have mastered concepts about Python OOP programming, but still found metaclass hard to understand. He knew it must be about introspection, but he still didn't quite understand it, and he wanted you to give some practical examples and code snippets to help you understand, and when metaprogramming is needed. E-satis responded with a divine response, which earned 985 points of approval, and a comment that it should be included in Python's official documentation. And e-satis's reputation on Stack Overflow is 64,271. Here's a great response (hint: very long)

Classes are also objects

Before you can understand metaclasses, you need to master classes in Python. The concept of classes in Python was borrowed from Smalltalk, which is a bit odd. In most programming languages, a class is a set of code snippets that describe how to generate an object. This is still true in Python:


>>> class ObjectCreator(object):
...        pass
...
>>> my_object = ObjectCreator()
>>> print my_object
<__main__.ObjectCreator object at 0x8974f2c>

However, classes in Python go far beyond that. A class is also an object. Yes, that's right, objects. As long as you use the keyword class, the Python interpreter creates an object during execution. The following code snippet:

>>> class ObjectCreator(object):
...        pass
...

An object is created in memory, the name of which is ObjectCreator. The object (class) itself has the ability to create an object (class instance), which is why it is a class. However, its essence is still an object, so you can do the following with it:

1. You can assign it to a variable
You can copy it
3. You can add attributes to it
4. You can pass it as a function parameter

Here's an example:


>>> print ObjectCreator     # You can print a class because it's actually an object
<class '__main__.ObjectCreator'>
>>> def echo(o):
...        print o
...
>>> echo(ObjectCreator)                 # You can pass a class as an argument to a function
<class '__main__.ObjectCreator'>
>>> print hasattr(ObjectCreator, 'new_attribute')
Fasle
>>> ObjectCreator.new_attribute = 'foo' #  You can add attributes to a class
>>> print hasattr(ObjectCreator, 'new_attribute')
True
>>> print ObjectCreator.new_attribute
foo
>>> ObjectCreatorMirror = ObjectCreator # You can assign a class to a variable
>>> print ObjectCreatorMirror()
<__main__.ObjectCreator object at 0x8997b4c>

Create classes dynamically

Since classes are also objects, you can create them dynamically at runtime, just like any other object. First, you can create a class in a function, using the class keyword.


>>> def choose_class(name):
...        if name == 'foo':
...            class Foo(object):
...                pass
...            return Foo     # Returns a class, not an instance of the class
...        else:
...            class Bar(object):
...                pass
...            return Bar
...
>>> MyClass = choose_class('foo')
>>> print MyClass              # The function returns a class, not an instance of the class
<class '__main__'.Foo>
>>> print MyClass()            # You can create class instances, called objects, from this class
<__main__.Foo object at 0x89c6d4c>

But that's not dynamic enough, because you still have to code the entire class yourself. Since classes are also objects, they must be generated by something. The Python interpreter automatically creates this object when you use the class keyword. But like most things in Python, Python still gives you a way to do it manually. Remember the built-in function type? This ancient but powerful function lets you know what the type of an object is, like this:


>>> print type(1)
<type 'int'>
>>> print type("1")
<type 'str'>
>>> print type(ObjectCreator)
<type 'type'>
>>> print type(ObjectCreator())
<class '__main__.ObjectCreator'>

Here, type has a completely different capability, which also creates classes dynamically. Type can take a description of a class as an argument and then return a class. (I know it's silly to have two completely different USES for the same function, depending on the parameters passed in, but this is for backward compatibility in Python.)

Type can work like this:


type( The name of the class , Tuples of the parent class (which can be null in the case of inheritance), containing dictionaries of attributes (names and values) )

For example, the following code:


>>> class MyShinyClass(object):
...        pass

You can manually create it like this:

>>> MyShinyClass = type('MyShinyClass', (), {})  # Returns a class object
>>> print MyShinyClass
<class '__main__.MyShinyClass'>
>>> print MyShinyClass()  #  Create an instance of this class
<__main__.MyShinyClass object at 0x8997cec>

You'll find that we use "MyShinyClass" as the class name and can also use it as a variable as a reference to the class. Classes and variables are different, and there's no reason to complicate things here.

Type accepts a dictionary to define properties for the class, therefore


>>> class Foo(object):
...        bar = True

Can be translated as:

>>> Foo = type('Foo', (), {'bar':True})

And you can use Foo as a normal class:

>>> print Foo
<class '__main__.Foo'>
>>> print Foo.bar
True
>>> f = Foo()
>>> print f
<__main__.Foo object at 0x8a9b84c>
>>> print f.bar
True

Of course, you can inherit from this class, so here's the code:

>>> class FooChild(Foo):
...        pass

Can be written as:

>>> FooChild = type('FooChild', (Foo,),{})
>>> print FooChild
<class '__main__.FooChild'>
>>> print FooChild.bar   # bar Attributes are by Foo inherited
True

Eventually you'll want to add methods to your class. Just define a function with the proper signature and assign it as a property.

>>> def echo_bar(self):
...        print self.bar
...
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
False
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True

As you can see, in Python, classes are also objects, and you can create classes dynamically. This is what Python does behind the scenes when you use the keyword class, and this is done through metaclasses.

What exactly is a metaclass

Metaclasses are the "things" used to create classes. You create classes to create instance objects of classes, right? But we've learned that classes in Python are also objects. Well, metaclasses are what create these classes. Metaclasses are classes of classes.


MyClass = MetaClass()
MyObject = MyClass()

You've seen that type allows you to do something like this:

MyClass = type('MyClass', (), {})

This is because the function type is actually a metaclass. Type is the metaclass that Python USES behind the scenes to create all the classes. Now you wonder why type is all lowercase instead of type? Well, I guess this is to be consistent with STR, which is the class that creates string objects, and int, which is the class that creates integer objects. Type is the class that creates the class object. You can see this by checking the class attribute. Everything in Python, notice, I mean everything -- all objects. This includes integers, strings, functions, and classes. They are all objects, and they are all created from a class.

>>> age = 35
>>> age.__class__
<type 'int'>
>>> name = 'bob'
>>> name.__class__
<type 'str'>
>>> def foo(): pass
>>>foo.__class__
<type 'function'>
>>> class Bar(object): pass
>>> b = Bar()
>>> b.__class__
<class '__main__.Bar'>

Now, what is the class attribute for any class?

>>> a.__class__.__class__
<type 'type'>
>>> age.__class__.__class__
<type 'type'>
>>> foo.__class__.__class__
<type 'type'>
>>> b.__class__.__class__
<type 'type'>

So metaclasses are the things that create objects like classes. If you like, you can call the metaclass a "class factory" (not to be confused with the factory class :D).

Metaclass property

You can add the metaclass attribute to aclass when you write it.


class Foo(object):
    __metaclass__ = something ...
[ ... ]

If you do that, Python will use the metaclass to create class Foo. Be careful. There's a trick here. You write class Foo(object) first, but the class object Foo hasn't been created in memory yet. Python will look for the metaclass attribute in the definition of the class, and if it is found, Python will use it to create the class Foo, and if it is not found, it will create the class with the built-in type. Repeat the following passage several times. When you write the following code:

class Foo(Bar):
    pass

Python does the following:

Is there a metaclass property in Foo? If so, Python creates aclass object called Foo in memory using metaclass (I'm talking about class objects, follow my lead). If Python does not find the metaclass, it continues to look for the metaclass attribute in the Bar (parent class) and tries to do the same thing. If Python cannot find the metaclass in any of the parent classes, it will look for the metaclass in the module hierarchy and try to do the same. If the metaclass is still not found,Python creates the class object with the built-in type.

Now the question is, what code can you put in your metaclass? The answer is: you can create a class thing. So what can be used to create a class? Type, or anything that USES type or subclasses type.

Custom metaclasses
The main purpose of metaclasses is to automatically change classes when they are created. Typically, you do this for an API, and you want to be able to create classes that match the current context. Imagine a silly example where you decide that all class properties in your module should be capitalized. There are several ways to do this, but one of them is by setting the metaclass at the module level. In this way, all the classes in the module will be created by this metaclass, so we just need to tell the metaclass to capitalize all the properties.

Fortunately, metaclass can actually be called arbitrarily, and it doesn't have to be a formal class (I know that something with a 'class' in its name doesn't have to be aclass, so it helps to draw a picture). So let's start with a simple function.


# Metaclass will automatically pass you to ' type' Is passed as its own parameter
def upper_attr(future_class_name, future_class_parents, future_class_attr):
    ''' Returns a class object that capitalizes the properties '''
    #  Choose none at all '__' Initial attribute
    attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
  # I'm going to capitalize them
    uppercase_attr = dict((name.upper(), value) for name, value in attrs)     # through 'type' To do the class object creation
    return type(future_class_name, future_class_parents, uppercase_attr) __metaclass__ = upper_attr  #  This applies to all classes in the module class Foo(object):
    # Or we could just define it here __metaclass__ , so it only works on this class
    bar = 'bip' print hasattr(Foo, 'bar')
# The output : False
print hasattr(Foo, 'BAR')
# The output :True f = Foo()
print f.BAR
# The output :'bip'

Now let's do it again, this time with a real class as the metaclass.


# Please remember, 'type' It's actually a class, like 'str' and 'int' The same
# So, you can go from type inheritance
class UpperAttrMetaClass(type):
    # __new__ Is in the __init__ The special method that was called earlier
    # __new__ Is a method used to create an object and return it
    # while __init__ It is simply used to initialize the incoming parameter to the object
    # You seldom use it __new__ Unless you want to be able to control object creation
    # In this case, the object we're creating is a class, and we want to be able to customize it, so we're rewriting it here __new__
    # You can be there if you want __init__ Do something about it
    # There are also some advanced USES that involve rewriting __call__ Special method, but we don't use it here
    def __new__(upperattr_metaclass, future_class_name, future_class_parents, future_class_attr):
        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)
        return type(future_class_name, future_class_parents, uppercase_attr)

However, this approach is not really OOP. We call type directly, and we don't overwrite the parent class's new method. Now let's do this:


class UpperAttrMetaclass(type):
    def __new__(upperattr_metaclass, future_class_name, future_class_parents, future_class_attr):
        attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)         # reuse type.__new__ methods
        # That's the basics OOP Programming, no magic
        return type.__new__(upperattr_metaclass, future_class_name, future_class_parents, uppercase_attr)

You may have noticed that there is an extra parameter, upperattr_metaclass, which is nothing special. The first argument to a class method always represents the current instance, just like the self argument in a normal class method. And, of course, for the sake of clarity, I'm going to have a long name here. But just like self, all arguments have their traditional names. Therefore, in real production code a metaclass would look like this:

class UpperAttrMetaclass(type):
    def __new__(cls, name, bases, dct):
        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__')
        uppercase_attr  = dict((name.upper(), value) for name, value in attrs)
        return type.__new__(cls, name, bases, uppercase_attr)

If we use the super method, we can also make it a little clearer, which will ease inheritance (yes, you can have metaclasses, inherit from metaclasses, inherit from type).


class UpperAttrMetaclass(type):
    def __new__(cls, name, bases, dct):
        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))
        uppercase_attr = dict((name.upper(), value) for name, value in attrs)
        return super(UpperAttrMetaclass, cls).__new__(cls, name, bases, uppercase_attr)

That's it. There's really nothing else to say about metaclasses. The reason behind the complexity of using metaclasses is not because of the metaclasses themselves, but because you often use metaclasses to do obscure things, rely on introspection, control inheritance, and so on. Indeed, using metaclasses to do some "dark magic" is particularly useful, and can lead to something complex. But as far as metaclasses go, they're pretty simple:

1. Create the interceptor class
2. Modify the class
3. Returns the modified class

Why use the metaclass class instead of the function ?

Since metaclass can accept any callable object, why use aclass, since it's obviously more complicated to use aclass? There are several reasons:

1. The intent will be clearer. When you read UpperAttrMetaclass(type), you know what to expect.
You can program using OOP. Metaclasses can be inherited from metaclasses and overwrite parent methods. Metaclasses can even be used.
3. You can organize your code better. When you use metaclasses it's not a simple scenario like the one I mentioned above, it's usually for more complex problems. Lumping multiple methods into a single class is helpful and makes the code easier to read.
4. You can use special methods like new, init, and call. They can help you with different tasks. Even though you can usually get rid of everything in new, some people are more comfortable with init.
5. Wow, the name of this thing is "metaclass".

Why use metaclasses at all?

Now back to our big topic, why on earth would you use such an error-prone and obscure feature? Well, generally speaking, you don't use it:

Metaclass is deep magic, and 99% of users shouldn't have to worry about it at all. If you want to know if you need metaclasses, you don't need them. Those who actually use metaclasses know exactly what they need to do and don't need to explain why. -- Tim Peters, leader of the Python community

The main use of metaclasses is to create apis. A typical example is Django ORM. It allows you to define it like this:


class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

But if you do it like this:


guy  = Person(name='bob', age='35')
print guy.age

Instead of returning an IntegerField object, this returns an int and can even fetch data directly from the database. This is possible because models.model defines the metaclass and USES some magic to transform the simple Person class you just defined into a complex hook for the database. The Django framework simplifies these seemingly complex things by exposing a simple API that USES metaclasses, recreates the code, and does the real work behind the scenes.

conclusion

First, you know that a class is actually an object that can create an instance of a class. Well, in fact, classes themselves are instances, of course, they are instances of metaclasses.


>>>class Foo(object): pass
>>> id(Foo)
142630324

Everything in Python is an object, either an instance of a class or an instance of a metaclass, except for type. Type is actually its own metaclass, which is not something you can do in a pure Python environment, but it's done by playing around with the implementation level. Second, metaclasses are complex. For very simple classes, you may not want to modify the class by using metaclasses. You can modify classes using two other techniques:

Monkey patching
class decorators

When you need to dynamically modify a class, you're better off using both of these techniques 99% of the time. Of course, 99% of the time you don't need to change classes dynamically at all.


Related articles: