Introduction to metaclass programming in Python

Review object-oriented programming

Let's review for 30 seconds what OOP really is. In object-oriented programming languages, classes can be defined for the purpose of bundling related data and behavior in one place. These classes can inherit some or all of the properties of their parent classes, but they can also define their own properties (data) or methods (behavior). At the end of the process of defining a class, the class often ACTS as a template for creating an instance (sometimes simply called an object). Different instances of the same class usually have different data, but the "look" is the same -- for example, the Employee objects bob and jane have.salary and.room_number, but both have different rooms and salaries.

Some OOP languages (including Python) allow objects to be introspective (also known as reflection). That is, the introspection object can describe itself: which class does the instance belong to? What are the ancestors of a class? What methods and properties can an object use? Introspection lets the functions or methods that process an object make decisions based on the type of object passed to the function or method. Even without introspection, functions are often partitioned based on instance data; for example, the route to jane.room_number is different from the route to bob.room_number because they are in different rooms. With introspection, you can also safely calculate the bonus for jane while skipping the calculation for bob, for example, because jane has the.profit_share property, or because bob is an instance of the subclass Hourly(Employee).

Metaclass programming (metaprogramming)

The basic OOP system outlined above is quite powerful. But there is one element missing from the above description: in Python (and other languages), the class itself is an object that can be passed and introspective. As mentioned earlier, since you can use classes as templates to generate objects, what can you use as templates to generate classes? The answer, of course, is metaclasses (metaclass).

Python 1 has metaclasses. But the methods involved in metaclasses are better exposed in Python 2.2. Python V2.2 explicitly no longer USES just one special (usually hidden) metaclass to create each class object. Programmers can now create subclasses of the original metaclass type, and can even dynamically generate classes with various metaclasses. Of course, just because you can manipulate metaclasses in Python 2.2 doesn't mean you might want to.

Also, there is no need to use custom metaclasses to manipulate class generation. A less brainy concept is the class factory: a common function that returns classes that are dynamically created inside the body of a function. With the traditional Python syntax, you can write:
Listing 1. The old Python 1.5.2 class factory

Python 1.5.2 (#0, Jun 27 1999, 11:23:01) [...]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> def class_with_method(func):
...   class klass: pass
...   setattr(klass, func.__name__, func)
...   return klass
...
>>> def say_foo(self): print 'foo'
...
>>> Foo = class_with_method(say_foo)
>>> foo = Foo()
>>> foo.say_foo()
foo

The factory function class_with_method() dynamically creates a class and returns the class containing the methods/functions passed to the factory. Manipulate the class itself in the body of the function before returning the class. The new module provides a cleaner way to code, but the options are different from the options for customizing the code in the class factory, such as:
Listing 2. The class factory in the new module

>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo

In all of these cases, instead of writing the behavior of the class (Foo and Foo2) directly as code, you use dynamic parameters to call functions at runtime to create the class's behavior. One point to emphasize here is that not only instances can be created dynamically, but classes themselves can be created dynamically.

Metaclasses: looking for a solution to a problem?

The magic of the       metaclass is so great that 99% of users have had unnecessary concerns. If you want to know if you need them, you can get rid of them (those who actually need metaclasses do know they need them, and don't need to explain why). -- Python expert Tim Peters

Methods can return objects as normal function 1. So in that sense, class factories can be classes, just as easily as they can be functions 1, which is obvious. In particular, Python 2.2+ provides a special class called type, which is just such a class factory. Of course, the reader will recognize that type() is not as "ambitious" as the built-in functions in the older version of Python -- fortunately, the behavior of the older version of type() function is maintained by the type class (in other words, type(obj) returns the type/class of the object obj). The new type class, as a class factory, works in the same way as the function new.classobj 1 does:
Listing 3. type as the class factory metaclass

>>> X = type('X',(),{'foo':lambda self:'foo'})
>>> X, X().foo()
(<class '__main__.X'>, 'foo')

But since type is now a (meta) class, you're free to subclass it:
Listing 4. type descendants as class factories

>>> class ChattyType(type):
...   def __new__(cls, name, bases, dct):
...     print "Allocating memory for class", name
...     return type.__new__(cls, name, bases, dct)
...   def __init__(cls, name, bases, dct):
...     print "Init'ing (configuring) class", name
...     super(ChattyType, cls).__init__(name, bases, dct)
...
>>> X = ChattyType('X',(),{'foo':lambda self:'foo'})
Allocating memory for class X
Init'ing (configuring) class X
>>> X, X().foo()
(<class '__main__.X'>, 'foo')

The "magic".s 98en__ () and.s 99en__ () methods are special, but conceptually they work the same for any other class. .s 100en__ () method allows you to configure the objects you create; .s 101en__ () method allows you to customize its distribution. Of course, the latter is not widely used, but it does exist for every Python 2.2 new style class (usually through inheritance rather than overwriting).

One feature of type descendants should be noted. It often ensnares people who are using metaclasses for the first time. By convention, the first argument to these methods is called cls, not self, because the methods operate on the generated class, not on the metaclass. In fact, there's nothing special about it; All methods are attached to their instances, and instances of metaclasses are classes. Non-special names make this more obvious:
Listing 5. Attach the class method to the generated class

>>> class Printable(type):
...   def whoami(cls): print "I am a", cls.__name__
...
>>> Foo = Printable('Foo',(),{})
>>> Foo.whoami()
I am a Foo
>>> Printable.whoami()
Traceback (most recent call last):
TypeError: unbound method whoami() [...]

All of these surprising but common practices and easy-to-master syntax make metaclasses easier to use, but they also confuse new users. There are several elements for other syntaxes. But the parsing order of these new variants is tricky. A class can inherit a metaclass from its ancestor - note that this is not the same as having a metaclass as an ancestor (another area that is often confusing). For older classes, defining a global _metaclass_ variable forces the use of custom metaclasses. But most of the time, the safest approach is to set the _metaclass_ class attribute of a class when you want to create it by customizing the metaclass. Variables must be set in the class definition itself, because if properties are set later (after the class object has been created), metaclasses will not be used. Such as:
Listing 6. Setting the metaclass with class properties

>>> class Bar:
...   __metaclass__ = Printable
...   def foomethod(self): print 'foo'
...
>>> Bar.whoami()
I am a Bar
>>> Bar().foomethod()
foo

Use this "magic" to solve problems

So far, we have learned a bit about metaclasses. To use metaclasses, however, is more complex. The difficulty with metaclons is that, in general, in OOP designs, classes don't really do much. Class inheritance structures are useful for encapsulating and packaging data and methods, but in concrete cases, people often use instances.

We think metaclasses are really useful in two broad categories of programming tasks.

The first (and probably more common, class 1) is that you don't know exactly what the class needs to do at design time. Obviously, you know something about it, but a particular detail may depend on the information you get later. "Later" itself falls into two categories :(a) when an application USES a library module; (b) at runtime, when a situation exists. This is close to what is commonly referred to as "aspect-oriented programming (Aspect-Oriented Programming, AOP)." We'll show you an example that we think is very fancy:
Listing 7. Runtime metaclass configuration

% cat dump.py
#!/usr/bin/python
import sys
if len(sys.argv) > 2:
  module, metaklass = sys.argv[1:3]
  m = __import__(module, globals(), locals(), [metaklass])
  __metaclass__ = getattr(m, metaklass)
class Data:
  def __init__(self):
    self.num = 38
    self.lst = ['a','b','c']
    self.str = 'spam'
  dumps  = lambda self: `self`
  __str__ = lambda self: self.dumps()
data = Data()
print data
% dump.py
<__main__.Data instance at 1686a0>

As you might expect, the application prints out a fairly generic description of the data object (the generic instance object). But if you pass the runtime parameters to the application, you can get quite different results:
Listing 8. Adding an external serialization metaclass

% dump.py gnosis.magic MetaXMLPickler
<?xml version="1.0"?>
<!DOCTYPE PyObject SYSTEM "PyObjects.dtd">
<PyObject module="__main__" class="Data" id="720748">
<attr name="lst" type="list" id="980012" >
 <item type="string" value="a" />
 <item type="string" value="b" />
 <item type="string" value="c" />
</attr>
<attr name="num" type="numeric" value="38" />
<attr name="str" type="string" value="spam" />
</PyObject>

This particular example USES the serialization style of gnosis.xml.pickle, but the latest gnosis.magic package also contains metaclass serializers MetaYamlDump, MetaPyPickler, and MetaPrettyPrint. Furthermore, users of dump.py "applications" can take advantage of this "MetaPickler" from any MetaPickler package that defines any MetaPickler expectations. For this purpose, write the appropriate metaclass as follows:
Listing 9. Adding properties with metaclasses

class MetaPickler(type):
  "Metaclass for gnosis.xml.pickle serialization"
  def __init__(cls, name, bases, dict):
    from gnosis.xml.pickle import dumps
    super(MetaPickler, cls).__init__(name, bases, dict)
    setattr(cls, 'dumps', dumps)

The beauty of this arrangement is that the application programmer doesn't need to know which serialization to use -- or even whether to add serialization or other capabilities across the command line.

Perhaps the most common use of metaclasses is similar to MetaPickler: add, remove, rename, or replace methods defined in the generated class. In our example, when creating the class Data (and thus each subsequent instance), the "native" Data.dump () method is replaced by a method outside the application.

Use this "magic" to solve the problem in other ways

There are programming environments where classes are often more important than instances. For example, the declarative mini-language (declarative mini-languages) is the Python library, which directly represents its program logic in the class declaration. David studied this problem in his article "Create declarative mini-languages". In this case, it is quite useful to use metaclasses to influence the class creation process.

A class-based declarative framework is gnosis.xml.validity. Under this framework, you can declare a number of "validity classes" that represent a set of constraints on a valid XML document. These statements are very close to those contained in DTD. For example, you can configure an "dissertation" document with the following code:
Listing 10. simple_diss.py gnosis.xml.validity rule

from gnosis.xml.validity import *
class figure(EMPTY):   pass
class _mixedpara(Or):   _disjoins = (PCDATA, figure)
class paragraph(Some):  _type = _mixedpara
class title(PCDATA):   pass
class _paras(Some):    _type = paragraph
class chapter(Seq):    _order = (title, _paras)
class dissertation(Some): _type = chapter

If you try to instantiate the dissertation class without the correct component child element, a descriptive exception will be generated. The same is true for each child element. When there is only one clear way to "promote" a parameter to the correct type, the correct child element is generated from a simpler parameter.

Even though validity classes are often (informally) based on pre-existing DTD, instances of these classes print themselves into simple XML document fragments, such as:
Listing 11. Creation of a basic validity class document

>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo

0

By using metaclasses to create validity classes, we can generate DTD from class declarations (we can add an additional method to these validity classes while doing so) :
Listing 12. Leverage metaclasses during module import

>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo

1

The package gnosis.xml.validity does not know DTD and the internal subset. Those concepts and capabilities are completely introduced by the metaclass DTDGenerator, with no changes to gnosis.xml.validity or simple_diss.py. DTDGenerator does not replace its own.s 222en__ () method with the class it produces -- you can still print simple XML fragments -- but the metaclass can easily modify this "magic" method.

Yuan brings convenience

To use metaclasses and some sample metaclasses that can be used in aspect-oriented programming, package gnosis.magic contains several utilities. The most important of these utilities is import_with_metaclass(). The function used in the above example enables you to import the 3rd side module, but you create all the module classes using custom metaclasses instead of type. Whatever new capability you want to give to a third party module, you can define that capability in the metaclass you create (or get it from somewhere else). gnosis.magic contains 1 pluggable serialization metaclasses; Other packages may contain trace capabilities, object persistence, exception logging, or other capabilities.

The import_with_metclass() function shows several properties of metaclass programming:
Listing 13. import_with_metaclass() for [gnosis.magic]

>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo

2

The notable style in this function is to generate the normal class Meta with the specified metaclass. However, once Meta has been added as an ancestor, a custom metaclass is also used to generate its descendants. In principle, a class like Meta can have either a metaclass generator (metaclass producer) or a set of inheritable methods -- the two aspects of the Meta class are unrelated.