Python standard module importlib details

1 module introduction

Python provides the importlib package as part 1 of the standard library. The purpose is to provide an implementation of the import statements in Python (and the s 7en__ function). In addition, importlib allows programmers to create their own custom objects that can be used to introduce procedures (also known as importer).

What is imp?

There is also a module called imp, which provides the interface to the Python import statement mechanism. This module was rejected in Python 3.4 in order to use importlib only.

This module is a bit complex, so we'll focus on the following topics in this post:

The & # 8226; Dynamic introduction
The & # 8226; Check whether the module can be introduced
The & # 8226; Introduce the source file itself
The & # 8226; The third module import_from_github_com

2 module usage

2.1 dynamic introduction

The importlib module supports passing in strings to introduce a module. We create two simple modules to verify this functionality. We will give both modules the same interface and have them print names so that we can distinguish between them. Create two modules, foo.py and bar.py, as shown below.

def main():
 print(__name__)

Now we use importlib to introduce them. Let's see how this code does it. Make sure you have this code in the same directory as the two modules created above.

#importer.py
import importlib

def dynamic_import(module):
 return importlib.import_module(module)

if __name__ == "__main__":
 module = dynamic_import('foo')
 module.main()

 module_two = dynamic_import('bar')
 module_two()

In this code, we manually introduce the importlib module and create a simple function dynamic_import. All this function does is call the import_module function in the importlib module, which is the string we passed in, and then returns the result of the call. Below the code snippet, we call the main method for each module, which will print out the name of each module.

In your code, you probably won't do that much. If you want to introduce this module when you only have 1 string, importlib allows you to do so.

2.2 module introduction inspection

One encoding specification for Python is EAPP: Easier to ask for forgiveness than permision. This means always assuming something exists (e.g., key in the dictionary) and catching exceptions if something goes wrong. You can look at the Python standard module - in the import article we tried to introduce a module, and when it doesn't exist, we'll capture ImportError. What if we want to check and see if a module can be introduced instead of just guessing? You can use importlib. The code is as follows:

#coding:utf-8

import importlib.util
import importlib

def check_module(module_name):
 module_spec = importlib.util.find_spec(module_name)
 if module_spec is None:
 print("Module :{} not found".format(module_name))
 return None
 else:
 print("Module:{} can be imported!".format(module_name))
 return module_spec
 
def import_module_from_spec(module_spec):
 module = importlib.util.module_from_spec(module_spec)
 module_spec.loader.exec_module(module)
 return module

if __name__ == "__main__":
 module_spec = check_module("fake_module")
 module_spec = check_module("collections")
 if(module_spec):
 module = import_module_from_spec(module_spec)
 print(dir(module))

Here we introduce util, a submodule of the importlib module. In the check_module function, we call the find_spec function to check whether the incoming string exists as a module. First, we pass in a fake name, then we pass in the real name of the Python module. If you run this code, you will see that you passed in the name of an uninstalled module, the find_spec function will return None, and our code will print out that the module was not found. If it does, we will return the description of the module.

We can get a description of the module and use it to actually introduce the module. Or you can pass the string into the import_module function, as we learned in section 2.1. But we've learned how to use the module's instructions. Let's take a look at the import_module_from_spec function in the code above. It accepts the module specification returned by the check_module function. We pass it into the module_from_spec function, which will return the imported module. The official documentation for Python recommends that it be executed after the module is introduced, so the next step is to call the exec_module function. Finally we return the module and run the dir function of Python to make sure that we are what we expect.

2.3 import from source file

In this section, I want to show that there is one more technique for the submodule util of importlib. You can use util to introduce a module by module name and file path. An example is shown below,

#coding:utf-8

import importlib.util

def import_source(module_name):
 module_file_path = module_name.__file__
 module_name = module_name.__name__
 
 module_spec = importlib.util.spec_from_file_location(module_name,module_file_path)
 module = importlib.util.module_from_spec(module_spec)
 module_spec.loader.exec_module(module)
 print(dir(module))

 msg = "The {module_name} module has the following methods:{methods}"
 print(msg.format(module_name = module_name,methods = dir(module)))
 
if __name__ == "__main__":
 import logging
 import_source(logging)

In the code above, we actually introduced the logging module and passed it into the import_source function. In this function, we first get the actual path and name of the module. We then pass this information into the spec_from_file_location function of util, which will return a description of the module. Once we have the module specification, we can use the same importlib mechanism as in section 2.2 to actually introduce the module.

Now let's look at a delicate 3rd cubic library, with Python's s 137en__ () function directly introducing the packages in github.

2.4 import_from_github_com

This nifty package, called import_from_github_com, can be used to discover and download packages on github. To install it, all you need to do is use pip as follows,

pip install import_from_github_com

This package USES the new import hooks from PEP 302, which allow you to import packages from github. What the package actually does is install the package and add it locally. You need Python 3.2 or higher, git and pip to use this package.

Once these are installed, you can enter the following command in Python shell,

>>> from github_com.zzzeek import sqlalchemy
Collecting git+https://github.com/zzzeek/sqlalchemy
Cloning https://github.com/zzzeek/sqlalchemy to /tmp/pip-acfv7t06-build
Installing collected packages: SQLAlchemy
Running setup.py install for SQLAlchemy ... done
Successfully installed SQLAlchemy-1.1.0b1.dev0
>>> locals()
{'__builtins__': <module 'builtins' (built-in)>, '__spec__': None,
'__package__': None, '__doc__': None, '__name__': '__main__',
'sqlalchemy': <module 'sqlalchemy' from '/usr/local/lib/python3.5/site-packages/\
sqlalchemy/__init__.py'>,
'__loader__': <class '_frozen_importlib.BuiltinImporter'>}

If you look at the source code for import_from_github_com, you will notice that it does not use importlib. In fact, it USES pip to install packages that are not installed, and then USES Python's s 174en__ () function to introduce the newly installed modules. This code is well worth learning.

2.5 summarize

By this point, you have learned how to use importlib and introduce hooks in your code. Of course, there's more to it than this article covers, and if you need to write a custom introducer or downloader, you'll have to spend a lot of time reading the official documentation and source code.