Analysis of Working Principle of Python Virtual Environment

Directory Introduction How Activation Scripts Work About sys. prefix Summarize Others

The virtual environment of Python is used to create a relatively independent execution environment, especially one dependent 3-party package. The most common one is that different projects depend on the same but different versions of 3-party packages. Moreover, the installation package in the virtual environment will not affect the installation package of the system.

However, its specific working principle is described in detail here.

Brief introduction

Almost every language includes its own package management tools, which is a very complicated topic, and different languages choose slightly different implementations, so they will make some choices and trade-offs. There are many package management solutions for Python, such as pip, virtualenv, pyenv and so on.

However, the mechanism of Python language itself determines its principle.

Use

The most commonly used tool is virtualenv, which can be referred to the detailed introduction in Guide to Python. In addition, Python3 also provides its own virtual environment creation module, which basically enables an independent environment through a script after creation.

For example, the following is the virtual environment creation process using virtualenv and venv.

$ mkdir /tmp/project && cd /tmp/project

$ virtualenv --no-site-packages foobar
$ python3 -m venv foobar

Then, you can pass source foobar/bin/activate Command to activate the new environment.

Activation script

The so-called stand-alone environment is nothing more than solving two problems: A) the version used to execute the Python parser; B) uses a separate package. Among them, the former, in Linux, is mainly through PATH Environment variable settings, in the activate The script has the following contents.

VIRTUAL_ENV="/tmp/project/foobar"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

That is, add the created directory to the PATH Environment variable, then the path will be searched first, thus solving the problem of python parser independence.

Working principle

If you want to use a separate package, the key is how to pass the import The required package was found when importing.

The search order of packages can be viewed in the introduction of Python module. Simply put, it is to check whether it is a built-in module first, and then from sys.path Searches the address specified in the list. So, the key here is sys.path Generation of the list.

About sys. prefix

When Python starts, it will load 1 strongly dependent os.py Package, and finding this package is based on the current path of the parser and fixed lookup rules.

Simply put, it is to add the current path lib/python${VERSION}/os.py Look up layer by layer, note that if it is a 64-bit operating system, it will use lib64 Replace the previous one PATH0 Path.

For example, the default parser path for Python3 is /usr/bin/python3.6 The underlying path is /usr/bin/ Therefore, the search order is.

/usr/bin/lib64/python3.6/os.py
/usr/lib64/python3.6/os.py
/lib64/python3.6/os.py

As long as it is found on any path os.py Package, the lookup will exit and the sys.prefix Variable, details can be passed through strace python Look, there will be the following search path.

stat("/usr/bin/Modules/Setup", 0x7fffb7146300) = -1 ENOENT (No such file or directory)
stat("/usr/bin/lib64/python2.7/os.py", 0x7fffb71462f0) = -1 ENOENT (No such file or directory)
stat("/usr/bin/lib64/python2.7/os.pyc", 0x7fffb71462f0) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=25910, ...}) = 0

After finding os.py After that, the path is set to sys.prefix Variable, and then the parser will go to ${sys.prefix}/lib/python${VERSION} Directory to find packages.

Summarize

Then it works by saving the python parser in the ${VENV_PATH}/bin/python And then create ${VENV_PATH}/lib/python${VERSION} Directory, and copy related files to that directory, you can copy files, you can also use soft connections.

Others

As above, if it is Python3, you can directly use the built-in venv Module, whose principle is the same as above, is passed through pyvenv.cfg Configuration file to identify the original home location, the contents of which are as follows.

home = /usr/bin
include-system-site-packages = false
version = 3.6.8

If include-system-site-packages For true When the interpreter starts, the system's library is added to the sys.path Inside, so that in the virtual environment, you can import The package installed in the system.

Note that Python3 provides venv Modules can only be created from the current version and cannot support Python2.

Refer to the official Virtualenv document for details.

Creation of virtual environments Python3 provides an introduction to venv, including common parameters and configuration files.

"Older