Generate the tutorial for egg under Python's setuptools framework

This article covered the setuptools framework, a side project of PEAK that provides simpler package management and distribution capabilities than distutils.
start

The setuptools module is very "evasive". For example, if we download a package built using setuptools instead of distutils, the installation should work as expected: python setup.py install usually works. To achieve this, a package bound in 1 using setuptools will contain a small boot module ez_setup.py in the archive. The only thing to note here is that ez_setup.py is trying to download and install the required setuptools in the background -- this, of course, requires a machine connected to the network. If setuptools is already installed on the local machine, this background step is no longer required; But if it needs to be installed manually, a lot of transparency is lost. However, most systems now have an Internet connection; It is not particularly troublesome to perform a few extra special steps for a machine that is not connected to the network.

The real advantage of setuptools is not that it implements what distutils does -- although it does enhance distutils and simplify the content of the setup.py script. The biggest advantage of setuptools is its enhanced package management capabilities. It can use a more transparent method to find, download, and install dependent packages; It can also be switched freely in multiple versions of a package, which are all installed on the same system. You can also declare requirements for a particular version of a package; You can also update to the latest version of a package with just one simple command. The most impressive thing is that we can still use packages even if their developers have never considered any setuptools compatibility issues.

Let's explore it in more detail.

guide

The tool ez_setup.py is a simple script that boots the rest of setuptools. Somewhat confusingly, the easy_install script provided in the full setuptools package does the same thing as the ez_setup.py script. But the former assumes that setuptools is already installed, so it skips the behind-the-scenes installation process. Both versions can accept the same parameters and switches.

The first step in this process is to download a small script ez_setup.py:
Listing 1. Download the boot script

% wget -q http://peak.telecommunity.com/dist/ez_setup.py

You can then run the script to install the rest of setuptools without any parameters (if you do not perform this step as a separate step, it will still be completed when the other packages are installed for the first time). You'll see something like this (depending on the version you're using, of course) :
Listing 2. Boot setuptools

% python ez_setup.py
Downloading http://cheeseshop.python.org/packages/2.4/s/
 setuptools/setuptools-0.6b1-py2.4.egg#md5=b79a8a403e4502fbb85ee3f1941735cb
Processing setuptools-0.6b1-py2.4.egg
creating /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Extracting setuptools-0.6b1-py2.4.egg to /sw/lib/python2.4/site-packages
Removing setuptools 0.6a11 from easy-install.pth file
Adding setuptools 0.6b1 to easy-install.pth file
Installing easy_install script to /sw/bin
Installing easy_install-2.4 script to /sw/bin

Installed /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Processing dependencies for setuptools

To complete. This is what we need to do to ensure that setuptools is installed on the system.

The installation package

For many Python packages, all you need to do to install them is pass the package name as a parameter to ez_setup.py or easy_install. Now that setuptools has been loaded using the boot script, we can use the internally simplified easy_install (which is actually very different from the version we chose).

For example, suppose you want to install the SQLObject package. The process is simple, as shown in listing 3. Note in the message that SQLObject depends on a package called FormEncode; Fortunately, this can be handled well:
Listing 3. Installing a typical package

% easy_install SQLObject
Searching for SQLObject
Reading http://www.python.org/pypi/SQLObject/
Reading http://sqlobject.org
Best match: SQLObject 0.7.0
Downloading http://cheeseshop.python.org/packages/2.4/S/
 SQLObject/SQLObject-0.7.0-py2.4.egg#md5=71830b26083afc6ea7c53b99478e1b6a
Processing SQLObject-0.7.0-py2.4.egg
creating /sw/lib/python2.4/site-packages/SQLObject-0.7.0-py2.4.egg
Extracting SQLObject-0.7.0-py2.4.egg to /sw/lib/python2.4/site-packages
Adding SQLObject 0.7.0 to easy-install.pth file
Installing sqlobject-admin script to /sw/bin

Installed /sw/lib/python2.4/site-packages/SQLObject-0.7.0-py2.4.egg
Processing dependencies for SQLObject
Searching for FormEncode>=0.2.2
Reading http://www.python.org/pypi/FormEncode/
Reading http://formencode.org
Best match: FormEncode 0.5.1
Downloading http://cheeseshop.python.org/packages/2.4/F/
 FormEncode/FormEncode-0.5.1-py2.4.egg#md5=f8a19cbe95d0ed1b9d1759b033b7760d
Processing FormEncode-0.5.1-py2.4.egg
creating /sw/lib/python2.4/site-packages/FormEncode-0.5.1-py2.4.egg
Extracting FormEncode-0.5.1-py2.4.egg to /sw/lib/python2.4/site-packages
Adding FormEncode 0.5.1 to easy-install.pth file

Installed /sw/lib/python2.4/site-packages/FormEncode-0.5.1-py2.4.egg

Just as you can see 1 sample from these messages, in www easy_install. python. org/pypi/find information about the package, and then find the place where you can download it really (egg package here at cheeseshop. python. org; More on egg later).

You can now install more than just the latest version of a package (this is the default). You can also provide a specific version requirement for easy_install if you wish. Now let's try installing a version of post-beta of SQLObject.
Listing 4. Installing the smallest version of a package

% easy_install 'SQLObject>=1.0'
Searching for SQLObject>=1.0
Reading http://www.python.org/pypi/SQLObject/
Reading http://sqlobject.org
No local packages or download links found for SQLObject>=1.0
error: Could not find suitable distribution for
 Requirement.parse('SQLObject>=1.0')

If (as is the case at the time of this writing) the latest version of SQLObject is less than 1.0, then nothing will be installed.

Install "naive" package

SQLObject can recognize setuptools; But what if you want to install a package that is not yet compatible with setuptools? For example, before this article, I had never used setuptools for my "Gnosis Utilities". However, for now, let's try to install 1 of this package, and only the HTTP (or FTP, SVN, CVS) location is known (setuptools understands all of these protocols). My Web download site has versions of Gnosis Utilities named in the usual style:
Listing 5. Installing a package that does not recognize setuptools

% easy_install -f http://gnosis.cx/download/Gnosis_Utils.More/ Gnosis_Utils
Searching for Gnosis-Utils
Reading http://gnosis.cx/download/Gnosis_Utils.More/
Best match: Gnosis-Utils 1.2.1
Downloading http://gnosis.cx/download/Gnosis_Utils.More/
 Gnosis_Utils-1.2.1.zip
Processing Gnosis_Utils-1.2.1.zip
Running Gnosis_Utils-1.2.1/setup.py -q bdist_egg --dist-dir
 /tmp/easy_install-CCrXEs/Gnosis_Utils-1.2.1/egg-dist-tmp-Sh4DW1
zip_safe flag not set; analyzing archive contents...
gnosis.__init__: module references __file__
gnosis.magic.__init__: module references __file__
gnosis.xml.objectify.doc.__init__: module references __file__
gnosis.xml.pickle.doc.__init__: module references __file__
gnosis.xml.pickle.test.test_zdump: module references __file__
Adding Gnosis-Utils 1.2.1 to easy-install.pth file

Installed /sw/lib/python2.4/site-packages/Gnosis_Utils-1.2.1-py2.4.egg
Processing dependencies for Gnosis-Utils

Fortunately, easy_install can do both well. It looks at the given download directory, identifies the highest version available, expands the package, and then repackages it in the "egg" format, which can then be used for installation. The import gnosis can now be run in one script. But what if you now need to test a script against a particular version before Gnosis Utilities? It's also very simple:
Listing 6. Installing a specific version of the "naive" package

% easy_install -f http://gnosis.cx/download/Gnosis_Utils.More/
 "Gnosis_Utils==1.2.0"
Searching for Gnosis-Utils==1.2.0
Reading http://gnosis.cx/download/Gnosis_Utils.More/
Best match: Gnosis-Utils 1.2.0
Downloading http://gnosis.cx/download/Gnosis_Utils.More/
 Gnosis_Utils-1.2.0.zip
[...]
Removing Gnosis-Utils 1.2.1 from easy-install.pth file
Adding Gnosis-Utils 1.2.0 to easy-install.pth file

Installed /sw/lib/python2.4/site-packages/Gnosis_Utils-1.2.0-py2.4.egg
Processing dependencies for Gnosis-Utils==1.2.0

Two versions of Gnosis Utilities are now usually installed, and the current active version is 1.2.0. Switching the active version back to 1.2.1 is also very simple:
Listing 7. Modify the "live" version system-wide

% easy_install "Gnosis_Utils==1.2.1"
Searching for Gnosis-Utils==1.2.1
Best match: Gnosis-Utils 1.2.1
Processing Gnosis_Utils-1.2.1-py2.4.egg
Removing Gnosis-Utils 1.2.0 from easy-install.pth file
Adding Gnosis-Utils 1.2.1 to easy-install.pth file

Using /sw/lib/python2.4/site-packages/Gnosis_Utils-1.2.1-py2.4.egg
Processing dependencies for Gnosis-Utils==1.2.1

Of course, only one version is active at a time. However, by placing two similar lines on top of each script, you can let the script choose the version it wants to use:
Listing 8. Using a version of a package in a script

from pkg_resources import require
require("Gnosis_Utils==1.2.0")

By using the above requirements, setuptools can add a specific version (the latest available version if greater than comparison is specified) when import statements are run.

Enable the package to recognize setuptools

I would prefer to let users install Gnosis Utilities without knowing its download directory. This usually works because Gnosis Utilities has a list of information on Python Cheeseshop. Unfortunately, because of not considering setuptools, so I in python org for my Gnosis Utilities established a "mismatch" entrance http: / / www python. org pypi/Gnosis % 20 Utilities / 1.2.1. Specifically, the archive is based on similar to Gnosis_Utils - N. N. N. tar. gz model named (also pack these tools into. zip and. tar. bz2 file, the latest of several versions have also packaged into win32. exe installation procedures, all of these files setuptools is a good way to deal with). However, the project name on Cheeseshop is spelled slightly differently than "Gnosis Utilities." In fact, in Cheeseshop management of a small version of the change will be http: / / www python. org/pypi/Gnosis_Utils / 1.2.1 - a created for a release version. Nothing has changed in the distribution archive itself, but a point of metadata has been added to Cheeseshop. With a little effort, you can use a simpler installer (note that for testing purposes, I ran an easy_install-m to remove the installed package).
Listing 9. Simply add the recognition of setuptools

% easy_install Gnosis_Utils
Searching for Gnosis-Utils
Reading http://www.python.org/pypi/Gnosis_Utils/
Reading http://www.gnosis.cx/download/Gnosis_Utils.ANNOUNCE
Reading http://gnosis.cx/download/Gnosis_Utils.More/
Best match: Gnosis-Utils 1.2.1
Downloading [...]

I've left out the rest of the process, because it's no different from what you saw earlier. But one difference is that a easy_install to Cheeseshop (in other words www. python. org/pypi) for metadata, can match the specified name and use this information to find the real download location. In this case, the listed.ANNOUNCE file does not contain anything helpful, but easy_install will continue to look at the other listed URL, which will prove that it is a download directory.

About egg

egg is a file package that contains all package data. Ideally, egg would be a file using zip compression, which includes all the required package files. However, in some cases, setuptools will decide (or be told by the switch) that the package should not be zip compressed. In these cases, egg is simply an uncompressed subdirectory, but the contents are the same. With the single 1 version you can easily convert and save 1 point of disk space, but the egg directory is functionally and organizationally identical. 1 directly use Java? of the JAR file; Users of the technology will find egg very familiar.

Since the latest version of Python (requiring 2.3.5+ or 2.4) imports hook changes, egg can be used simply by setting PYTHONPATH or sys.path and importing the package as usual. If you want to use this method, you don't need to use setuptools or ez_setup.py. For example, in the working directory used in this article, I put an egg for the PyYAML package. Now I can use the package as follows:
Listing 10. egg on PYTHONPATH

% export PYTHONPATH=~/work/dW/PyYAML-3.01-py2.4.egg
% python -c 'import yaml; print yaml.dump({"foo":"bar",1:[2,3]})'
1: [2, 3]
foo: bar

However, the operation of PYTHONPATH (or script or sys.path in Python shell session) is somewhat fragile. The discovery of egg is best performed in the.pth file at new point 1. Any.pth file in site-packages/or PYTHONPATH is parsed to perform other import operations, similar to checking those directory locations that might contain packages. If you are using setuptools for package management, you will need to modify a file named easy-install.pth when installing, updating, and deleting packages. And you can name the.pth however you like (as long as the extension is.pth). For example, here is the content of my easy-install.pth file:
Listing 11. The.pth file used as the configuration for the egg location

% python ez_setup.py
Downloading http://cheeseshop.python.org/packages/2.4/s/
 setuptools/setuptools-0.6b1-py2.4.egg#md5=b79a8a403e4502fbb85ee3f1941735cb
Processing setuptools-0.6b1-py2.4.egg
creating /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Extracting setuptools-0.6b1-py2.4.egg to /sw/lib/python2.4/site-packages
Removing setuptools 0.6a11 from easy-install.pth file
Adding setuptools 0.6b1 to easy-install.pth file
Installing easy_install script to /sw/bin
Installing easy_install-2.4 script to /sw/bin

Installed /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Processing dependencies for setuptools

0

This format is a bit unusual: it is close to an Python script, but not quite. Note that you can add the egg listed extra there; Even better, easy_install does this at run time. You can also create any number of.pth files under site-packages /; Each can list which egg are available.

Enhanced installation script

The ability to install the setuptools naive package described above (see listing 6) is only partially valid. That is, the package Gnosis_Utils is installed, but it is not complete. All the usual functions work, but a lot of support files are ignored in the automatic generation of egg -- mostly documents with an extension of.txt and test files with an extension of.xml (along with a few other README,.rnc,.rng,.xsl, and files around subpackages). At installation time, all of these support files are "preferred" rather than strictly required. However, we still want to be able to include all the supporting files.

The setup.py script used by Gnosis_Utils is actually quite complex. In addition to listing the basic metadata, in line 467, it fully tests the functionality of the Python version and bug; Fixed some problems with 1 in the older version of distutils; Backtracking skips the unsupported part of the installation (for example, if pyexpat is not included in the Python distribution); Handles the conversion of the OS line terminator; Create multiple archive/installer types; Rebuild the MANIFEST file based on the test results. The ability to handle this work is thanks to another maintainer of the package, Frank McIngvale; These capabilities allow Gnosis_Utils to be successfully installed back to version 1.5.1 of Python, as long as you need to (earlier versions were less features-rich). But the script I'm going to show you here doesn't do anything as complicated as distutils script 1: it simply assumes that a "normal" version of Python has been installed on the system. Even so, it's fascinating that setuptools makes it so easy to install scripts.

On the first try, let's create an setup.py script, which is borrowed from the setuptools manual, and try to use it to create an egg:
Listing 12. setuptools setup.py script

% python ez_setup.py
Downloading http://cheeseshop.python.org/packages/2.4/s/
 setuptools/setuptools-0.6b1-py2.4.egg#md5=b79a8a403e4502fbb85ee3f1941735cb
Processing setuptools-0.6b1-py2.4.egg
creating /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Extracting setuptools-0.6b1-py2.4.egg to /sw/lib/python2.4/site-packages
Removing setuptools 0.6a11 from easy-install.pth file
Adding setuptools 0.6b1 to easy-install.pth file
Installing easy_install script to /sw/bin
Installing easy_install-2.4 script to /sw/bin

Installed /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Processing dependencies for setuptools

1

This effort can already work; At least in part. It is true that you can create an egg with these lines, but this egg has a similar disadvantage to egg created with easy_install: lack of support for files that do not use the.py name. So let's try it one more time, just a little bit harder:
Listing 13. Add the missing package_data

% python ez_setup.py
Downloading http://cheeseshop.python.org/packages/2.4/s/
 setuptools/setuptools-0.6b1-py2.4.egg#md5=b79a8a403e4502fbb85ee3f1941735cb
Processing setuptools-0.6b1-py2.4.egg
creating /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Extracting setuptools-0.6b1-py2.4.egg to /sw/lib/python2.4/site-packages
Removing setuptools 0.6a11 from easy-install.pth file
Adding setuptools 0.6b1 to easy-install.pth file
Installing easy_install script to /sw/bin
Installing easy_install-2.4 script to /sw/bin

Installed /sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Processing dependencies for setuptools

2

That's all you need to do. Of course, depending on the situation, you usually want to make some adjustments to it. For example, it might list the following:
Listing 14. Packaging a specific type of file type

package_data = {'doc':['*.txt'], 'xml':['*.xml', 'relax/*.rnc']}

Include the.txt file in the doc/ subpackage, the.xml file in the xml/ subpackage, and all the.rnc files in the xml/relax/ subpackage.

conclusion

This article has really only covered the surface of what you can do with a distribution that supports setuptools. For example, if you now have a distribution (either the preferred egg format or another archive type), you can upload the archive and metadata to Cheeseshop using a single command. Obviously, the full setup.py script should contain the same detailed metadata as the older distutils script. For the sake of simplicity, this article skips this, but the parameter names are compatible with distutils.

Although it will take a while to fully get used to the huge capabilities offered by setuptools, it does make it easier to maintain your own packages and install external packages than distutils. If all you care about is installing packages, everything you need to know is covered in this article. It's just that you might find some complexity when describing your own packages, but it's still not as complex as using distutils.