Packaging and publishing the Python module method

preface

Yesterday I packaged my VASP file processing library and uploaded it to PyPI. Now you can install VASPy directly through pip and easy_install (and welcome to the kids who use VASP for computational chemistry to star and participate).

VASPy GotHub address: https: / / github com/PytLab/VASPy
VASPy PyPI address: https: / / pypi python. org/pypi vaspy /

As my memory is really not good, I am afraid that I will forget it after a long time. Therefore, I will take my own VASPy program as an example to summarize the packaging and uploading of python.

VASPy package file structure

First, write the entire file structure of the VASPy package posted on the page, and the rest of the content will take this as an example to illustrate:

VASPy/
 ├ ─ ─  LICENSE
 ├ ─ ─  MANIFEST
 ├ ─ ─  MANIFEST.in
 ├ ─ ─  README.rst
 ├ ─ ─  requirements.txt
 ├ ─ ─  scripts
 │    ├ ─ ─  change_incar_parameters.py
 │    ├ ─ ─  create_inputs.py
 │    └ ─ ─  ...
 ├ ─ ─  setup.cfg
 ├ ─ ─  setup.py
 ├ ─ ─  tests
 │    ├ ─ ─  incar_test.py
 │    ├ ─ ─  __init__.py
 │    ├ ─ ─  oszicar_test.py
 │    ├ ─ ─  outcar_test.py
 │    ├ ─ ─  testdata
 │    │    ├ ─ ─  CONTCAR
 │    │    ├ ─ ─  DOS_SUM
 │    │    ├ ─ ─  ELFCAR
 │    │    └ ─ ─  ...
 │    └ ─ ─  ...
 └ ─ ─  vaspy
   ├ ─ ─  __init__.py
   ├ ─ ─  iter.py
   ├ ─ ─  matstudio.py
   └ ─ ─  ...
 
4 directories, 54 files

Tools for packaging and installing the third package

Here, we need to use setuptools and pip tools to package, release and install our own packages. If we need to build wheel, we also need to install wheel module. If the python version > = 2.7.9 or > =3.4, setuptools and pip are already installed and may need to be updated to the latest version

pip install -U pip setuptools

You can use package management tools, for example

yum install pip
sudo apt-get install pip

The get-pip.py script is installed, and if it detects that wheel and setuptools are not installed, they will be installed automatically

python get-pip.py


I won't talk about the installation and introduction of the specific tools, please refer to requirements for installing packages

The role of different files in a package

setup.py

This file is the most important file to package the entire project. It provides two main functions:

The setup() function, whose arguments specify how to configure your own project.
Command line tools, including packaging, testing, publishing, and more. You can view it with the following command;

python setup.py --help-commands

setup.cfg

This file contains some default parameters at build time such as the --universal parameter when building bdist_wheel

[bdist_wheel]
universal=1

The universal parameter is used by default every time you package. The effect is similar to this:

python setup.py bdist_wheel --universal

README.rst

I wrote this in markdown at first. After packaging and publishing it to PyPI, I found that PyPI did not support the rendering of markdown. It was a mess on the page, so I wrote it again in reStrutruedText's grammar. After all, markup language syntax can be used in a second, so you can't just look for a template.
The grammar rules of reStructureText can be found in the official documentation :Quick reStructuredText

In fact, there is another way to convert markdown to rst using pandoc, and one of the easiest ways to do this is to use the pyandoc module to convert automatically when you publish.
The specific method can be referred to: Use Markdown README's in Python modules

MANIFEST.in

This file tells setuptools that it needs to package those additional files when it is packaged. For example, I used this file to include the test data files of unit tests in VASPy. Of course, README, LICENSE and so on can also be packaged with it.
Here is my own MANIFEST.in:

include README.rst
include requirements.txt
include LICENSE
recursive-include scripts *
recursive-include tests *

For specific grammar rules, please refer to: The MANIFEST.in template

vaspy/

This folder is the package where the vaspy source code resides.

tests/

This folder is also a subpackage and contains the unit test scripts. In order to use python setup.py test for unit test, you have to add init__.pys to make it a package.

The parameters of setup ()

Here introduced me to use only a few parameters, other parameters can refer to use: https: / / docs python. org / 3 / distutils setupscript. html

name

versions = "vaspy"

Is the name of the entire project, which is packaged with the version number.

version

from vaspy import __version__
version = __version__

description

Is a short description of the project, 1 general 1 sentence is good, will be displayed on pypi name lower end.

long_description

Is a long description, equivalent to a concise one for the project, if the string is rst format, PyPI will automatically render to HTML display. Here you can read directly from README.rst.

url

A connection to a package, usually a link on GitHub or a link on readthedocs.

packages

setuptools provides find_packages() to help us find packages in the root path. This function distutil does not have.

setup_requires

This parameter defines the other (most basic) dependencies required for the VASPy installation and smooth operation that will be installed with the pip installation.
For the difference between this parameter and requirements.txt, please refer to: install_requires vs Requirements files

classifier

This parameter provides a series 1 classification, which will be placed in a different directory in PyPI to classify items.
Specific categories name and rule reference: https: / / pypi python. org/pypi & # 63; % 3 Aaction = list_classifiers

test_suite

This parameter helps us to use

python setup.py test

To run unit tests, you no longer need to write a separate script like run_tests.py to run unit tests.
Official explanation of this parameter:

A string naming a unittest.TestCase subclass (or a package or module containing one or more of them, or a method of such a subclass), or naming a function that can be called with no arguments and returns a unittest.TestSuite. If the named suite is a module, and the module has an additional_tests() function, it is called and the results are added to the tests to be run. If the named suite is a package, any submodules and subpackages are recursively added to the overall test suite.

That is to say, this parameter can accept multiple types of parameters:

Accept the unittest.TestCase subclass, and we can write all the unit tests into one test case, and then import comes in, and you send it to test_suite
Take the function object, which takes no arguments, and return 1 unittest.TestSuite.so we can write a single function, combine multiple test cases into 1 suite and return, and then pass the function import in to test_suite.

Module and package name, I just use this way, your own test before are separate multiple scripts, so I add 1 __init__. py can turn it into one package, the package name to test_suite, setuptools will magic will all the test is run under the package 1 side, so I later add the test script directly is to add a new script, other all don't need to change.

Operation effect:

zjshao@SHAO-PC:/mnt/d/Dropbox/Code/CentOS_code/VASPy$ python setup.py test
running test
running egg_info
creating vaspy.egg-info
writing vaspy.egg-info/PKG-INFO
writing top-level names to vaspy.egg-info/top_level.txt
writing dependency_links to vaspy.egg-info/dependency_links.txt
writing manifest file 'vaspy.egg-info/SOURCES.txt'
reading manifest file 'vaspy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'vaspy.egg-info/SOURCES.txt'
running build_ext
test_compare (tests.incar_test.InCarTest)
Make sure we can compare two InCar objects correctly. ... ok
test_eq (tests.incar_test.InCarTest)
Test __eq__() function. ... ok
...
 Several outputs are omitted here 
 
----------------------------------------------------------------------
Ran 22 tests in 3.574s
 
OK

Release your own python package

1. First go to PyPI to register your account

2. Configuration ~/.pypirc is as follows:

[distutils]
index-servers =
  pypi
  pypitest
 
[pypi]
username:ShaoZhengjiang
password:mypassword
 
[pypitest]
username:ShaoZhengjiang
password:mypassword

3. Then register and upload your package to the test server

pypi provides a test server on which we can do tests.

python setup.py register -r pypitest


then

python setup.py sdist upload -r pypitest


If there are no problems we should not get any errors.

4. Upload to PyPI

If the above test is successful, we can follow the same steps to register and upload the package.

python setup.py register -r pypi
python setup.py sdist upload -r pypi

Ok, then we can in PyPI (https: / / pypi. python. org pypi/vaspy /) to see our own package.