Tutorial on using the test modules unittest and doctest in Python

  • 2020-05-09 18:48:25
  • OfStack

I have a confession to make. Although I am the creator of a fairly widely used public domain Python library, the unit tests introduced in my module are very unsystematic. In fact, most of those tests were included in Gnosis Utilities of gnosis.xml.pickle and were written by contributors to the subpackage (subpackage). I also found that most of the 3rd party Python packages I downloaded lacked a complete set of unit tests.

Not only that, but the existing tests in Gnosis Utilities suffer from another flaw: you often need to infer the desired output in excruciating detail to determine the success or failure of the test. Testing is actually -- in many cases -- more like a little utility that USES parts of a library. These tests (or utilities) support input and/or output in a descriptive data format from any data source (of the correct type). In fact, these test utilities are more useful when you need to debug minor errors. But for self-explanatory integrity checks (sanity checks) for changes between library versions, these class tests are not up to the task.

In this installment, I try to use the Python standard library modules doctest and unittest to improve the testing of my utility set and take you through the experience with me (and point out some of the best ways).

Script gnosis/xml/objectify/test/test_basic py 1 on the shortcoming of the current testing is given and the solution of a typical example. Here's the latest version of the script:

Listing 1. test_basic. py


"Read and print and objectified XML file"
import sys
from gnosis.xml.objectify import XML_Objectify, pyobj_printer
if len(sys.argv) > 1:
 for filename in sys.argv[1:]:
  for parser in ('DOM','EXPAT'):
   try:
    xml_obj = XML_Objectify(filename, parser=parser)
    py_obj = xml_obj.make_instance()
    print pyobj_printer(py_obj).encode('UTF-8')
    sys.stderr.write("++ SUCCESS (using "+parser+")\n")
    print "="*50
   except:
    sys.stderr.write("++ FAILED (using "+parser+")\n")
    print "="*50
else:
 print "Please specify one or more XML files to Objectify."

The utility function pyobj_printer() generates a non--XML representation of any Python object (specifically, an object that does not use any other utility of gnosis.xml.objectify, or anything else in Gnosis Utilities). In future versions, I will probably move this function to another part of the Gnosis package. In any case, pyobj_printer() USES the indentation and notation of various class-Python to describe objects and their properties (similar to pprint, but extending instances, not just the built-in data types).

If some special XML might not be "objectualized (objectified)" correctly, the test_basic.py script provides a good debugging tool -- you can visually view the properties and values of the resulting object. In addition, if you redirect STDOUT, you can view a simple message on STDERR, as in this example:

Listing 2. Parsing the STDERR result message


$ python test_basic.py testns.xml > /dev/null
++ SUCCESS (using DOM)
++ FAILED (using EXPAT)

However, the definition of success or failure in the example run above is not obvious: success simply means that no exceptions have occurred, not that the output (redirected) was correct.
Using doctest


The doctest module lets you embed comments in the docstring (docstrings) to display the expected behavior of various statements, especially the results of functions and methods. This is very much like making the docstring look like an interactive shell session; One simple way to accomplish this task is to copy-paste from an Python interactive shell (or from Idel, PythonWin, MacPython, or another IDE with an interactive session). This 1 improved test_basic.py script illustrates the addition of self-diagnostic functionality:

Listing 3. The test_basic.py script with self-diagnosis


import sys
from gnosis.xml.objectify import XML_Objectify, pyobj_printer, EXPAT, DOM
LF = "\n"
def show(xml_src, parser):
 """Self test using simple or user-specified XML data
 >>> xml = '''<?xml version="1.0"?>
 ... <!DOCTYPE Spam SYSTEM "spam.dtd" >
 ... <Spam>
 ... <Eggs>Some text about eggs.</Eggs>
 ... <MoreSpam>Ode to Spam</MoreSpam>
 ... </Spam>'''
 >>> squeeze = lambda s: s.replace(LF*2,LF).strip()
 >>> print squeeze(show(xml,DOM)[0])
 -----* _XO_Spam *-----
 {Eggs}
  PCDATA=Some text about eggs.
 {MoreSpam}
  PCDATA=Ode to Spam
 >>> print squeeze(show(xml,EXPAT)[0])
 -----* _XO_Spam *-----
 {Eggs}
  PCDATA=Some text about eggs.
 {MoreSpam}
  PCDATA=Ode to Spam
 PCDATA=
 """
 try:
  xml_obj = XML_Objectify(xml_src, parser=parser)
  py_obj = xml_obj.make_instance()
  return (pyobj_printer(py_obj).encode('UTF-8'),
    "++ SUCCESS (using "+parser+")\n")
 except:
  return ("","++ FAILED (using "+parser+")\n")
if __name__ == "__main__":
 if len(sys.argv)==1 or sys.argv[1]=="-v":
  import doctest, test_basic
  doctest.testmod(test_basic)
 elif sys.argv[1] in ('-h','-help','--help'):
  print "You may specify XML files to objectify instead of self-test"
  print "(Use '-v' for verbose output, otherwise no message means success)"
 else:
  for filename in sys.argv[1:]:
   for parser in (DOM, EXPAT):
    output, message = show(filename, parser)
    print output
    sys.stderr.write(message)
    print "="*50

Note that I put the main code block in the improved (and extended) test script, so that if you specify the XML file on the command line, the script continues to perform the previous behavior. This allows you to continue to analyze XML beyond your test cases and focus only on the results -- or find errors in what gnosis.xml.objectify is doing, or just understand its purpose. In a standard way, you can use the -h or --help parameter to get a description of the usage.

Interesting new features are found when test_basic.py is run without any parameters (or with the -v parameter used only by doctest). In this example, we are running doctest on the module/script itself -- you can see that we are actually importing test_basic into the script's own namespace, so that we can simply import other modules that we want to test. The doctest.testmod () function walks through all the docstrings in the module itself, its functions, and its classes to find all the contents of an interactive shell session. In this example, such a session will be found in the show() function.

The documentation string for show() illustrates several small "traps" (gotchas) during a designed doctest session. Unfortunately, doctest handles empty lines as the end of the session when parsing an explicit session -- so output like the return value of pyobj_printer() needs some protection (be munged slightly) for testing. The easiest way to do this is to use a function like squeeze() defined by the docstring itself (which simply removes the line feed that follows). In addition, because docstrings are, after all, string substitution (escape), sequences like \n are extended, making the substitution of line breaks a little confusing within the code sample. You can use \\n, but I find that the definition of LF solves these problems.

The self-test defined in the documentation string of show() does more than just ensure that no exceptions occur (as opposed to the original test script). Check at least one simple XML document for proper objectification (objectification). Of course, it is still possible that some other XML documents may not be handled correctly -- for example, the namespace XML document testns.xml, which we tried above, encountered an EXPAT parser failure. A docstring processed by doctest may contain a traceback (traceback) within it, but in special cases, a better approach is to use unittest.

Using unittest


The other test included in gnosis.xml.objectify is test_expat.py. The primary reason for creating this 1 test is simply that a user of a subpackage using the EXPAT parser will often need to call a special setup function to enable the processing of the namespace XML document (the reality is that this has evolved, not been designed, and may change in the future). The old test would try to print the object without Settings, catch an exception if it occurred, and print it again if needed (with a message about what happened).

If you use the test_basic.py, test_expat.py tools, you can analyze how gnosis.xml.objectify describes a novel XML document. But like before 1, there are many specific behaviors that we might want to verify. An enhanced, extended version of test_expat.py USES unittest to analyze what happens during the execution of various actions, including assertions holding certain conditions or (approximately) equations, or certain expected exceptions. 1 look at:

Listing 4. The self-diagnosed test_expat.py script


"Objectify using Expat parser, namespace setup where needed"
import unittest, sys, cStringIO
from os.path import isfile
from gnosis.xml.objectify import make_instance, config_nspace_sep,\
         XML_Objectify
BASIC, NS = 'test.xml','testns.xml'
class Prerequisite(unittest.TestCase):
 def testHaveLibrary(self):
  "Import the gnosis.xml.objectify library"
  import gnosis.xml.objectify
 def testHaveFiles(self):
  "Check for sample XML files, NS and BASIC"
  self.failUnless(isfile(BASIC))
  self.failUnless(isfile(NS))
class ExpatTest(unittest.TestCase):
 def setUp(self):
  self.orig_nspace = XML_Objectify.expat_kwargs.get('nspace_sep','')
 def testNoNamespace(self):
  "Objectify namespace-free XML document"
  o = make_instance(BASIC)
 def testNamespaceFailure(self):
  "Raise SyntaxError on non-setup namespace XML"
  self.assertRaises(SyntaxError, make_instance, NS)
 def testNamespaceSuccess(self):
  "Sucessfully objectify NS after setup"
  config_nspace_sep(None)
  o = make_instance(NS)
 def testNspaceBasic(self):
  "Successfully objectify BASIC despite extra setup"
  config_nspace_sep(None)
  o = make_instance(BASIC)
 def tearDown(self):
  XML_Objectify.expat_kwargs['nspace_sep'] = self.orig_nspace
if __name__ == '__main__':
 if len(sys.argv) == 1:
  unittest.main()
 elif sys.argv[1] in ('-q','--quiet'):
  suite = unittest.TestSuite()
  suite.addTest(unittest.makeSuite(Prerequisite))
  suite.addTest(unittest.makeSuite(ExpatTest))
  out = cStringIO.StringIO()
  results = unittest.TextTestRunner(stream=out).run(suite)
  if not results.wasSuccessful():
   for failure in results.failures:
    print "FAIL:", failure[0]
   for error in results.errors:
    print "ERROR:", error[0]
 elif sys.argv[1].startswith('-'): # pass args to unittest
  unittest.main()
 else:
  from gnosis.xml.objectify import pyobj_printer as show
  config_nspace_sep(None)
  for fname in sys.argv[1:]:
   print show(make_instance(fname)).encode('UTF-8')

Using unittest adds quite a bit of power to the simpler doctest approach. We can divide our tests into several classes, each of which inherits from unittest.TestCase. Within each test class, each method whose name begins with ".test "is considered another test. The two additional classes defined for ExpatTest are interesting: run.setUp () every time you use the class to perform a test, and run.tearDown () at the end of the test (whether the test succeeds, fails, or fails). In our example above, we did a bit of bookkeeping for the dedicated expat_kwargs dictionary to ensure that each test ran independently.

By the way, the difference between failure (failure) and error (error) is important. A test may fail because some specific assertion is invalid (the assertion method either begins with ".fail "or begins with".assert "). In a sense, failure is expected -- at least in the sense that we have already analyzed it. On the other hand, errors are unexpected problems -- because we don't know in advance what will go wrong, we need to analyze backtracking in actual test runs to diagnose the problem. However, we can design failures to give a hint of a diagnostic error. For example, if Prerequisite.haveFiles () fails, an error will occur in some TestExpat tests; If the former is successful, you will have to look elsewhere for the source of the error.

In the unittest.TestCase inheritance class, specific test methods might include 1.assert... . () or fail... () method, but it might just have 1 series of actions that we believe should be executed successfully. If the test method does not work as expected, we get an error (and a backtrace describing the error).

The _main_ block in test_expat.py is also worth looking at. In the simplest case, we could just use unittest.main () to run the test cases, which would determine what needs to be run. With this approach, the unittest module will accept a -v option to give more detailed output. Based on the specified file name, after performing the namespace setting, we print out the representation of the specified XML file, roughly maintaining backward compatibility with the older version of the tool.

The most interesting branch of _main_ is the branch that expects the -q or --quiet tags. As you would expect, this branch will be silent (quiet, that is, minimize output) unless a failure or error occurs. Not only that, but since it is silent, it only displays a one-line report of the failure/error location for each problem, rather than the entire diagnostic traceback. In addition to making direct use of the silent output style, this branch also illustrates custom testing relative to the test suite and control over the reporting of results. The default output of unittest.TextTestRunner () is directed to StringIO out -- if you want to see it, you are welcome to look it up at out.getvalue (). However, the result object allows us to test for full success and, if not complete success, to handle failures and errors. Obviously, because they are values in variables, you can easily log the contents of result objects or display them in GUI, anyway, rather than just printing them to STDOUT.

Combination test


Perhaps the best feature of the unittest framework is the ease with which you can combine tests that contain different modules. In fact, if you use Python 2.3+, you can even convert the doctest tests into the unittest suite. Let's combine the tests we've created so far into one script, test_all.py (admittedly, it's an exaggeration to say that it's the tests we've done so far) :

Listing 5. test_all.py combines the unit tests


"Combine tests for gnosis.xml.objectify package (req 2.3+)"
import unittest, doctest, test_basic, test_expat
suite = doctest.DocTestSuite(test_basic)
suite.addTest(unittest.makeSuite(test_expat.Prerequisite))
suite.addTest(unittest.makeSuite(test_expat.ExpatTest))
 unittest.TextTestRunner(verbosity=2).run(suite)

Since test_expat.py only contains test classes, they can be easily added to the local test suite. The doctest.DocTestSuite () function performs the conversion of the docstring test. Let's take a look at what test_all.py does when it runs:

Listing 6. Successful output from test_all.py


$ python2.3 test_all.py
doctest of test_basic.show ... ok
Check for sample XML files, NS and BASIC ... ok
Import the gnosis.xml.objectify library ... ok
Raise SyntaxError on non-setup namespace XML ... ok
Sucessfully objectify NS after setup ... ok
Objectify namespace-free XML document ... ok
Successfully objectify BASIC despite extra setup ... ok
----------------------------------------------------------------------
Ran 7 tests in 0.052s
OK

Note the description of the tests performed: in the case of the unittest test method, their description comes from the corresponding docstring function. If you do not specify a documentation string, the class and method names are used as the most appropriate descriptions. It's also interesting to see what happens if one of these tests fails (the backtracking details were removed for this article) :

Listing 7. Results when some tests fail


$ mv testns.xml testns.xml# && python2.3 test_all.py 2>&1 | head -7
doctest of test_basic.show ... ok
Check for sample XML files, NS and BASIC ... FAIL
Import the gnosis.xml.objectify library ... ok
Raise SyntaxError on non-setup namespace XML ... ERROR
Sucessfully objectify NS after setup ... ERROR
Objectify namespace-free XML document ... ok
Successfully objectify BASIC despite extra setup ... ok

Incidentally, the last line of STDERR for this failure is "FAILED (failures=1, errors=2)", which is a good summary if you need it (as opposed to the final "OK" for success).

So let's start here


This article introduced you to some typical USES of unittest and doctest, which have improved testing in my own software. Read the Python documentation to gain insight into the full range of methods available for test suites, test cases, and test results. They all follow the pattern described in the example.

It is good software practice to conform yourself to the methodology laid down in Python's standard test module. Test-driven (test-driven) development is popular in many software cycles; However, it is clear that Python is a language suitable for test-driven models. Furthermore, a package or library that is accompanied by a comprehensive set of tests is more useful to the user than a package or library that lacks these tests, if only considering that the package is more likely to work as planned.


Related articles: