In defense of doctests
Many of the projects I maintain in open source make heavy use of doctests. Supplying doctests affords a luxury and elegance not offered by unit tests, namely:
- canonical and representative examples are described alongside publishable guidance, reducing duplication of work and opportunities for inconsistency.
- the documented behavior is guaranteed to be correct (assuming tests are run).
- the syntax is necessarily standardized.
- the instructions can be readily parsed by machines.
- the examples can be mechanically re-written for consistent style.
- behavior and tests can often be consolidated in a single code block for simple patches and easy review.
- encourages stateless functions with simple inputs and outputs.
- doctests contribute to code coverage and often are sufficient to capture the full range of supported use-cases.
- zero config framework; tests can be exercised simply from the standard library without any additional imports.
- tests can be supplied on non-importable modules (such as scripts).
- tests can be supplied for functionality where only one file is present (e.g. gist, script, StackOverflow answer).
- tests will capture lingering “print” statements or other unexpected output, output which is normally ignored by test frameworks (example).
Therefore, lines in the docstrings that begin with >>>
are executed and will fail if the output is not a match for what’s written. I acknowledge that this form of test is often discouraged and is suboptimal for many cases (see Soapbox). In particular, if doctests have to spend a high percentage of the user’s attention to facilitate testing, it’s probably better migrated to a unit test. With judicious use of fixtures and aggressive migration to unit tests when appropriate, doctests can be quite valuable and intuitive. That’s why all skeleton-based projects run doctests as a matter of course.
Where these doctests exist in upstream code, it’s preferable to exercise those tests as the upstream project does instead of maintaining a port of those doctests to unit tests.
Some projects (e.g. jaraco.text, jaraco.context) rely entirely on doctests. More sophisticated projects strike the balance between doctests and unit tests (jaraco.functools, pip-run, irc) and others have nearly no doctests (cherrypy).