To help ensure the quality of items in the Charm Store, we’ve been working on automated testing for charms and bundles. What follows is a summary of the current state of charm testing, plans for the future, and some testing-related tips and recommendations for charm authors.
We’ve set up a Jenkins slave dedicated to charm and bundle testing. A new test run is triggered when:
- A new or updated charm or bundle is pushed to the store.
- A new or updated charm or bundle enters the Review Queue.
All test results are published here. Items in the Review Queue are also automatically updated to reflect the status of their latest test run.
Tests are executed by bundletester, which by default will run:
charm proof, the charm linter
make lint, if a Makefile exists in the charm and these targets are defined.
- Any executable files in a
tests/dir in the charm. Typically these will be amulet tests.
The Jenkins job doesn’t actually run
bundletester directly. Instead,
bundletester that adds some features handy for
automated testing, including:
- Fetching charm and bundles sources from a variety of different url formats
- Testing related bundles when a charm changes
- Running tests in parallel across different substrates
- Aggregating json test results
Tips for Charm Authors
Automated testing is becoming the first step in every charm review. Tests will need to pass before a Charmer will review your charm. Here are some guidelines for good tests:
To make sure your tests pass, run them the way Jenkins will – with
bundletester -F in your charm directory.
Ensure dependencies are installed
If your charm has
make lint or
make test targets, make sure
they have prerequisite targets that install the dependencies. Here’s a
sample charm Makefile that does this:
Configure bundletester behaviour
Bundletester uses reasonable defaults to find and execute your tests,
but you can change its
by including a
tests/tests.yaml file in your charm.
- Merge charmguardian and bundletester projects, combining best features of both into one tool.
- Add container (lxc or docker) support to bundletester so that test runs can be executed in a clean, isolated, repeatable environment.
Thanks for reading! If you have questions or comments, feel free to ping tvansteenburgh in #juju on Freenode!