Charm Testing

To help ensure the quality of items in the Charm Store, we’ve been working on automated testing for charms and bundles. What follows is a summary of the current state of charm testing, plans for the future, and some testing-related tips and recommendations for charm authors.

Overview

We’ve set up a Jenkins slave dedicated to charm and bundle testing. A new test run is triggered when:

All test results are published here. Items in the Review Queue are also automatically updated to reflect the status of their latest test run.

What’s Tested?

Tests are executed by bundletester, which by default will run:

The Jenkins job doesn’t actually run bundletester directly. Instead, it runs charmguardian, a wrapper around bundletester that adds some features handy for automated testing, including:

Tips for Charm Authors

Automated testing is becoming the first step in every charm review. Tests will need to pass before a Charmer will review your charm. Here are some guidelines for good tests:

Run bundletester

To make sure your tests pass, run them the way Jenkins will – with bundletester. Install it, then run bundletester -F in your charm directory.

Ensure dependencies are installed

If your charm has make lint or make test targets, make sure they have prerequisite targets that install the dependencies. Here’s a sample charm Makefile that does this:

Configure bundletester behaviour

Bundletester uses reasonable defaults to find and execute your tests, but you can change its behaviour by including a tests/tests.yaml file in your charm.

Next Steps

Thanks for reading! If you have questions or comments, feel free to ping tvansteenburgh in #juju on Freenode!

comments powered by Disqus