DHX: A Customized Hook Debugging Environment Plugin

Following on from my previous post, I created a Juju plugin that allows you to fully and automatically customize the machines created by Juju, for the purposes of making developing and debugging hooks as painless as possible.

Overview

The machines created by Juju are completely standard, based off the Ubuntu Cloud Image. While this consistency is great for running charms, when developing and debugging those charms, most developers would prefer to be able to tweak things such as their editor configuration, bash configuration, install additional editors or debugging libraries, et cetera. But doing this each time on a new machine spun up by Juju is repetitive and tedious.

So, I created dhx (or debug-hooks-ext) as part of the Juju Plugins bundle, which allows you to define setup and customizations that will be automatically run the first time you start a debug session on a new machine. Additionally, it has options for assisting with debugging, such as automatically pulling down changes made to the remote charm during debugging, or restarting the failed hook and automatically dropping you into a debug session for that hook.

Here is an overview of the features:

Installation and Setup

First, follow the instructions in the README to install the plugin bundle. This should be straightforward and will give you access to DHX as well as all the other very useful Juju plugins in the bundle.

Next, you'll want to create a configuration file to define what customizations to perform:

cat > ~/.juju/debug-hooks-rc.yaml <EOF
use_tmux_keybindings: true
uploads:
  - '~/.vimrc'
  - '~/.vim'
init: '~/.juju/my-dhx-init.sh'
sync_excludes:
  - '.*'
  - '*.pyc'
import_ids: []
auto_sync: false
auto_restart: false
EOF

This configuration will use the standard keybindings (Ctrl-b) for tmux instead of the screen keybindings (Ctrl-a) that debug-hooks normally uses. It will also upload your VIM configuration and execute my-dhx-init.sh on the remote machine upon the first connect to perform any additional customizations (e.g., installing ipdb for improved debugging of charms written in python).

(The other options will be described below.)

Running DHX

DHX is a drop-in replacement for juju debug-hooks. So, whenever you would use juju debug-hooks to start a debugging session, you should instead use juju dhx (or juju debug-hooks-ext, if you want to be verbose). This will automatically detect if the environment you're connecting to has been customized and, if not, apply your customizations.

Improved Unit Selection

Instead of typing out the full unit name, in the form service/0, you can let DHX figure out which unit you want to debug. It will use cues such as which units are in error state, the charm you're working on in the current directory, or a service name you give it instead of a unit name.

If DHX can't unambiguously choose a unit, it will present you with its best guess along with a list of units that you can choose from by number or name, so that just pressing Enter to accept the default will usually do the right thing.

For example:

$ juju pprint
- chamilo/0: 10.0.3.107 (error)
- mysql/0: 10.0.3.139 (started)
- mysql/1: 10.0.3.181 (error)
$ juju dhx
Units:
  0: chamilo/0 (error)
  1: mysql/0
  2: mysql/1 (error)

Select a unit by number or name: [chamilo/0]
$ juju dhx mysql
Units:
  0: mysql/0
  1: mysql/1 (error)

Select a unit by number or name: [mysql/1]

Retrying Failed Hooks

The most common reason why you need to start a debug-hooks session is because a hook failed and you want to debug it to figure out why. Thus, once you are in the debug-hooks session, you want to restart the failed hook and start debugging it.

Instead of manually running juju resolved --retry $unit, you can just add the --retry (or just -r) option to dhx:

juju dhx -r

You will then be connected to the unit, it will be customized if necessary, and the hook will be immediately restarted and paused for you to debug.

You can also set auto_retry: true in your config file to always assume the --retry to be given.

Syncing Changes from the Remote Unit

When debugging hooks, you will almost always make changes to the charm to figure out what went wrong and resolve the problem. However, it is easy to forget to apply all of those changes to your local copy of the charm once you're done.

There is a plugin in the bundle, sync-charm, which pulls down any changes made to the charm on a remote unit, and dhx makes it super easy to use. Just invoke dhx with the --sync (or just -s) option, and any changes you make during your debugging session will be automatically pulled back down when you are done:

juju dhx -s

You can also set auto_sync: true in your config file to always assume the --sync option to be given.

When using this option, however, you will sometime pick up files that are created by the charm while running which should not be pulled back down. Any file you add to the list of sync_excludes in your config will be skipped when performing the sync. The list also supports the use of wildcards.

Remote Paired Debugging

It's often useful to get another set of eyes on a problem, so dhx also makes it easy to do paired development when debugging charms. When creating a dhx session, you can import another developer's Launchpad ID to allow them to join your session.

Let's say you want to have a paired session with Bob. You would start your session with:

juju dhx -i bob

For Bob to join your session, you will need to tell him the public address of the unit, which you can get from juju status. Then, Bob can join your session using the --join (or just -j) option:

juju dhx -j $public_address

Bob will be connected and immediately brought into your tmux session.

Bob can also join the session without using DHX, using the following:

ssh -t ubuntu@$public_address 'sudo tmux attach'
comments powered by Disqus