Deploying ElasticSearch clusters with Juju and Ansible

ElasticSearch is one of those tools that’s just handy to have. When combined with Logstash and Kibana, you can use the “ELK” stack for all sorts of things.

One thing we’d like to see is to bring these capabilities to our users. So I’ve been working on bundling together some of our ElasticSearch resources in Ubuntu so we can bring them to the cloud.

This is a standalone ElasticSearch cluster. I’ve added a few things:

A Production-ready Stack

One of the nice things about this bundle is it uses our ElasticSearch charm. Right off the bat you’re getting a charm that we’re using in production today. That means it’s tested (see the included tests, and it also uses many of the modern techniques for writing a charm. In this case, Michael Nelson leveraged Ansible to do most of the heavy lifting. Our ability to consume multiple tools to get the job done is one of the nice things that we can support in charms, so if you prefer a certain tool, then by all means use it! It also just consumes upstream packages, so you’ll always have a fresh version.

Why ElasticSearch?

The major use of ElasticSearch is within Ubuntu’s Software Store for Unity 8. On Ubuntu Touch anytime a user is searching for and navigating the store they’re using ElasticSearch. ElasticSearch was chosen for a few reasons.

The quick way to get up and running

Ok so let’s deploy this badboy. Assuming you’re brand new:

sudo apt-get install juju juju-quickstart
juju quickstart bundle:elasticsearch/cluster

At this point follow the ncurses menus to pick which cloud you want to deploy to and add your credentials and then Juju will bootstrap and start deploying ES.

Total deployment time will depend on where you’re deploying to, but on AWS US East 1 I can go from zero to cluster in about 10 minutes. By default I give you one Kibana node and one ElasticSearch node. For ElasticSearch we ensure you’re node has at least 4 CPU cores and 16GB of RAM. You can follow the instructions to load up the sample data and the other sample plugins I’ve included. Just go to the ip address of the Kibana unit in your browser and start searching:

Now you’re ready to horizontally scale:

 juju add-unit elasticsearch

To add a new node.

And that’s it

You now have an ElasticSearch cluster. Using the same best practices that we’re doing in production. Amir Sanjar and I are prototyping bundling the ElasticSearch Hadoop plugin in a bundle as well, though that’s not quite ready (testing and help wanted!).

As you can see, we’ve barely scratched the surface. Chuck Butler’s been reworking the Logstash charm for Trusty, we expect that to land soon. We plan to make the charm a subordinate, you can just plop it onto any unit. The idea is to enable people to put logstash’s indexer on every unit they can shove all they care about into ElasticSearch.

comments powered by Disqus