Automated Ambari Install

I routinely have need for a single node install of HDP for integration work. Of course I could script all of that by hand and not use a cluster manager but what would I have to blog about. I routinely prep a single instance in EC2 with a small script and then punch through the GUI setup by hand. In the interest of time and as a part of larger set of partial automation projects I am working on, I decided it would be nice to get a single node install working as hands free as possible. Here we go:

The Setup

This should get you to a clean install of Ambari with no actual Hadoop packages installed.

Blueprint Registration

The balance of this code installs the actual Hadoop packages:

Now in order to run the last section a blueprint file is needed. I created this simply by installing via the Ambari GUI to get everything how I wanted it and then dumping the blueprint to JSON to allow me to replicate it.

I did edit this file slightly by hand such as reducing the hdfs replication factor. The file that it produced was FAR more detailed than anything I found here or here.

You wont see any real output when you register the blueprint. There is a process of topology validation that can be disable though. You can check however to make sure it was accepted. I suggest you do this if you are still testing the blueprint as it will fail silently if there are errors.

Cluster Install

For the next step you need a file that maps hosts to components which is fairly simple for a single node. You need a hostmapping file.

I typically add a single sed to swap out the fqdn for the internal AWS hostname in my script.

Then submit using the following to kick off the cluster install:

When you are successful there will be some output like this:

Monitoring Progress

At this point you can for sure monitor the progress via RESTful calls like this:

But its far easier at this point to just open the web interface of Ambari on port 8080. You will see your services installing and all the pretty lights turn green letting you know things are installed and ready to use.

Ambari Services Installing

Troubleshooting etc.

PENDING HOST ASSIGNMENT

While I played with this I did see an issue where it seemed like things launched but when I checked the Ambari GUI I saw a message in the operations dialog that list the problem as “PENDING HOST ASSIGNMENT” which apparently means Ambari-agent was not online and or “registered”. I only saw this once and upon trying a clean new install test I didn’t see it again.

RESET Default password

Don’t be the person who runs Ambari on Amazon with the default password. Just don’t do it.

All these steps together should allow you to put together a nice script for launching a single node test cluster of your own. There are instructions for adding nodes to your cluster or starting with a more complex setup in hostmapping available in the Ambari docs. Depending upon your needs you could even capture an AMI once you are setup allowing you to launch a single version even faster. The nice part about this script is that you always get the newest HDP release.