TIL: Using consul-template to test local rendering of template files before using them in Nomad jobs

I’ve been using Nomad for a while, as an alternative scheduler to Kubernetes. It’s good, and I generally think I’m in good company when using it. I learned a handy trick with consul-template today and I think it’s useful enough to share here for others, and my future self.

A pain point – generating templates with Nomad

One thing I didn’t like about it was that it felt pretty fiddly to see what generated templates looked like before I used them in jobs to sent to a Nomad server.

Today though, I learned you can use consul-template, the library Nomad uses for rendering templates independently.

A worked example

Let’s say you have want to template out a file for Caddy, a popular web server, and you have a RabbitMQ server that you want to make reachable via Nomad’s fairly recent service features, that let you declare services without needing to hardcode IP addresses.

Let’s also assume you use something like Cloudflare for managing DNS, and you want to use Caddy’s nifty secure connection support to make rabbit.mysite.org available over HTTPS.

Caddy offers a few routes to demonstrate that you control a domain, as part of the domain validation flow is uses when connecting to a certificate authority like LetsEncrypt. While the default is to use the HTTP route, the DNS route is also extremely handy, and in this case probably a better fit for your use case.

You might want to end up with a template that looks a bit like this, and have it generated on the server eventually. the IP ranges and ports are fairly arbitrary in this context, as is the token, it’s just something you want to fetch from a Nomad server.

# the desired Caddyfile
rabbit.my-site.org {
	tls {
		dns cloudflare my_long_alphanumeric_token
	}

	reverse_proxy 1.2.4.5:15672
}

You can declare a service like this using a Nomad “service” job, and within that job you might have a task that looks like this. The key things I want you look at here is the use of the template stanza, and then referencing the generate Caddyfile in the call to caddy run a few lines below, where we are explicit about using the caddy file at the required path /local/Caddyfile for that job.

task "caddy" {
	driver = "docker"

	template {
		data        = file("./nomad/local/Caddyfile.tpl")
		destination = "/local/Caddyfile"
	}

	config {
	args = [
		"caddy",
		"run",
		"--config",
		"/local/Caddyfile",
		"--adapter",
		"caddyfile",

	]
	
	image = "path/to/docker-image-for-caddy-with-dns-support"
	}
}

In the template stanza, we’re reading a template file, ./nomad/local/Caddyfile.tpl on the operator machine (i.e. my laptop), which might look like this.

As you’ll see below, we’re fetching a stored Nomad ariable for Cloudflare using the call to nomadVar "nomad/jobs/caddy", then once we’ve recovered it, we’re accessing the cloudflare_token with .cloudflare_token in the go template.

Further down, you’re using Nomad’s service support for fetching information about a service rabbit-dashboard that has been declared in another job, running on another node in your cluster.

You don’t know the IP address and port, and that’s a good thing – it’s Nomad’s job to keep track of this, and deal with port or IP conflicts, and even bring RabbitMQ service online if that node running the service dies.

So, you access the necessary IP address and port using the {{.Address}}:{{.Port}} bit.

rabbit.my-site.org {
	tls {
		dns cloudflare {{ with nomadVar "nomad/jobs/caddy" }}{{ .cloudflare_token }}{{ end }}
	}

	{{ range nomadService "rabbit-dashboard" }}
	reverse_proxy {{.Address}}:{{.Port}}
	{{ end }}
}

If you want to see what this template looks like, one way is to just try running this job on the server, but that’s a bit YOLO. If there is a error in your Caddy file, then you’ve just brought down your web server.

So, you can use consul-template instead to generate the file locally first.

You can pass in a config file, but if you already have environmnent variables set, consul-template knows to use them, and fetch data from a Nomad serve when compiling the template

Here’s an example below of using consul-template below to generate the file locally at ./Caddyfile.rendered.caddyfile. I’ve passed in two environment variables to the command, and also set consul-template to run just once with -once, instead of the default of watcing for any changes made to the Caddyfile.app.tpl file:

NOMAD_ADDR="https://nomad.mysite.org" \
NOMAD_TOKEN="long-token" \
consul-template -template="./nomad/local/Caddyfile.app.tpl:Caddyfile.rendered.caddyfile" -once

This is helpful for manually eyeballing the output so you have no surprises, but crucially, you can also do things like run Caddy’s own validation checks safely on your own machine first against your rendered file, like so:

caddy validate --config Caddyfile.rendered.caddyfile

I’ve used Caddy as an example here because it’s a fairly common tool with nice validation support of files, but you this approach should work with basically any tool that has support for validating files before you use them on servers.

Summary

Nomad is a really nice alternative to Kubernetes if you want many of the things Kubernetes offers in a somewhat simpler package, and others have gone into detail about why this is the case. Karan Sharma has a nice post about this from last summer, and also has a good blog post about using Nomad as a home server too.

I think using consul-template like this makes working with Nomad quite a bit nicer, and while there are some docs on the project on github, didin’t come across any tutorials or guidance about using consul template in this way.

Hopefully this is useful to others as well as my future self.

The inevitable note about the license change: The change of the Nomad software license from being totally open source to a business source one might take some of the shine off Nomad. I’m not sure if would have been used inside Cloudflare or Fly.io if it was. But for many of us not building a commercial product, Nomad still remains a compelling option, and I hope this post makes it a bit more accessible to folks considering using it.