Your puppet code base makes you fear the apocalypse

Let me paint you a picture. At some point in time someone said ‘hey, wouldn’t it be great if we could manage our servers with that new puppet thing’. ‘Great’, said everyone, ‘Let’s do that and the way we have always done it.’. And that, my friends, is how you end up where we are.

Reading our puppet code base reads much like a bad folding story. Everyone had a plot line and tried to weave into a flowing narrative. And like all badly written stories it has many dead ends, twists, continuity issues and obscure meanings. You get that from many different authors - all code bases face the same problem.

So in this particular story I’m going to tell how we began solving some of the mysteries within this Odyssey like tail.

Without mixing my metaphors, it starts with two families living in a village. The Webs the Web-Trans. Web-Trans were an old family of servers that had been managed with the ‘old ways’ of building servers. Rather than being a respected family of web servers, people started to fear them because they are just not like other web servers.

The Webs on the other hand, were a new addition to the village. Much more certain in themselves. But still with that titillating vulnerability that you don’t quite know what you’re getting.

Conformity became a very strong movement in the village - no one liked odd things anymore. Something had to change. Now the simple way of fixing this conflict would have been to start a village war where hopefully one side would be wiped out and forever erased from the commit history. But no, not yet at least.

The messy problem is that these two web families sometimes used different and sometimes the same paths through the village to get to the end of whatever they are getting to the end of; let’s just call it a happy wedding where they serve web requests. And along each of these pathways that the families take, they often stop and ask the village Gods for directions because they forgot their maps, or lost their way or whatever. So the people that have planned the wedding don’t always know if one of the web families will turn up, be dressed the right way, or have their heads on.

So how do we make certain that whatever path our web families take they turn up to the wedding the way we want them? We first of all describe to them how they should look and then we test that they are what we expect.

Now I want to talk about Zombies - see it is very hard to stay in your storyline, but I do want to break out of this villagey weddingy metaphor and just start getting to the real story.

So we have two ‘types’ of web servers that do exactly the same thing but take two different paths to get there, one with more certainty than the other. Okay, pick the more certain one and start testing against that.

In puppet world (man, should have used a story about muppets) we can test modules with things like puppet-rspec and other tools including using vagrant to provision a host for us. That is a good way of doing some quick testing of your manifests. But we wanted to test that the two types of web server actually produced the same web server configuration state. So when the village war started we could be assured that the victor would be what we wanted. With that information we can then even wipe out the entire village, make a new village with completely different paths and lights and stop signs and discos and everything and still have the same type of web server at the end.

We decided to introduce ServerSpec to the village. As you may know it is a way of testing your actual server state with rspec code. It ties in nicely with vagrant and is a great addition to your TDD infrastructure.

You can use the documentation for serverspec to set up your testing, but here is how we test both types of web hosts without duplicating the testing. Having the same expected state means we can begin to remove the obvious differences and validate that we still get what we expect in a web server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
spec/
├── mp_web
│   ├── alerting_spec.rb
│   ├── common_spec.rb
│   ├── logrotate_spec.rb
│   ├── nginx_spec.rb
│   └── splunk_spec.rb
├── mp_web_trans -> mp_web
├── shared
│   ├── common
│   │   ├── init.rb
│   │   └── mp_common.rb
│   ├── logging
│   │   ├── init.rb
│   │   └── mp_logrotate.rb
│   └── monitoring
│       └── init.rb
└── spec_helper.rb

In the above layout, a vagrant host called mp_web will execute a bunch of tests including some shared tests. By using the symlink for mp_web_trans we can perform the same tests for mp_web as we do for mp_web_trans hosts. Now when a change is made to the mp_web_trans hosts we can validate that they will work the same as a mp_web host. Of course this makes deploying changes a much more assured thing.

So when the Zombie apocalypse comes and the village is under attack and the story starts taking a surprising turn, our wedding can still go on and be a happy event driven experience.

oh no

source