What hasn't been deployed?

26 May 2015

I am obsessed about automation, with that, comes the notion of asking simple
questions and getting simple answers you can create actions with.

One of the questions I constantly ask myself is “What hasn’t been deployed yet?”.

We deploy multiple times a day, we try not to have more than 2-3 commits merged
without being deployed, this way, it’s easy to track on production whether a
commit is causing performance issues or something like that.

Another reason for that is to “give a heads up” to engineers with big features
going into production to check and make sure everything is working as expected.

How do we do it?

We use our own version of Hubot in Slack, we use it for all kinds of things (I blogged about it in the past as well).

So, we just ask gbot show me ENV diff.

It’s pretty simple really, all you need is an HTTP endpoint on your server that will return the git sha and than you just construct a Github link from it.

I originally thought about doing it using Github API but the return text to Slack was too long sometimes, this way it’s just a click away and Github does a good job with the compare view IMHO.

Here’s the coffee-script (Redacted version with only a single ENV)

	module.exports = (robot) ->
	  robot.respond /show me prod diff/i, (msg) ->
	
	
	    msg.http("http://#{YOUR_DOMAIN}/git_sha?rand=#{Math.random()}")
	      .get() (err, res, body) ->
	        commit_link = "YOUR_REPO_LOCATION/compare/#{body}...master"
	        message = "This is the diff from [#{original_env}] is: #{commit_link} "
	        msg.send(message)

What are we using it for?

1. Do we need to clear cache for parts of the website?

If we launched a redesign for example, added A/B testing, or any other thing
that might be affected by parts of the website being cached, we need to make
sure we clear the cache after the deployment

2. Do we need to migrate the DB?

We don’t use Rails migrations on production, the database is just to big and
the risk of it bringing down the site is too great, so we have a different
strategy for migrations. If there’s migration in the code that’s not in
production yet, this is the last chance to make sure we don’t deploy before it
happens.

3. Do we need to push bulk actions?

We often have features that count on other bulk actions being done. For
example, we recently added a feature that will bump places with photos above
others in your search results, this needs a bulk action to re-index our search
cluster completely, this means, we have to do it before we deploy this to
production.

4. Do we need configuration changes?

We manage all of our servers with chef, so if you are counting on some
configuration file with a secret or some 3rd party API key being there, we need
to make sure that configuration is converged through all of the cluster.

Lets take this piece of code for example, this code is counting on a distanceScore function in SOLR. This is something I wrote in order to get distance rings score in SOLR (post coming soon), if this is not deployed to SOLR before this code is in production it will break.

5. Number of commits

Like I mentioned in the beginning of this post, if we have more than 2-3
commits currently unmerged we consider this to be a higher risk deployment. So
this is really to catch whether we have more than the usual amount of things.

If that happens we usually just go ahead and deploy before merging anything
else into master.

Last resort

From this post, you might think we just push to master and check after the
fact. That is not true, we are code reviewing anything that goes into master
and catch that before, but just in case something happens through, we want to
make sure we catch it before we release it.

Counting on git flow

We kinda adopted git flow at Gogobot a few years back, we each work on
feature branches and master is always deployable.

This is why we can adopt this approach, we just each work in a black box (That
black box can be deployed to dev/staging servers) while production is being
deployed many times in the process.

Make your own

Even if you are not working on a highly dynamic web application, you can still
make your own like: “Which client has this version”, “Which server has which
release” and more.

If you find yourself needing a couple of clicks that will break your current
flow, automate it.

Since we do everything in the chat room, this is a perfect workflow for us

Comments?

If you have any comments or questions, leave it under the post here, I’d love
to hear/read it.