Docker Composing for Fun and Profit

Our application at Domino is complex to say the least. I think one of the best accomplishments of the engineering team so far is that the interface into our product is probably deceptively simple. But the mechanics of getting that experience right are - as you might imagine - consistently require a lot of deep thought.

As our very own jobs page states: "This isn't your run-of-the-mill CRUD app." And that's true.

One of the difficulties that comes out is that as a result of the complexity of our application our development environment has deviated a good deal from our productionized environment. Pretty much anyone on the team can boot the app, run some basic jobs, and generally tinker around with it. But there are a number of things that won't work quite correctly. Or worse, they work but very differently than they do in a live environment.

Luckily we have the capability to spin up a live environment pretty easily, but ideally we want to catch things (and generally be able to develop against the product) without involving AWS whenever possible. It's a lot quicker to save a file and reload the application locally than it is to generate artifacts and deploy them to a server.

So I decided to go on an adventure with docker-compose yesterday to see how close I could get us to that reality.

It turns out I got pretty damn close. As of today, I can docker-compose up in our application and get an array of docker containers that have all the crucial bits of our applications. Along the way I learned some interesting bits that I thought I would share.

VirtualBox file sharing is not sbt's friend

One of the most painful bits about my attempts to set this up were the abysmal performance characteristics of VirtualBox's file sharing system. One of the nicest things about docker-machine is that it will mount /Users into the VirtualBox VM for you. So anything under /Users is directly mountable to a Docker container as if you were on a linux machine. Unfortunately, the fact that it's going through VirtualBox causes all sorts of weird performance characteristics that aren't happy.

To get around this I picked a folder on the VirtualBox VM that I wanted all our files to go to, and wrote a script that executes rsync over an SSH connection to the VM to push file changes to the root of the VM file system.

Using the environment variables that docker machine sets for me (plus one that I added), I was able to make it pretty generic:

rsync --archive \
--rsh="ssh -i $DOCKER_CERT_PATH/id_rsa -o StrictHostKeyChecking=no" \
--exclude ".git" \
--exclude "*/target" \
$(pwd) \
docker@$DOCKER_HOST_IP:/

Executed from the root of my project, this will package and upload all the relevant files to VirtualBox. Note the DOCKER_HOST_IP isn't a default environment variable. That's one I defined like so:

DOCKER_HOST_IP=$(docker-machine ip $DOCKER_MACHINE_NAME)

All other references to the DOCKER_HOST include a bunch of "tcp://..." and other garbage like that when, sometimes, you just want the IP address.

This pretty much eliminated all my file system related performance issues and had the added bonus of putting an end to the ridiculous Play-auto-reload that annoys me to no end. (I'm officially declaring my intent to burn our Play app to the ground and replace it with Lift - but that's going to be an entirely different blog post.)

Give it More Juice!

In order to get things humming nicely I had to give the VirtualBox VM some more juice. I upped its memory allowance to 4GB and gave it 4 cores to play with. You are, after all, running sbt in there.

Yo Dawg, I Heard You Like Docker

I'm using Docker to use an app that wants to use Docker. YO DAWG. 

But seriously, nesting Dockers, while possible, probably isn't the best idea for active development because then you've got to shell into the container running your application to take a look at what containers its running and tinker around with them. It's a much nicer experience to have that all available from my OS X shell with my normal docker commands. So, I did just that.

Turns out docker-compose is quite clever. If, in your environment section of your docker-compose.yml file, you define an environment variable without giving it a value it will pull the value from the currently running shell. So if you're composing a container that has docker installed, you can tell that docker CLI inside the container to connect to the very same docker daemon that is running it very easily.

In the context of an entire compose file that looks like:

Next Steps

For our situation at least, we have a bit of duplication between what I've done to get our system running locally and our devops stack that deploys our servers. We're going to be looking at de-duplicating some of that moving forward. The worst offender at the moment is a particular config file that's > 200 lines that has to be manually altered when someone wants to run this setup on their machine.

That aside though, I'm thrilled that I got to play with docker-compose a bit and it has solved a very real problem of not having a realistic environment to test my code in inside OS X.

Write Code That is Easy To Delete

This blog post has some genuinely good advice:

Every line of code written comes at a price: maintenance. To avoid paying for a lot of code, we build reusable software. The problem with code re-use is that it gets in the way of changing your mind later on.

We're always super eager to DRY up our code. The truth is, not all code needs to be DRY. I think in my career I've developed a few different philosophies for how I interact with a codebase depending on the situation that I'm in.

I think the separation between library and concrete application and the separation between long-lived and frequently-changing code are the two most impactful in my engineering style.

If I'm writing a library I'm going to be aggressive about providing consumers of the library levers to pull to make it do what they want. If I'm writing an application, I'd prefer to repeat some code or a pattern a few times before promoting it to an abstraction of its own.

If I'm writing long-lived code, I'd prefer to sit and think it through intentionally. Choose good names, think about the relationships between the moving parts. If I'm writing something that's probably going to change next week I (for better or worse) will likely concern myself less with those things.

That's kind of the tradeoff I think. It's inevitable, right? Either way, this article was thought provoking for me about how I write code. Hopefully it'll do the same for you.

All the Life Changes. At Once.

I suspect that October and November 2015 will go down in the record books. I seem to have happened upon several life changes all at once. The first, and biggest of which being getting engaged. Truthfully, if you told me at this point last year I'd be writing that sentence today I would have had a good laugh at you. Secondary to that, I left my role at Elemica at the end of October and on Monday I'm starting a new position slinging code for Domino Data Lab.

With these changes I felt that this was an appropriate time to do an introspective blog post. So, here is what has been on my mind for the past week arranged in no particular order.

Leave the Team Better Off

I was at Elemica for nearly three years. Two months shy of that marker, actually. That's the longest position I've held since graduation. It was also one of those most rewarding positions I've held.

Now is the right time to make a move, but I can't help but take a bit of pause at the fact that there are some things and people I'm going to miss every day. I helped build some stuff that's pretty damn impressive. You can actually watch Arun, the VP of the Engineering Department, presenting it at Elemica's conference this year. Maybe the full impact of it doesn't translate outside the world of supply chain. If not then just trust me. It's awesome.

I'm hoping to be at Domino much longer than three years. But however long I'm there, however long I'm anywhere, I've realized that I have a responsibility to improve the team I'm on as much as I do to improve the product we're building. Whether I'm there a year or ten, I have an opportunity to improve any team I'm on.

At Elemica, that looked like being honest when I had a perspective that was contrary to the prevailing one on the team. It looked like being willing to slog through writing and rewriting Selenium tests to try and make them reliable. Sometimes it looked like taking a moment to tweak a Jenkins plugin or our Hubot. It could also look like walking someone through Lift's form handling step by step.

These are all things that are a little bit tangental to my primary goal of delivering functionality. They are, however, equally (if not more) important.

Commutes are Toxic

During my week off I've been working on passion projects. Mostly stuff like moving the Georgia General Assembly API to a free Heroku instance, working on making TravisCI build the Lift Framework snapshot releases, restructuring the finances for Crazy Goat Creative a bit, and working on a pet project that I hope to share soon.

Meanwhile I went ahead and moved into a coworking space that's not far from home. I debated a lot about actually joining a coworking space. Going to the office was formerly this incredibly depressing proposition for me. I wasn't sure whether or not I wanted to pay for one.

This week I've seen that now it's largely a non-event. On Monday I was able to leave to space, go get gas, pick up some stuff from Publix and go home. This took the same amount of time my commute home from Elemica would have taken. And guess what, I was in a much better mood at the end of it! I didn't have to sit in stop and go traffic down 400 and then on the downtown connector to get home. Regardless of the time of day I can normally get home in about 10 minutes.

Net-net: I don't think humans were designed to sit in cars. I realized that was having a huge impact on my mood and with that element removed from my regular, weekly routine I'm already much happier. Here's hoping that the sun comes out and I can start biking – and then become much healthier.

Life Things Matter

Getting engaged has been a whirlwind of an experience. That timed with DHH's Medium post "RECONSIDER" got me thinking again about the whole work/life balance thing. David said it pretty well, so I'll just quote him:

I wanted a life beyond work. Hobbies, family, and intellectual stimulation and pursuits beyond Hacker News, what the next-next-next JavaScript framework looks like, and how we can optimize our signup funnel.

I wanted to embrace the constraints of a roughly 40-hour work week and feel good about it once it was over. Not constantly thinking I owed someone more of my precious twenties and thirties. I only get those decades once, shit if I’m going to sell them to someone for a bigger buck a later day.

Minus the (what I think is a) typo at the end, this largely articulates my feelings on the matter: "Shit if I'm going to give that time away." I only get each year once and there are no do-overs.

I want to put a dent in the universe with my work. But while there will be periods of long days, I will not allow that to become a permanent lifestyle. It will be the exception to the rule and it will be time boxed and for a specific reason (e.g. "We broke the production system and it has to be fixed ASAP" or "We're backlogged and working on hiring more people or reducing scope to fix that problem."). If the day comes where I'm not holding to that standard, then I will move to rectify that as quickly as possible.

Maybe that means I'll never be a founder of a venture-backed startup or an Executive Vice President of anything. If so, I'm okay with that. My nights sitting next to Katie on the couch drinking wine and binging on How I Met Your Mother are far more precious to me than either of those distinctions.

Honestly, companies, investors, and everyone else should be encouraging that ordering of priorities. My last post was on the importance of empathy in software engineering. The truth is that if I'm building software for humans to use, I'm probably going build software more empathetically if I'm a complete, healthy human – physically, emotionally, and spiritually. At the end of the day, that translates to better software for the organization that writes my paycheck. That, in turn, translates to software that is more valuable to the customer and capable of bearing a higher price tag.

Also, I am guaranteed of one thing in this life: I will affect others. Software comes and goes. It's unlikely my name will ever be listed alongside Alan Turing for my contributions to Computer Science. But how I impact the people that fill my life is an effect that will ripple for decades. It is my responsibility to enjoy and be present with those people and impact them well. In the long term that will mean being a present husband, a present father, and a present friend. The challenge with that is that impacting people isn't something that can be put on autopilot. If I mentally check out of those relationships because work becomes too demanding I don't stop affecting those people. I start affecting them negatively.

No paycheck or dent in the universe matters if to achieve it I end up hurting the people closest to me, missing out on the beauty and the wonder that life has to offer, and breaking myself in the process.

Conclusion

Hopefully you've enjoyed reading this somewhat random collection of thoughts. Things are pretty exciting in my life right now and that's prompted a lot of thinking. If you made it this far, thank you for entertaining my little outlet.

Thoughts on Empathy in Engineering

I've become more and more aware recently of the importance of empathy in how I do my job. Which, in some ways, is a bit comical because if you were to list some adjectives to describe the classic stereotype of the software engineer you likely wouldn't mention "empathetic" in that list. I don't know that I would either. But, nonetheless, empathy for my users – understanding them – is increasingly important for what I do.

If I don't understand my users, their goals, and the realities of their day to day then I will build software that's frustrating for them to use. If I build software that's frustrating for them to use, then they're not going to want that software, regardless of how well it solves the problem for them. In software design we spend a good bit of time thinking about usability and how it impacts the decisions users make about what software they use. Oftentimes solving a problem isn't the hard part. Solving a problem in a way that makes the user want to buy and continue using your product is far more challenging.

I can write you a piece of software that does the same thing that Mint does in a few weeks, for example, but you probably wouldn't choose to use it over Mint because it wouldn't be solving that problem in a usable way. You would be so frustrated by how my solution works that you wouldn't use it.

This gap is becoming more important for a lot of verticals that have gotten away providing a poor user experience to users. My generation is getting older. And as we're getting older, we're moving up in the ranks of the organizations we're a part of. The number of people who grew up with iPods in their pockets will eventually take over leadership positions in organizations. As they do, the expectations they place on software their companies will purchase will increase.

Playing with new toys is fun. Whether your poison is Lift (like it is for me), React, Angular, Symphony, Rails, Sinatra, Express, Meteor, or whatever else you can name — if you want to build the best software for your users, you have to have a deep understanding of who they are, what they're doing, and why they're doing it.