Thinking About Tests

I've been thinking a lot about tests lately. The automated, software development kind, not the school kind. I've been thinking a lot about tests because I've realized that my thinking about them has been broken. Specifically, around how to test Lift web applications. Something that I've had to come to terms with is that my schooling and experience up until a few years ago did a horrible job of educating me regarding when tests are needed and why.

I remember being taught a lot of text book mantras when I was in school regarding development. Things such as, "testing will prove your software works" and "testing will help you know if you broke something else." For a long time, I regarded the former mantra with a bit of incredulity ("I test it when I'm writing it, why am I going to the effort of writing more code to do the thing I've already done?") and the latter with an air of hubris ("If I build my code correctly, the scope of any change will be limited and I'll catch problems myself.")

For what it's worth, I think the Test Driven Development evangelists got it wrong too. At least some of them did. Testing as a matter of doctrine without understanding the whys and wherefores will tend toward the software engineer's version of hoarding if done without care. Tests keep adding up and adding up, some may not be written properly, and changing any minor detail in the application causes an increasing number of tests to fail over time - at least some of which are failures that are a product of how a particular test is being run rather than what is being tested. I understand now that there are lots of good tools to help avoid that, but there were (and are) enough examples of people not using those tools that my general reaction to the practice was summed up in "Why?"

Though I haven't sipped of the TDD kool-aid, I do take testing a lot more seriously than I once did. So, what changed?

Well, at work we're engineering at a much larger scale than anything I've ever been a part of before. It's literally impossible for one person to have the entire scope of the system in their head at any given time. It doesn't all fit. As a result of this, it's very easy for someone on our team to make a change without knowing the downstream implications before they make the change. Work in that kind of environment without sufficient testing, intentionally or otherwise, and you're in for a world of hurt as a software engineer. All of a sudden your change to a line of XML is causing buttons to go missing in the running application and you have no idea why.

So, maybe the answer is to just write as many tests as you can and by doing so you'll win, right? If only it were so.

I've also learned, first with Anchor Tab and again later on, that if you choose to test incorrectly, then you're also in for a world of hurt. Take Anchor Tab as an example. I decided to use Selenium to test everything when I started adding tests to the codebase. I thought I was incredibly clever, because, after all, what I cared about was that the user could do what they wanted to do.

What I've found to be true as a result of that experience is that Selenium isn't really a great API to do the same task as a unit test. The inherent contract it imposes of acting like a user is simultaneously its biggest strength and the blade that you will cut yourself on if you try to bend that contract to your will in unintended ways. Selenium, and technologies like it, is fantastic and certainly has a role in a larger testing strategy, but it isn't the larger testing strategy – even for a web application. It doesn't replace unit or integration testing, as I thought, but supplements it in different ways.

As a result of these realizations and ponderings, I've realized my Selenium misadventures stemmed from what College Student Matt learned about testing. I would teach College Student Matt differently today.

Instead of teaching things like "tests prove your app works" and so on, I would do my best to drill into his head the idea that tests are tools for answering a question. They are not something that magically makes your code harder, better, faster, or stronger. A test is merely a piece of code that answers a question about your code and you should probably know what that question is before writing your test. Also, the question itself affects the kind of test you need to write and making the right decision on the type of test is important for minimizing future work on the test while simultaneously maximizing the usefulness of the test. At the end of the day, you're answering this question for someone else and and making that person's life as easy as possible should be a priority.

So, for example:

  • If you are writing a method that parses an <a> tag in HTML into a data structure, one question you'll want to ask is "Will my method behave sensibly for all kinds of input it could see?" and you'll want to write a unit test on that method, where the method itself is completely isolated from any other component of the system.
  • If you've finished writing and wiring up a series of components that all work together, the question you'll want to ask is "Is the entire series of components behaving sensibly today?" and you'll want to write an integration test on that series of components where everything is As Real As Possible™.
  • If you've finished building out a new series of screens in your web application, the question you'll want to ask is "Can a user successfully accomplish their goal in the web application?" and you'll want to write an acceptance test using a tool like Selenium or PhantomJS.

The unit tests and integration tests should try to think of all reasonable scenarios and ensure that things function correctly, the former focusing on an individual method and the latter on a series of components. (Do note I wrote all reasonable scenarios, not all possible scenarios.) Meanwhile, acceptance tests should focus on determining whether or not a user interacting with an application can accomplish a set of goals. What's the difference, you ask?

Well, I hold the opinion that acceptance tests should not overly concern themselves with a lot of error conditions. I came to that conclusion after writing a lot of Selenium tests that do test for a lot of error conditions. What I ended up with was a series of tests that took a long time to run, tended to be a bear to update when things changed, and gained a reputation on our team for being unreliable due to a lot of obscure interactions that I believe were the result of my distorting the intended Selenium contract.

Verily, things will slip through even if you have an excellent net of unit tests, integration tests, and acceptance tests. But here's the deal, even with 100% code coverage things will slip through. Sorry, but it's true. That one obnoxious customer is going to enter that one unicode character you didn't test for and your whole app is probably going to blow up because your particular OS-specific version of your database recognizes that as an escape sequence due to an obscure bug in character processing. Welcome to software!

The question for you is how tightly can you close that net without making you and everyone else on your team want to pull their hair out? For my part, the new guidelines that I'm going to be suggesting for adoption to our team at work will probably look something like this:

  1. Lift snippets, actors, etc get unit tested unless they're basic snippets. Where "basic" means an ID has been pulled from a URL, I've retrieved something from the database, and rendered its contents on a page. That removes about 99% of the admin screens we have for just doing things like listing a bunch of objects with links for edit and delete.
  2. JavaScript gets tested if it contains any business logic. If any JS is looking at data that originated from our database and evaluating what it "means" and acting on that, it needs unit testing at least. Perhaps integration testing in addition to / in place of the unit tests depending on the situation.
  3. The app has acceptance tests that are defined that represent goals from the point of view of a user. These tests are ideally run in some sort of headless mode locally and for every pull request we make for speed, and run in our entire browser matrix before release. We've gotten into the habit of making them more unit-test like than any of us enjoy working with and we have to run the browser to run them today, which is a bear for different reasons.

The goal, as always, is for tests to be useful without being frustrating to maintain. And I think these guidelines will strike that balance for us.

So, hopefully you've found my ramblings on testing enlightening. I hope you'll consider sharing this with your friends and letting me know what you think. I always love hearing good feedback on these posts.

Until next time, folks.

To: Sen. Isakson / Re: Net Neutrality

I wrote an email asking Senator Isakson to support the FCCs moves to help control the abuses of companies like Comcast on the internet. I got a canned response (no surprise), and tried to reply to that only to find that the email address that Isakson's staff sends emails from isn't a real email address. There's no way for me to actually have a conversation with someone from his office about this issue.

Below, I've reproduced his response to my initial email and my response to him. I don't have record of my original email.

Read More

Trouble Staying Motivated

Life tends to present itself as a series of seasons. Some characterized by career success, some by personal growth, and others by something else entirely. Whatever season I'm in now, it hasn't been the easiest for any of my side projects.

The Get Open student mentors hip initiative was a total flop. The rubber hit the road in the pilot and the model just broke down. Communication stopped after a few emails. It was a limited, perhaps foolish idea. But I don't regret trying it. And to say that Depend On has hit a wall is a bit of an understatement. I'm still having continual issues with version number comparison aside from the multitude of other things needed for MVP, and trying to open source a significant portion of the application without just open sourcing the entire thing turned out to be more of a nightmare than I first imagined because it made my build pipeline five degrees more complex. Just looking at the code seems to paralyze me these days.

I've done some good things under the Georgia Open Data Project imprint. They're just things that nobody else is particularly interested in, which is unfortunate. The functionality of my GAODP projects is about what you can expect of most things that only have a single person with a full time job on the side doing: minimal. And unfortunately, though there have been some good small wins on Lift recently I haven't gotten the itch to dig into it recently. There are some things I'd like to see done, and have speculated about how to do them, but finding the energy to do so these days has been harder than I recall in recent history.

I wrote last summer about the Economies of Scaling Back and the importance of maintaining a balance between all the things you want to do and still having a life. Maybe I failed to scale back enough things or scale back soon enough and am now going to have to tolerate a period of burn out. Or maybe it's something else entirely – I do have a notorious history of being incredibly lethargic when the weather gets cold. And man has it been cold recently. I do my best to take it as it comes, but verily my Type-A frustration at myself for not being able to do more is easier to deal with one some days than others.

But I'm sure you're wondering: why am I typing this into a box with the intent of sharing it with the Internet?

I know there are a lot of folks that are interested in Depend On, and on some level I feel as though I owe you all an explanation as to why nothing has happened lately. But more important than that, I think that we – as the people on the front lines of the technology industry – commit a travesty in how we represent our work sometimes and that's reinforced when people don't say "Hey I love what I do, but I haven't been able to like it as much as normal recently."

Believe me when I say it: I love coding. I love technology. There's no feeling quite like the rubber hitting the road and pulling off some magic for a real person. It's what I would be doing even if I couldn't do it full time as my job. But I go through periods – months sometimes – where I come home at the end of the day and the last thing I want to look at is a line of code. I get these big great ideas, spend a lot of money to start making them happen, and then halfway through it's as if my brain decides it was never interested in it in the first place and getting an ounce of productive work done on it becomes about as easy as pushing an F-350 truck up a hill by yourself. (And yes, I'm aware that my southerner is showing with that choice of mental picture.)

It could be the case that this experience is entirely unique to me, though I doubt it. So if anyone tells you that if you learn to code your life will be awesome because really you just get to do what you love all the time, they're telling you the truth. But let me disabuse you of any mental picture of being Super Coding Machine™ that lives, breathes, and eats code and never gets tired of coding code all the code day long. Because just like everyone else there are weeks and months on end where even though I'm doing what I love I find myself lacking that spark that drives me to push it to the next level.

In spite of this rut, for lack of a better term, I'm still going to get up tomorrow and put everything I can into my day job and be so thankful that I have the privilidge of doing so. And that's a bit of why I still say that I love coding: I'm still thankful to be doing it even in the seasons it's not as likeable / interesting / insert positive adjective here as it was a few months ago. And I do that knowing that it will be that again for me before too long. These things are always cyclical. C'est la vie. But it might be a bit before you see a lot of side work from me.

So, what's next?

Well I know better than to try and force things when I'm not in the state of mind to do them well. I imagine I'm going to be buying some books that have been sitting on my waiting list for awhile. I'll probably still be tinkering around with GAODP, so if you have an interest in helping to make Georgia's Data more accessible hit me up on Twitter (@farmdawgnation). Some extra interest would definately help kick up my momentum on that. I'm also going to spend some more time writing than I have in recent months, so hopefully this blog will be getting some more love, too.

I would love to hear your experiences with what I've talked about in this post. Have you experienced these cycles like me or are you the Super Coding Machine™ that I made a bit of fun of earlier in the post?

Until next time, folks.

So, I discovered Patreon

So, this week I discovered Patreon. It's a nice service that allows me to chip in some cash to creatives whose content I really enjoy. To be frank, I'm actually quite giddy about this. It's probably going to destroy my coffee budget if I'm not careful. And we all know there are few things that can convince me to sacrifice coffee.

What's funny is that I'm an unapologetic skeptic. I actually had a friend last night say that I'm a good "no man," and intended it as a compliment. I consider my ability to poke holes and iterate on ideas an asset. When I come to the conclusion that something is a "good idea" it means that I've hit it with a mental baseball bat a few times and am convinced its solid. As a result, it's rare that I come across something and get immediately lit up with unbridled enthusiasm. Since that's happened with Patreon, I thought it'd be cool to share why in blog format.

All of you know that I'm a software engineer. I get to build things for a living and, though it may surprise you to hear, there's a significant amount of creativity that goes into this. I'm fortunate enough to be compensated very well by the industry for doing what I love to do. But, to be honest, writing code is what I would be doing even if I wasn't getting paid. It's my craft. I have lots of little side projects I build out for giggles. It's actually become a bit overwhelming lately because I'm also "Project ADD," but that's a different topic.

Unfortunately there are a lot of talented creatives out there who aren't in my situation. For them, working on their passion means doing sucky full-time or part-time jobs that may not even pay the rent. In many creative arenas, if the Big Industry™ hasn't decided they want you, you'll have a hard time making a living. And I happen to hold the opinion that Big Industry™ tends to do a sucky job at evaluating whether things are good or bad. Lots of things fall through the cracks of their rubrics and lot of talented folks never catch their eye. (I'm looking at you in particular, Fox.)

Patreon allows me to participate in a rich history of patronage. I get to support artists I love in a way that is both easy and accountable, as they're held to doing something in return for contributions that are sent their way. In some ways, it's like a much longer term Kickstarter arrangement. I'd argue that as someone who has disposable income and immensely enjoys the things these folks produce, that I have something of an obligation to make my voice heard – both with my voice and with my wallet. They're producing stuff that is enjoyable and valuable to me and that I want them to continue doing that. These folks are participating in the creation of our era's culture. That's worth supporting.

So, I bet I can guess the next question in your head: who am I supporting? Well, let me tell you a bit about the first two creators I've backed.

Walk off the Earth

©2012 Walk off the Earth. Click for the original.

©2012 Walk off the Earth. Click for the original.

I'm sure that anyone who I've talked music with recently is sick of me talking about Walk off the Earth, but I'm still mesmerized by the amount of talent they have. Each member of the band plays some insane number of instruments and each of their music videos is a pure joy to watch. Of particular note they hold the distinction of being the first band to ever do a rendition of a Taylor Swift song that I actually enjoy. (That's an accomplishment.)

 I decided to back them because the things they're producing are ambitious and consistently enjoyable. Whenever I can log into YouTube and see "Hey! WOTE has posted a new video!" it's always a good day. I'm also pretty sure Sarah Blackwood's voice is hypnotic. My current favorite demonstration of their talent is their cover of Lorde's Royals, where their instruments play musical chairs throughout the song (musical players?).

I can't say enough good things about them, so I'll just say this: check them out.

Mary Kate Wiles

©2014 Mary Kate Wiles. Click for the original.

©2014 Mary Kate Wiles. Click for the original.

I actually originally discovered Mary Kate Wiles for the first time earlier last week. Kate, a friend of mine from church, posted a link to her video blog about post-job depression. After that I started clicking around to see some of the stuff she had been involved in. As a result, I stumbled into watching The Lizzie Bennet Diaries, a web series she had acted in that is based on Pride and Prejudice.

(I may or may not have watched the entire series in three days.)

Mary Kate portrayed Lizzie's younger sister, Lydia. To be honest, my expectations for the character and the series as a whole were low going in. (See aforementioned skepticism.) But I would have had to eat my words if I'd ever publicized that opinion before finishing the series. Lydia is the character in the series that wins the award for giving me the most feels, and that is not an easy award to come by. One scene, in particular, where Lydia realizes she has been seriously betrayed took the cake for me.

This is the look of heartbreak. This moment broke my heart. "Consequences - Ep: 85." ©2013 Pemberly Digital.

This is the look of heartbreak. This moment broke my heart. "Consequences - Ep: 85." ©2013 Pemberly Digital.

I could tell you what's happening in this scene, or link you directly to the video, but I honestly feel that I'd be doing you a disservice. Watch the series from the beginning, and thank me later. Lizzie's videos are the "primary" narrative (for lack of a better term), but Lydia's are absolutely essential for understanding the character and made a huge difference for me. The link in this paragraph is a playlist containing all the relevant videos from the series.

After watching The Lizzie Bennet Diaries and a few of Mary Kate's other videos, supporting her was an obvious decision. I think she's got a ton of talent, and her video blogs have a really raw, down-to-earth quality that resonates with me. I want people with that talent and those personality qualities to be successful. Hands down.

If you're interested in finding out more about her, check out these links:

Tradition of Patronage

By supporting these artists, I'm joining in a long history of patronage, and I encourage you to do the same. There are good people out there doing new, interesting things with their creative talent. Find them, and figure out how to support them. It's worth your time and money to do so.

If you're also a Patreon user, I'd love recommendations for folks who are on there that you think are talented. I think that supporting these artists is actually going to get a line item in my budget in 2015, so I may be looking for a few more to add to my list of support for the year. And, as always, if you want to leave me a comment for any other reason please feel free to do so.

Until next time, kids.

Open Source Software at Elemica

So I did a write up for Elemica about our use of Open Source Software within the organization. It covered some allegorical business examples as an argument for OSS, talks a good bit about Lift and Chef, and goes on to discuss a few of the things that we've published recently. If that sounds like a something you'd be interested in reading about, it went live today.

Read the post here.

As always, I would love to know what you think. :-)