Posts

CloudKick Monitoring of Load-Balanced HTTP Services

So, recently we made the jump at OpenStudy from a single monolithic process to a multiple-process architecture for the main site. When engineering this, Antonio essentially split out our Actors (which manage much of our business logic) and our Web frontend components (Comets for realtime push, session information, etc) into two separate entities designed to run with RabbitMQ in the middle. The good part about this is it means that we can scale out our web load horizontally, and we’ve been serving the site using two web processes since we made this change, which has been great. Naturally, however, any good change introduces some bugs. We have had quite a few that proved troublesome, but even more so by the fact that we didn’t really have a good way to monitor individual web processes from CloudKick. There didn’t seem to be a published solution for monitoring individual nodes in a load-balanced array without pulling down CloudKick IP addresses and allowing those through your firewall, which we hated because it would require our Firewall rules to change regularly and in an automated fashion. So, we were stuck with an external monitor on the public URL which, in the event one process was failing while the others were fine, would send many failure and recovery emails as HAProxy round-robined between servers as each of the monitors subsequently hit the server. Not ideal.



Pattern Matching: Flow Control Made Beautiful

One of the things that really drew me to OpenStudy when I was considering it as the next step for my career was the use of Scala as the company’s weapon of choice. If you’re anything like me you never got any direct experience with Scala in your formal Computer Science education. In fact, the only reason that I had ever heard of Scala was because I had a professor at UGA, Dr. Miller, who was using it for his research. After about eight months of working for OpenStudy, I still find things to be amazed at with Scala. One of the first examples of things that blew my mind was the Scala pattern matching. In most Scala pattern matching scenarios, what you’ll see is pattern matching on Scala case classes. These case classes are pretty clever concepts in and of themselves, but that’s an entirely different topic for a different day. Let’s take Scala’s Option type. Options can have one of two values: they can either be a None or a Some case class containing an instance of a specific type. So, I could, for example, have this variable:


Bring a Snack to Work

Something that I’ve noticed about my development process in the past is that while I’m turning problems over in my head I’m really susceptible to distractions. In fact, I’m really susceptible to distractions when it comes to doing anything that involves what I call a “change of mental context” — or, rather, anytime my brain has to shift gears. Unfortunately for my desire to go all ahead full on tasks, I’m in the middle of a bug fixing cycle and this “change of context” thing happens a lot when I’m working on bugs. Worse yet, later in the afternoon when I start getting hungry for dinner, or if my stomach is agitated because of a rough cup of coffee, things get worse. If I don’t keep this aspect of my attention span in check I run the risk of blowing an entire afternoon of potentially productive time. So, yesterday I decided to try something new. During my grocery run that morning I decided to purchase a box of the large Kroger trail mix. While packing my lunch, I scooped a few handfuls of the trail mix into a bag and packed it alongside my usual sandwich. In between my stints of working on the email verification feature that we shipped at OpenStudy this morning I was working on fixing various bugs that we had found around the site. Naturally, different problems call for different solutions and different solutions call for some thought into those solutions. These various thoughts about different bugs collectively represent a lot of opportunity for distraction. So, in an attempt to avoid any stomach-fueled aimless staring, I brought a snack to work.