photo

Part 1 - API

Lions and tigers eh? This is the first installment of a three part series, as I find it easier to split things up into smaller, manageable pieces, I decided to apply it to this blog post as well. The first thing we will be covering is the API.

API

One of our many projects here at MLS is a content API we use internally to support our other other projects. Building a REST API can be a fairly straight forward process, there are many tools available to streamline the workflow, but it can also provide some interesting challenges that, if not prepared for, can cause some major headaches.

Once we had our tools in place, the API itself was pretty straight forward to build. We chose node.js for the language, and restify as the main framework.

var restify = require('restify')
  , server = restify.createServer()

server.get({path: '/', version: '0.0.1'}
, function(req, res, next) {
  return res.send(200)
})

server.listen(8080)

With just a few lines of code, we have the beginnings of an API. Of course there are many other steps involved, but codewise, we have an API with a single route ready to go. So, let’s talk about a few things that may improve development.

Versioning

One of the decisions made early on was to require versioning in our API, which has helped out tremendously in pushing new code forward while still supporting existing routes and functionality. With restify, we have the ability to version right out of the box, all we have to do is require the client always specify which version of the route they want for a requests:

server.use(function(req, res, next) {
  if (!req.headers['x-api-version'] && !req.headers['accept-version']) {
    return next(new restify.BadDigestError('InvalidVersion'))
  }
  return next()
}

We strongly encourage requiring a specific route version from your clients right from the beginning, if only to make it clear which version of a route is being used for both you and the client. Having a route default to the latest version may seem like a good thing at the time, but what happens when you make breaking changes between versions? Did that Friday afternoon deploy incidentally break other apps? This is something that should be considered when designing your API.

It may seem fairly trivial to have a route default to a version, and then document the behavior. However, nobody will read your documentation and you will still have angry users when those breaking changes arrive. The versioning system we use for our API is semver. If the client really wants to automatically be upgraded to the latest API version, they can send us an wildcard version (‘*’) with each request and get that behavior. But that means they have made an explicit choice and we’re ok with that.

Tracking

OK, versioning is great, and now you have multiple versions of a route, which means that at some point, hopefully, the older versions and supporting code can be removed. But how can you tell? We use Datadog to track metrics such as this. Using node-datadog, it is simple to add metrics throughout your code. We added specific metrics for requests to each route for a given version and API key. Now we can track which versions for which routes are being used, and by who.

var datadog = require('node-dogstatsd')
  , statsd = new datadog.StatsD('localhost', 8125)

// Setup the middleware to capture
server.use(function(req, res, next) {
  var tags = []
  tags.push('version:' + req.version(req))
  tags.push('path: ' + req.route.path)
  tags.push('username:' + myCustomGetUsername(req))
  statsd.increment('requests', 1, tags)
  return next()
})

This snippet shows how we collect stats and push them into Datadog. It doesn’t really matter how you get the information, as long as you find it and track it. Logging would also work just fine, the important part here is that we need a way to find out if anyone is using code paths marked for removal.

Marking a route or version as deprecated is great for transparency, but trying to enforce a specific time duration of a route’s continued life won’t always work with your client’s release schedule. How important is it to remove old code when you still have a large percentage of consumers using it? Getting the information about which routes and versions of your API are actively used will allow you to make these decisions more easily.

Here is an example graph of what Datadog can do:

The requests are from a development environment so the stats are a bit wonky. Datadog provides many ways to configure the graphs and tags sent by the app, use what works best for you.

Send Extras

While it is important to know how your API is getting used, it is also important to let your users know what you are doing with it. At MLS, as we push our API forward, we ensure that version numbers are consistent across all routes. Sometimes this means simply adding an accepted version to a route:

server.get({path: '/', versions: ['0.0.1', '0.0.2']}
, function(req, res, next) {
  return res.send(200)
})

Great, that was easy, but it would be useful for our users to know if they can upgrade to the new version. Let’s add some code and give our users more information:

server.use(function(req, res, next) {
  var v = req.route ? req.route.versions || [req.route.version] : null
  if (v && v.length) {
    res.header('x-api-versions-available', v.join(', ')) // 0.0.1, 0.0.2
    res.header('x-api-version', v[v.length - 1])         // 0.0.2
  }
  return next()
})

We also might want let the end user know that a route may no longer be available in the future, so let’s add a made up ‘x-api-deprecated’ header:

function deprecate(req, res, next) {
  res.header('x-api-deprecated', 'true');
  return next()
}

server.get({path: '/', version: '0.0.1'}
, deprecate
, function(req, res, next) {
  return res.send(200)
})

server.get({path: '/', version: '0.0.2'}
, function(req, res, next) {
  return res.send(200, 'Hello there!')
})

Self-documenting APIs are helpful, but they do not replace traditional docs. However, adding this kind of information to your API responses opens the door for your clients to add more intelligence in their own apps, like tracking which API calls can be easily upgraded without having to look back and forth between code and documentation, or accidentally skipping over a 1.1.0 and 1.0.1 version change.

So far, we’ve briefly touched on the subject of APIs, in the next segment, we will discuss API specification, more specifically, route specification, and take another step forward towards self-documentation.

Part 2 - Specifications

 photo

We kicked off our first-ever MLSDev Hack Day last month! The idea was to give our developers a chance to experiment and work on creative projects. We started the morning by taking over the conference room with wires and monitors. We ate bagels and listened to each person describe their plans for the day. Developers divided up, donned headphones, and started coding. The day passed rapidly with pizza, snacks, and beer. The next morning we gathered for a bit of show and tell. Here are some of the ideas and prototypes that were created in less than 24 hours.

DynamoDB acceleration by @jdslatts

Justin developed a process to use Amazon Web Services’ DynamoDB NoSQL database as a MySQL replication slave for de-normalized query data. This allowed him to replace the backend for EOS, the MLS API, with DynamoDB as a low latency, highly scalable back-end for data stored in MySQL by our CMS. A completely unscientific benchmark showed roughly a 40X improvment in uncached query performance. The prototype was built using Python, Node.js, and MySQL triggers.

Draftr by @hansyg and @sorensen

Hans and Beau developed Draftr, a real-time representation of a player draft, such as MLS SuperDraft. They built an the interface that allows for teams to go “on the clock” with a countdown ticker and timeout options. Once a player is chosen, administrators broadcast out the pick (top right box). Admins can also broadcast one-way color-commentary to users in the box below. Draftr also allows for the pick order to be reorganized based on trades. All moves, trades and commentary are in real time and are broadcast to any user connected on the client side. The prototype is built in node.js and websockets (sockjs).

Visual Schedule by @brycekahle

Bryce reimagined the season schedule for MLSSoccer.com. This project was done almost entirely in CSS and is an entirely new way to browse the match schedule. Users can see the entire year in a traditional wall calendar month-by-month view, with matches displayed as a pair of colors to represent each team. Clicking on a day brings up a close-up view of that match week. Then by clicking a club logo on the left, the schedule will quickly display that team’s fixtures using opponent logos and indicating home matches.

Ooyala Hero Videos by @LouisAJimenez

Louis’s hack day project was designed to integrate video into the hero section on the homepage. It allows editors to select video nodes and place them into the nodequeue, just like normal. Instead of rendering the thumbnail, it will render the embed code with the video. It supports various hero layouts (1, 3, and 4 articles or videos) and allows the embed to play in all spots. Louis was able to integrate the display area with our video player in order to hide the title overlay once the video starts playing to ensure an unimpeded video experience.

Design and templates by @chrisbettin

Chris decided to focus on hacking out design improvements to our Drupal 6 CMS. He built three major new features: Full screen video player, video integration into article content types, and mobile web optimizations. Check out the before and after GIFs below.

The team had some fun and ‘horsed around’ for a bit.

 photo

MLS MatchDay reached a great milestone earlier this year. For the first time ever, we have sent over 100 million push notifications in a single season–and the season is not even over yet! It is really exciting to see the growing number of fans receiving alerts from our apps. I’d like share two key metrics.

First, an aggregate view of total content alerts sent over the last 2 years. The most obvious observation is a dramatic change in the growth rate. We made an effort to make them discoverable and allow fans to customize the alerts in our apps this year. While we saw strong growth in our app install base, the growth on alerts was significantly higher.

Second, the graph below shows monthly alert rates for 2012 and 2013. This year-over-year comparisons offers two interesting facts. You can see the monthly send rate is 600% higher in 2013, and continues to trend upward. Additionally, our monthly rate grew throughout 2013. We saw a slight plateau at the end of the year in 2012.

Be sure to check out my previous post on how MLS does mobile notifications. We are working on some great 2014 updates for our native apps, and there will be even more ways to receive push notifications next season.

 photo

A developer is only as good as his tools and at MLS Digital we are constantly looking for better ones to improve our products and workflow. We thought it might be helpful to put together a list of some of the more general tools we like and to see what the rest of the development community is using. So in no particular order, here are some of the tools we use day in and day out:

  • Trello - Fog Creek’s lightweight project management tool allows us to stay organized without getting too bogged down in workflow. Trello’s draggable cards and intuitive design helps us keep track of a myriad of issues.

  • DataDog - Datadog’s versatile monitoring service helps us keep track of our servers and metrics. Detailed graphs, alerts, and a wide range of third-party integrations makes Datadog one of our most utilized tools. We have also been contributing checks back to the Datadog agent, including a new check for Couchbase metrics and MySQL replication.

  • Github - Version control makes any developer’s job easier and things are no different at MLS. Our team uses Github to help manage multiple repositories for our 19 clubs and league site. We share our public repos at github.com/majorleaguesoccer

  • HipChat - Although we love IRC, we found Atlassian’s chat/IM client perfect for communicating within the team and with non-developers. In addition to being reliable and mobile friendly, Hipchat has a full suite of emoticons. And we added a few custom ones including: poolparty party soccer thundercats nyannyannyancat

  • Hangouts - Google hangouts make our remote standup-meetings quick and painless. Plus the iPad app and Chrome plugin make it easy to hold meetings on short notice.

  • Evernote Business - Evernote for business brings team collaboration to an already fantasic product. Evernote’s accessibility and support for nearly every device makes it our go to tool for documentation, release notes, and project collaboration.

  • 1Password - 1Password is great for managing multiple passwords on desktop and mobile devices. The new 4.0 version allows for multiple vaults and by using Dropbox we found a secure way to store any shared passwords between team members.

  • Jenkins - We use Jenkins for continuous integration and production deployments. The MLS Jenkins setup is designed to kick off builds after Github pull requests, run automated tests and deployments, and then notify the team of build statuses via HipChat.

  • PagerDuty - The team uses Pagerduty to manage on-call schedules and distribute and escalate alerts to the appropriate team member.

  • Pingdom - Pingdom provides uptime and latency monitoring for our infrastructure, various sites, and endpoints. Pingdom integrates with PagerDuty and HipChat to send alerts.

  • iDoneThis - This simple performance management tool allows us to stay up to date with each project the team is working on by asking each team member to email small snippets of what they did each day. It is a great way to record accomplishments and to encourage cross project communication in an asynchronous fashion. One of the cooler features is that iDoneThis integrates with GitHub and now displays commit messages in each user’s daily log.

  • Google Analytics - Google Analytics is our primary web metric platform and provides key metrics for gauging the MLS network’s performance.

  • Chartbeat - Chartbeat helps us track real time traffic and get a better idea of how our users are interacting with the site. Paired with Google Analytics, it helps us build a detailed picture of our audience.

  • Minigroup - This group communication service helps our team communicate release notes and other updates to stakeholders. Minigroup provides email integration and custom groups to keep members informed.

  • Splunkstorm - Splunkstorm provides us with centralized server logging and allows us to analyze our logs for meaningful trends.

  • Flurry - Flurry’s extensive mobile data analytics help us track the performance of MLS MatchDay.

  • Geckoboard - Geckoboard offers one place to monitor multiple services ranging from Google Analytics to Twitter. At MLS Digital, we mainly use its dashboard for page view tracking and metrics.

  • New Relic - New Relic helps us keep an eye on application performance on our Drupal sites and also provides monitoring for our servers.

  • Dropbox - We use Dropbox to sync our 1Password vault files and share docs across the team.

See anything you’re a fan of? Have any ideas for a better tool we could be using? Tweet any recommendations to me @LouisAJimenez and stay tuned for a blog post on our full Web Stack in the near future.

Louis Jimenez - @LouisAJimenez