Saturday, September 3, 2011

Madison Transit API Homebrew

A post on my Madison Transit API homebrew is long overdue. It was an incredibly fun project and when you see how it has enabled others, you realize how much more compelling it is then the simple SMS app I had originally created.

The API has been out for a long time, but there was very little usage early on. I'm not sure what sparked the flame but in the last few months or so there has been a lot of interest and lot of activity from developers. It dawned on me that I've yet to write about it. And that's a shame because the API is actually the most clever piece of the whole SMSMyBus effort.

In the beginning
In the beginning, there was a very specific goal of creating an SMS app that delivered real-time arrival estimates. That was it. It was in response to my own need - as a commuter - for such an app. It was also going to be an entry into the Twilio developer contest that was run the week they released the SMS API publicly.

I knew going in that the Metro didn't provide a formal API or even an XML feed, but I'd done enough HTML screen scraping before to know how to overcome it. Little did I know how tedious that would become with the Metro data. So what started out as a simple texting application turned into a day of cursing the poorly formatted HTML I found on my screen. And every minute of the way I could be heard asking the same question, "Why doesn't the Metro have an API?!?"

When I finished, I swore that I would never let anyone else suffer the pain that I endured so decided to abstract the work I did for the texting app and provide access to my data so other developers could get to the transit data through a standard web service interface.

Why it's clever
The ugly work of screen scraping was now done and buried in the implementation of the SMS application. But by creating a couple of other web handlers, I could easily make the same scheduling data accessible via a web service call.

For example - http://www.smsmybus.com/api/v1/getarrivals?key=nomar&stopID=1101 - will return real-time arrival estimates for routes traveling through the stop at Main and Carrol on the square. That's the way every developer wants to query data from a web service.

With a little more scraping, I could also find the physical location of every stop (latitude and longitude) so developers could query the location of stops by a stop ID and could also query all nearby stops based on a lat/long coordinate. In the end, there was a pretty impressive looking transit API - simple to use and understand for new developers. And it didn't matter if it was ugly in its technique of screen scraping inside the implementation. It had a clean interface for the developers.

Supported API methods include...
getarrivals - Get real-time arrival estimates for any stop ID with the option of filtering by route ID.
getroutes - Get a list of every route in the Metro system.
getstops - Get the details for every stop a route travels through.
getnearbystops - Get the details for all stops within a given distance of a latitude/longitude.
getstoplocation - Get the location details for a stop given a sop ID.
getvehicles (not implemented) - Get details for a particular vehicle on the road.
getservicebulletins (not implemented) - Get a list of service bulletins in the Metro system.

Why I Built It
As I noted, a big reason for building the API was simply to save the next person the trouble of building up a new set of screen scraping tricks to get the data. But I also knew that most people that have attempted this have stopped at the schedule data. I don't know anything that has gone the extra step of getting the route and stop location data as well.More importantly, I wanted to find a way to contribute to the larger mission of opening up a public dataset to make it more accessible. Open data systems are important for lots of reasons, but most importantly, it allows communities to operate more efficiently. And nowhere is that more evident than in transit. Every city, state, and national government in the world should be on a path towards open data right now.

Unfortunately, Madison has been slow to find its course. My dream was that by creating the API and recruiting some more developers to use it, we'd have enough applications to take back to the city to say, "Look! If you open more data, developers will do great things!"

I also did it for the intellectual challenging of build an API. I've been an API consumer for a long time, but had never constructed one myself. It's an exercise I encourage any API consumer to go through. You'll have a whole new perspective for what it takes to build, document and support an API.

Sample Applications
In theory, all of the original interfaces - SMS, google chat, email, and phone - could have been re-implemented using this API, but I didn't go back and do that. But lots of new apps have been built. For example, the bus kiosk displays hanging in a few local businesses use the API to visualize arrival estimates for nearby stops.

Larry Walker used the API to build a Chumby app and also used it to display arrival estimates on a small LED display on an Arduino. He also built a mobile browser app to more easily access arrival estimates. And I used the API to build a Google gadget so you can get scheduling data in the sidebar of gmail. The attached gallery shows off some of these examples.

Go get started building your own Madison transit app! http://www.smsmybus.com/api/


36 comments:

  1. A great read Greg! I feel your pain for HTML scraping, but the product is fantastic! Did you send your work to Madison Metro? I feel like they should have hired you to implement the official API!

    ReplyDelete
  2. I would gladly hand this over to the Metro, but they don't seem to be interested.

    ReplyDelete
  3. Awesome work Greg! Not just on the technical front, but on the civic improvement front.

    ReplyDelete
  4. Great story Greg, thanks for sharing.I wanted to write up a couple stories on this, but would like to talk to you more first. Can you email me at info@apievangelist.com

    ReplyDelete
  5. what were you using to scrape the site with? Great post by the way.

    ReplyDelete
  6. @todd I use BeautifulSoup - http://www.crummy.com/software/BeautifulSoup/

    ReplyDelete
  7. Right here is the right web site for everyone who would like to find out about this topic. You know so much its almost tough to argue with you (not that I really will need to…HaHa). You certainly put a new spin on a topic which has been discussed for a long time. Excellent stuff, just wonderful!
    Cloud tech

    ReplyDelete
  8. it is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
    how to call malaysia from singapore

    ReplyDelete
  9. With so many books and articles appearing to usher in the field of making money online and further confusing the reader on the real way to make money.

    Business Analytics Course in Bangalore

    ReplyDelete
  10. I bookmarked your website because this site contains valuable information. I am very satisfied with the quality and the presentation of the articles. Thank you so much for saving great things. I am very grateful for this site.

    Data Analytics Course in Bangalore

    ReplyDelete
  11. Amazing Article,Really useful information to all So, I hope you will share more information to be check and share here.thanks for sharing .
    Website: Cycling tours package

    ReplyDelete
  12. I enjoyed over read your blog post. This was actually what i was looking for and i am glad to came here!
    Website: Antique jewellery designs

    ReplyDelete
  13. Post is very good its amzazing post I love them thanks for sharing.
    visit here- election hindi comedy

    ReplyDelete
  14. MyAssignmentHelpNow is famous for providing quality assignment help services at reasonable prices. We have been serving quality writing service on various subjects to reach out the maximum number of students in Australia by ensuring that our prices remain cheap and affordable. Our experts are quite skilled and knowledgeable to accomplish challenging tasks and we always strive to help you with best and cheap assignment writing service in Australia.
    Assignment Help Sydney

    ReplyDelete
  15. Very well written information. It will be supportive to everyone who employess it, including yours truly. Keep up the good work - can'r wait to read more posts.
    Visit here :- Best Seo Company In India

    ReplyDelete
  16. wonderful article contains lot of valuable information. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.
    This article resolved my all queries.good luck an best wishes to the team members.learn digital marketing use these following link
    Digital Marketing Course in Chennai

    ReplyDelete
  17. it is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
    Website: New year bumper lottery

    ReplyDelete
  18. I am new here. I like your post very much. It is very usefull post for me.
    website: Automotive Internet of Things Market

    ReplyDelete
  19. Excellent post.....Thanks for sharing...
    Best Location to Buy Real Estate in Pune.Some factor shapes many other major decisions such as: Lifestyle, Environment ,Weather, Education, Career, Social Networks.

    ReplyDelete
  20. Amazing Post....Keep it up...
    Nio Stars Technologies LLP is the best Grocery App Development Company in Pune. We deliver quality web solutions with creative designs for services.

    ReplyDelete
  21. It is really a good posting and i was searching for the same and have been satisfied after reading it,thanks for sharing it. how to play fantasy cricket

    ReplyDelete
  22. I am really enjoying reading your well written articles. I think you spend numerous effort and time updating your blog.
    online electronics shopping sites in india

    ReplyDelete
  23. I always like to read a quality content having accurate information regarding the subject and the same thing I found in this post.

    Website : Website Development Company |

    ReplyDelete
  24. Good day! This is kind of off topic but I need some help from an established blog. Is it very hard to set up your own blog? I m not very technical but I can figure things out pretty quick. I'm thinking about setting up my own but I'm not sure where to begin. Do you have any tips or suggestions? Thanks .
    Craigslist Posting Service for Car Dealers |

    ReplyDelete
  25. This is really amazing website that I have been found on google regarding website Blog Commenting sites. and I would like to thank admin who also given us to post the link on his side.
    Website : Lubbock moving company |

    ReplyDelete
  26. This was something I was looking for, really helpful, and great work is done. Thank you so much for sharing such valuable information.
    Website : Car Auction Software |

    ReplyDelete
  27. It’s really a cool and helpful piece of information. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing.
    Best CRM for Small Businesses |

    ReplyDelete
  28. I am really like it very much for the interesting info in this blog that to this website is providing the wonderful info in this blog that to utilize the great technology in this blog.
    Web Development Company in Gwalior |

    ReplyDelete
  29. Very nice, thanks a lot for everything you wrote here. you gave the right info about it. you sharing such a great blogs to us. sms marketing usa

    ReplyDelete
  30. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.


    เว็บแทงบอล
    ufabet
    ufa
    พวงหรีด
    โควิด
    บาคาร่า

    ReplyDelete
  31. This article is increasing the interest to learn more about this topic. Continue the sharing your new updates, regularly for my future.


    คาสิโนออนไลน์
    ufabet
    ufa
    เว็บบอล
    relx

    ReplyDelete
  32. Movie-watching websites that are more than movie-watching websites Because we are the number 1 free movie site in Thailand for a long time, including new movies, Thai movies, Western movies, Asian movies, we have all kinds of ways for you Including new series Full of all stories without interstitial ads to keep annoying anymore. One place sa-movie.com.

    Android and IOS operating systems. Watch online movies, Thai movies, Western movies, Asian movies, Cartoon movies, Netflix Movie, Action Movies, Comedy Movies, Crime Movies, Drama Movies, Horror Movies, Adventure Movies, Crash Movies and still have many new movies to watch. You can watch for free anytime, anywhere 24 hours a day at see4k.com.


    GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, all titles, anywhere, anytime, on mobile, tablet, computer. Android and IOS operating systems. Read top comics, action dramas, comedy, adventure, horror and manga. New coming every day to watch many more. Can be read for free anytime anywhere 24 hours a day at gangmanga.com..

    It is no secret that football is among the most popular and widely watched sports. Everybody who likes football tries to find the best platform for free soccer streaming. So, what are the best free sports streaming sites? We are going to answer this question. On this page, you can find a detailed overview of the most widespread soccer streaming websites. Keep on reading and make the best choice for you live24th.me.

    ReplyDelete