Saturday, September 3, 2011

Madison Transit API Homebrew

A post on my Madison Transit API homebrew is long overdue. It was an incredibly fun project and when you see how it has enabled others, you realize how much more compelling it is then the simple SMS app I had originally created.

The API has been out for a long time, but there was very little usage early on. I'm not sure what sparked the flame but in the last few months or so there has been a lot of interest and lot of activity from developers. It dawned on me that I've yet to write about it. And that's a shame because the API is actually the most clever piece of the whole SMSMyBus effort.

In the beginning
In the beginning, there was a very specific goal of creating an SMS app that delivered real-time arrival estimates. That was it. It was in response to my own need - as a commuter - for such an app. It was also going to be an entry into the Twilio developer contest that was run the week they released the SMS API publicly.

I knew going in that the Metro didn't provide a formal API or even an XML feed, but I'd done enough HTML screen scraping before to know how to overcome it. Little did I know how tedious that would become with the Metro data. So what started out as a simple texting application turned into a day of cursing the poorly formatted HTML I found on my screen. And every minute of the way I could be heard asking the same question, "Why doesn't the Metro have an API?!?"

When I finished, I swore that I would never let anyone else suffer the pain that I endured so decided to abstract the work I did for the texting app and provide access to my data so other developers could get to the transit data through a standard web service interface.

Why it's clever
The ugly work of screen scraping was now done and buried in the implementation of the SMS application. But by creating a couple of other web handlers, I could easily make the same scheduling data accessible via a web service call.

For example - http://www.smsmybus.com/api/v1/getarrivals?key=nomar&stopID=1101 - will return real-time arrival estimates for routes traveling through the stop at Main and Carrol on the square. That's the way every developer wants to query data from a web service.

With a little more scraping, I could also find the physical location of every stop (latitude and longitude) so developers could query the location of stops by a stop ID and could also query all nearby stops based on a lat/long coordinate. In the end, there was a pretty impressive looking transit API - simple to use and understand for new developers. And it didn't matter if it was ugly in its technique of screen scraping inside the implementation. It had a clean interface for the developers.

Supported API methods include...
getarrivals - Get real-time arrival estimates for any stop ID with the option of filtering by route ID.
getroutes - Get a list of every route in the Metro system.
getstops - Get the details for every stop a route travels through.
getnearbystops - Get the details for all stops within a given distance of a latitude/longitude.
getstoplocation - Get the location details for a stop given a sop ID.
getvehicles (not implemented) - Get details for a particular vehicle on the road.
getservicebulletins (not implemented) - Get a list of service bulletins in the Metro system.

Why I Built It
As I noted, a big reason for building the API was simply to save the next person the trouble of building up a new set of screen scraping tricks to get the data. But I also knew that most people that have attempted this have stopped at the schedule data. I don't know anything that has gone the extra step of getting the route and stop location data as well.More importantly, I wanted to find a way to contribute to the larger mission of opening up a public dataset to make it more accessible. Open data systems are important for lots of reasons, but most importantly, it allows communities to operate more efficiently. And nowhere is that more evident than in transit. Every city, state, and national government in the world should be on a path towards open data right now.

Unfortunately, Madison has been slow to find its course. My dream was that by creating the API and recruiting some more developers to use it, we'd have enough applications to take back to the city to say, "Look! If you open more data, developers will do great things!"

I also did it for the intellectual challenging of build an API. I've been an API consumer for a long time, but had never constructed one myself. It's an exercise I encourage any API consumer to go through. You'll have a whole new perspective for what it takes to build, document and support an API.

Sample Applications
In theory, all of the original interfaces - SMS, google chat, email, and phone - could have been re-implemented using this API, but I didn't go back and do that. But lots of new apps have been built. For example, the bus kiosk displays hanging in a few local businesses use the API to visualize arrival estimates for nearby stops.

Larry Walker used the API to build a Chumby app and also used it to display arrival estimates on a small LED display on an Arduino. He also built a mobile browser app to more easily access arrival estimates. And I used the API to build a Google gadget so you can get scheduling data in the sidebar of gmail. The attached gallery shows off some of these examples.

Go get started building your own Madison transit app! http://www.smsmybus.com/api/


60 comments:

  1. A great read Greg! I feel your pain for HTML scraping, but the product is fantastic! Did you send your work to Madison Metro? I feel like they should have hired you to implement the official API!

    ReplyDelete
  2. I would gladly hand this over to the Metro, but they don't seem to be interested.

    ReplyDelete
  3. Awesome work Greg! Not just on the technical front, but on the civic improvement front.

    ReplyDelete
  4. Great story Greg, thanks for sharing.I wanted to write up a couple stories on this, but would like to talk to you more first. Can you email me at info@apievangelist.com

    ReplyDelete
  5. what were you using to scrape the site with? Great post by the way.

    ReplyDelete
  6. @todd I use BeautifulSoup - http://www.crummy.com/software/BeautifulSoup/

    ReplyDelete
  7. Right here is the right web site for everyone who would like to find out about this topic. You know so much its almost tough to argue with you (not that I really will need to…HaHa). You certainly put a new spin on a topic which has been discussed for a long time. Excellent stuff, just wonderful!
    Cloud tech

    ReplyDelete
  8. it is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
    how to call malaysia from singapore

    ReplyDelete
  9. With so many books and articles appearing to usher in the field of making money online and further confusing the reader on the real way to make money.

    Business Analytics Course in Bangalore

    ReplyDelete
  10. I bookmarked your website because this site contains valuable information. I am very satisfied with the quality and the presentation of the articles. Thank you so much for saving great things. I am very grateful for this site.

    Data Analytics Course in Bangalore

    ReplyDelete
  11. Amazing Article,Really useful information to all So, I hope you will share more information to be check and share here.thanks for sharing .
    Website: Cycling tours package

    ReplyDelete
  12. I enjoyed over read your blog post. This was actually what i was looking for and i am glad to came here!
    Website: Antique jewellery designs

    ReplyDelete
  13. Post is very good its amzazing post I love them thanks for sharing.
    visit here- election hindi comedy

    ReplyDelete
  14. MyAssignmentHelpNow is famous for providing quality assignment help services at reasonable prices. We have been serving quality writing service on various subjects to reach out the maximum number of students in Australia by ensuring that our prices remain cheap and affordable. Our experts are quite skilled and knowledgeable to accomplish challenging tasks and we always strive to help you with best and cheap assignment writing service in Australia.
    Assignment Help Sydney

    ReplyDelete
  15. Very well written information. It will be supportive to everyone who employess it, including yours truly. Keep up the good work - can'r wait to read more posts.
    Visit here :- Best Seo Company In India

    ReplyDelete
  16. wonderful article contains lot of valuable information. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.
    This article resolved my all queries.good luck an best wishes to the team members.learn digital marketing use these following link
    Digital Marketing Course in Chennai

    ReplyDelete
  17. it is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
    Website: New year bumper lottery

    ReplyDelete
  18. I am new here. I like your post very much. It is very usefull post for me.
    website: Automotive Internet of Things Market

    ReplyDelete
  19. Excellent post.....Thanks for sharing...
    Best Location to Buy Real Estate in Pune.Some factor shapes many other major decisions such as: Lifestyle, Environment ,Weather, Education, Career, Social Networks.

    ReplyDelete
  20. Amazing Post....Keep it up...
    Nio Stars Technologies LLP is the best Grocery App Development Company in Pune. We deliver quality web solutions with creative designs for services.

    ReplyDelete
  21. It is really a good posting and i was searching for the same and have been satisfied after reading it,thanks for sharing it. how to play fantasy cricket

    ReplyDelete
  22. I am really enjoying reading your well written articles. I think you spend numerous effort and time updating your blog.
    online electronics shopping sites in india

    ReplyDelete
  23. I always like to read a quality content having accurate information regarding the subject and the same thing I found in this post.

    Website : Website Development Company |

    ReplyDelete
  24. Good day! This is kind of off topic but I need some help from an established blog. Is it very hard to set up your own blog? I m not very technical but I can figure things out pretty quick. I'm thinking about setting up my own but I'm not sure where to begin. Do you have any tips or suggestions? Thanks .
    Craigslist Posting Service for Car Dealers |

    ReplyDelete
  25. This is really amazing website that I have been found on google regarding website Blog Commenting sites. and I would like to thank admin who also given us to post the link on his side.
    Website : Lubbock moving company |

    ReplyDelete
  26. This was something I was looking for, really helpful, and great work is done. Thank you so much for sharing such valuable information.
    Website : Car Auction Software |

    ReplyDelete
  27. It’s really a cool and helpful piece of information. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing.
    Best CRM for Small Businesses |

    ReplyDelete
  28. I am really like it very much for the interesting info in this blog that to this website is providing the wonderful info in this blog that to utilize the great technology in this blog.
    Web Development Company in Gwalior |

    ReplyDelete
  29. Very nice, thanks a lot for everything you wrote here. you gave the right info about it. you sharing such a great blogs to us. sms marketing usa

    ReplyDelete
  30. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.


    เว็บแทงบอล
    ufabet
    ufa
    พวงหรีด
    โควิด
    บาคาร่า

    ReplyDelete
  31. This article is increasing the interest to learn more about this topic. Continue the sharing your new updates, regularly for my future.


    คาสิโนออนไลน์
    ufabet
    ufa
    เว็บบอล
    relx

    ReplyDelete
  32. Movie-watching websites that are more than movie-watching websites Because we are the number 1 free movie site in Thailand for a long time, including new movies, Thai movies, Western movies, Asian movies, we have all kinds of ways for you Including new series Full of all stories without interstitial ads to keep annoying anymore. One place sa-movie.com.

    Android and IOS operating systems. Watch online movies, Thai movies, Western movies, Asian movies, Cartoon movies, Netflix Movie, Action Movies, Comedy Movies, Crime Movies, Drama Movies, Horror Movies, Adventure Movies, Crash Movies and still have many new movies to watch. You can watch for free anytime, anywhere 24 hours a day at see4k.com.


    GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, all titles, anywhere, anytime, on mobile, tablet, computer. Android and IOS operating systems. Read top comics, action dramas, comedy, adventure, horror and manga. New coming every day to watch many more. Can be read for free anytime anywhere 24 hours a day at gangmanga.com..

    It is no secret that football is among the most popular and widely watched sports. Everybody who likes football tries to find the best platform for free soccer streaming. So, what are the best free sports streaming sites? We are going to answer this question. On this page, you can find a detailed overview of the most widespread soccer streaming websites. Keep on reading and make the best choice for you live24th.me.

    ReplyDelete
  33. Great article! This is the type of information that are meant to
    be shared across the internet. Thank you for sharing such a useful post. Very Interesting Post! I regularly follow this kind of Blog.
    read more

    ReplyDelete
  34. I really enjoyed reading your blog. It was very well written and easy to understand. Unlike other blogs that I have read which are actually not very good. Thank you so much!

    Data Science Training in Bangalore

    ReplyDelete
  35. Really wonderful blog completely enjoyed reading and learning to gain vast knowledge. Eventually, this blog helps in developing certain skills which in turn helpful in implementing those skills. Thanking the blogger for delivering such beautiful content and keep posting the contents in the upcoming days.

    Digital Marketing Training in Bangalore

    ReplyDelete
  36. I wanted to leave a little comment to support you and wish you the best of luck. We wish you the best of luck in all of your blogging endeavors.

    Artificial Intelligence Training in Bangalore

    ReplyDelete
  37. Truly incredible blog found to be very impressive due to which the learners who go through it will try to explore themselves with the content to develop the skills to an extreme level. Eventually, thanking the blogger to come up with such phenomenal content. Hope you arrive with similar content in the future as well.

    Machine Learning Course in Bangalore

    ReplyDelete
  38. one fast brownIn case you are looking for a good site, UFA, UEFA Bet casino site. Which can be played as a thorough answer, in a position to answer Quality and Performance It's ideal to get a great deal of the issues. It can be something very punching and fascinating. Excellently, the items that UFABET football betting is absolutely nothing even with the practical experience of quality. Plus accessibility that are able to see final results It's a model that is very sharp and also different. Full of performance of creating wealth Attractiveness With the most beneficial opportunities it is quite nothing You will find opportunities and also probabilities for making profits. Quality and also somewhat diverse For people who have come to make use of the service excellently fox
    บาคาร่า
    สล็อต
    ufa
    แทงบอล

    ReplyDelete
  39. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
    Data Science Course Syllabus

    ReplyDelete
  40. We are the one of the certified and top-rated Riverside Chevrolet Dealership Peoria Il.
    Visit here - Riverside Chevrolet Dealership

    ReplyDelete
  41. I wanted to leave a little comment to support you and wish you the best of luck. We wish you the best of luck in all of your blogging endeavors.

    Data Science Training in Bangalore

    ReplyDelete
  42. I am more curious to take an interest in some of them. I hope you will provide more information on these topics in your next articles.

    Machine Learning Course in Bangalore

    ReplyDelete
  43. Really impressed! Everything is a very open and very clear clarification of the issues. It contains true facts. Your website is very valuable. Thanks for sharing.

    Digital Marketing Training in Bangalore

    ReplyDelete
  44. A good blog always contains new and exciting information, and reading it I feel like this blog really has all of these qualities that make it a blog.

    Artificial Intelligence Training in Bangalore

    ReplyDelete

  45. Nice useful post. I have come here for the first time. I found a lot of interesting stuff in your blog, especially its discussion. Really, it's a very Nice article. Keep it up.Applying for an e visa Turkey is fast and straightforward. Eligible citizens are required to complete a simple online application form.

    ReplyDelete
  46. After reading this I thought it was extremely and very informative. I appreciate you taking the time and energy to put this information together. Eligible citizens can apply for a Azerbaijan electronic visa through a simple online application form to obtain an approved e-Visa electronically linked to their passport.

    ReplyDelete
  47. Thanks for your thoughts. Travelers can apply online for their kenya transit visa. kenya e-visa can get online via kenya e-visa.

    ReplyDelete
  48. Thanks for posting the best information and the blog is very good.Cloud Computing course in Bangalore

    ReplyDelete
  49. A good blog always contains new and exciting information, and reading it I feel like this blog really has all of these qualities that make it a blog.

    IoT Training Institute in Bangalore

    ReplyDelete