Celebrating National Day of Civic Hacking

This weekend was National Day of Civic Hacking. An incredibly well organized, distributed and disconnected network of events across the country where people came together to build tools and policies to make their communities better.

I love this type of work. It's the inspiration behind SMSMyBus, the underlying API that powers 40+ transit applications, and our local civic hacking organization, HackingMadison.

Although we didn't get an event organized for Madison, I wanted to contribute something. I had a few hours each morning this weekend to mash my keyboard on a project so I picked something small. There has always been a lot of interest around the data in the transit API so I decided to expose some of it by visualizing the API requests in real-time.

The browser app displays a Google Map and drops red circles on the bus stop locations as API requests come in for those stops. Fun. Simple. Transparent.

http://api.smsmybus.com/map

Life should be a bowl of ice cream

A public service announcement for my peers... Enjoy your parents. Spend time with them like it's your last.

My dad is visiting this weekend and we're not doing anything important. We watch basketball. Drink beer. Compare career notes. Turn society's debates into math problems. Tour the city like tourists. Run errands. Sit in silence. And eat ice cream.

It's wonderful.

The truth is, these visits are numbered. His mobility is decreasing and his mental acumen is dulling. Enough to make everyone quite sad if we stopped to focus on it. If we stopped to stress on any of the messy, strained bits of families that weave into the fabric of life, we would miss opportunities like this weekend.

But we don't. And that is the secret sauce of the wonderful.

These moments have been unfortunately aided by tragic events. The loss of a sister and daughter and premature mortality that Parkinson's brings changes your lens.

That's the tragedy behind the tragedy. What if you didn't need to have your lens prescribed?

That's your challenge. Look passed the messy stuff. Scoop your dad ice cream. Sit back. And enjoy the simple part of life.

Goodbye Posterous. Hello Blogger.

I've been hosting my blog on Posterous since I started writing in 2008. Earlier in the year, they sold the business to Twitter with no guarantees that the service would be around long term.

I kept waiting for them to offer a tool to extract/port content and it hasn't arrived. So I decided to scratch an itch and do it myself. I ported my blog to Blogger this weekend and here's how I did it.

Problem scope

The blog content is the easy(ish) part, but if you want to preserve the links to the original content, it can be tricky. Depending on your search engine and social media traffic, those thinks can be valuable.

First, every blogging platform uses its own slug path structure. Second, if your blog is in a subdomain, you have to find a way to preserve that subdomain with the addresses on the new blogging platform. For example, the parking API post has two different URIs.

Posterous :

www.gregtracy.com/adding-parking-to-the-madison-api-homebrew

Blogger :

blog.gregtracy.com/2012/01/adding-parking-to-madison-api-homebrew.html

Migrating content

Content migration is a tedious process. But the current option for getting Posterous content to Blogger is well documented on the web.

1. Import Posterous content into Wordpress 
2. Export Wordpress to an XML file
3. Use wordpress2blogger to transform the XML file for Blogger import
4. Import XML file into Blogger

This flow is documented in detail here. Lifehacker has also documented a process for migrating content using the auto-post feature inside Posterous, but it is tedious.

It doesn't necessarily end there, however. There are two glaring problems that make this process linger on forever. First, the content is poorly formatted so you'll find yourself editing posts in Blogger to clean up formatting. Second, multimedia sometimes doesn't import correctly or references files on Posterous or Wordpress so you need to upload that content to Blogger.

Someone will eventually build a migrator by mashing up the Posterous API with the Blogger API to make this all go away. In a different life, I might have taken the time to do it. But until someone does, the content migration is a little messy.

Building the redirector

This was the piece I was most interested in because it's simple and fun to build. 

1. Use the Posterous API to get a list of all post URI slugs and their post dates
2. Construct a slug map for Posterous URIs to Blogger URIs utilizing the dates and slugs
3. Build an app that takes any Posterous URI slug and redirects the page to the Blogger URI
4. Setup DNS so your old blog points to your app

Posterous listing

I wrote the following nodejs app to grab the full list of public posts from Posterous.

Blogger listing

If you can get access to the Blogger API, which requires a special request to the Blogger team, you can do the same thing I did with Posterous. The challenge is finding the ways Blogger changes slugs. For example, they remove "the" and "a" from most post slugs. 

Finding these differences is tedious so I actually recommend making sure you get the mapping correct for your top ten blog posts and then just crowd source the rest. Keep track of the page misses with logging (see below for my solution) and then update the map as you find mistakes. 

In the end, the map looks something like this:

{ 'awesome-post-title-wins : { date : '/2012/11/' , blogger_slug : 'awesome-post-title' }

With those details, you can get from any legacy slug - /awesome-post-title-wins - to any Blogger URI - /2012/11/awesome-post-title

Redirector application

I chose to use Google App Engine to host my redirector, but you could use anything. The logic is very simple. Just lookup the inbound request in your slug map and redirect to the Blogger URI. It looks a lot like the following.

My full implementation, including the slug map can be found on github. My implementation also includes a miss tracker which is what I've used for understanding which redirects are failing. I'm persisting the routes and a counter for the number of occurrences to understand what's missing. I can find all of the misses using the datastore viewer in the App Engine dashboard.

The incidentals

There are a couple of miscellaneous resources that you'll need to add to your map file as well. 

• RSS feed
• favicon.ico
• robots.txt
• Apple touch icons
• Posterous "tags" (Blogger "labels")

All of those can be redirected, but creating static files for your favicon and apple touch icons allow you to personalize that content better on your new domain.

Domain mapping

Now that you have an application that redirects content to Blogger, you need to map that application to your old domain. In my case, I updated the DNS for www.gregtracy.com to point to my app. The combination of App Engine and Google Apps makes this very easy.

Inside your App Engine dashboard, select "Application Settings" and use "Add Domain" to map the application to your subdomain. These steps walk you all the way through DNS setup.

That's it. What details did I miss?  http://blog.gregtracy.com

Cancer stole one of the good ones today

Around 1,500 people die from cancer every day.
 
But today that number feels different. Today, my own sister became a statistic.

Marla Tracy Pak was diagnosed with glioblastoma multiforme last August. Fought an ugly and losing battle for 10 months, and left us today, leaving a big void in the TracyPak Tribe.

I was asked to do a PechKucha talk the day I got the news last August. I called an audible and instead of talking about civic hacking, I talked about human qualities that make families, tribes and communities better. The qualities that my sister had.

I neglected to share that presentation last year, but here are the highlights...

1. Be kind to every person you interact with and talk to strangers like you’ve met them before.


2. Have empathy.


3. Support the things your friends are passionate about regardless of your opinion.


4. Celebrate nostalgia within your tribe.


5. Spoil your nieces, nephews and neighbors’ kids.


6. Converse with others with infinite optimism in your voice.


7. Send surprise packages to people.


8. Call an out of town friend or sibling and setup a long-weekend visit just because you can.


9. Make a choice to relate to people rather than drawing contrasts.

Success in your personal relationships defines who you are. We should all be a little bit more like Marla.

Spam blockers and colors

The Astronomy Picture of the Day email service I created several years ago has been getting pummeled by spam critters. I was getting up to five signups a day with bogus email addresses. Most of them came from Hotmail, AOL and Yahoo. With the new pricing structure of Google App Engine, it was beginning to cost me in the pocketbook so I had to address it.

The solution I came up with was part spam filter and part fundraiser. I decided to draw more attention to the PayPal donate button by requiring new signups to type the color of the button into one of the form fields. A couple of interesting things happened.

First, the rate of donations didn't change. I've had one modest donation in two weeks. 

Second, the spam ended. Overnight. It was fantastic.

Most interestingly, the little test showed fun insight into how we all see colors. When I originally wrote the test, the answer was "yellow". At the very last minute I decided that I should throw in "orange" as a possible answer as well and it was a good thing I did. Since deploying that change 15 days ago, there have been 51 signups - 26 orange and 25 yellow. I don't know why, but his fascinates me. It seems obvious to me that it is yellow. :)

What color do you see?

Adding parking to the Madison API homebrew

It started with this tweet.

... and then I found myself on the couch during New Year's Eve with a really bad movie. Laptop out...  in the wee hours of the New Year, I released a new web service for Madison. It provides a simple JSON wrapper around Madison's recently released data for parking availability on the isthmus. I added it to the list of services available in the SMSMyBus API.

http://smsmybus.com/api/v1/getparking
I was greeted in the morning by a lot of great feedback so I hooked it up to the SMSMyBus interfaces - SMS, xmpp and email - to create an app for it. For example, you can find the percentage of open spaces in downtown lots by texting 'parking' to the service...

... but what I really hope for, is that others take this and build something creative with it.

Although it's great to get so much positive feedback from the Madison community on these little projects, it's really (really) important to recognize that the work I've done with the API is going in the wrong direction in many ways. I'd rather be building the apps on top of services created by Madison. Eventually, this great city of ours needs to stop building websites and start producing the core feeds that I've done. That's the foundation. That is the government as a platform.

It's been two months since I canceled my data plan

Nearly two months ago, I wrote that I had canceled the data plan on my mobile phone. I'm reporting back that it has been a smashing success. I love being connected... to the people around me rather than the interwebs.

If you're disciplined about keeping the phone in your pocket, good for you. If you're not, this might be an option for you.

The experiment was given an extra boost when two weeks into the experiment I promptly lost my non-smart phone. Brilliant move in retrospect. I avoided the nervous, fidgety grabbing of the phone to look to see if a new message arrived. One might say that I went cold turkey, and it likely made the whole transition easier. So easy, in fact, I often questioned whether I needed any kind of phone at all!

That's not to say that the data plan isn't missed. My expectations were pretty spot on. I miss Google Maps and posting photos for the family. Here are the few notes I've jotted down along the way...

The pot

Let's get it right out there... I miss having reading material when I'm on the pot. If you have a smart phone, you know what I'm talking about.

My calendar

I miss access to my calendar... I haven't missed big meetings, but I have missed those non-vital but still important events that aren't necessarily on my radar every day. I've looked into the Google API to get access via SMS and will try to implement this one.

Twitter

I'd like to find a better desktop tool for Twitter. Now that I see less of it throughout the day, it would be nice to find a tool that helps me catchup on some feeds I don't want to miss.

Email

Email becomes less important - which is good. Everyone talks about the email tax where you can't control inbound email. I've found that by sending less and reading less frequently, I've been able to lower the burden of managing email. It now comes in well controlled bursts - those blocks of time that I dedicate to my inbox. I've also become ruthless with unsubscribe options.

Better focus

I haven't quantified this, but it feels like I have better focus through less distraction. If you're not pulling your phone out all the time to check in on your online life, you've improved your chances of focusing on a specific task whether it's work, a game with the kids or cleaning up around the house.

You all have become more annoying

I'm now way more annoyed by people who choose their phones over me. Whether it's to take a phone call, respond to a text message and check the sports scores, it's nothing short of annoying. I don't know how much I did this to other people two months ago, but I'm glad I don't do it anymore.