Life should be a bowl of ice cream

A public service announcement for my peers... Enjoy your parents. Spend time with them like it's your last.

My dad is visiting this weekend and we're not doing anything important. We watch basketball. Drink beer. Compare career notes. Turn society's debates into math problems. Tour the city like tourists. Run errands. Sit in silence. And eat ice cream.

It's wonderful.

The truth is, these visits are numbered. His mobility is decreasing and his mental acumen is dulling. Enough to make everyone quite sad if we stopped to focus on it. If we stopped to stress on any of the messy, strained bits of families that weave into the fabric of life, we would miss opportunities like this weekend.

But we don't. And that is the secret sauce of the wonderful.

These moments have been unfortunately aided by tragic events. The loss of a sister and daughter and premature mortality that Parkinson's brings changes your lens.

That's the tragedy behind the tragedy. What if you didn't need to have your lens prescribed?

That's your challenge. Look passed the messy stuff. Scoop your dad ice cream. Sit back. And enjoy the simple part of life.

Goodbye Posterous. Hello Blogger.

I've been hosting my blog on Posterous since I started writing in 2008. Earlier in the year, they sold the business to Twitter with no guarantees that the service would be around long term.

I kept waiting for them to offer a tool to extract/port content and it hasn't arrived. So I decided to scratch an itch and do it myself. I ported my blog to Blogger this weekend and here's how I did it.

Problem scope

The blog content is the easy(ish) part, but if you want to preserve the links to the original content, it can be tricky. Depending on your search engine and social media traffic, those thinks can be valuable.

First, every blogging platform uses its own slug path structure. Second, if your blog is in a subdomain, you have to find a way to preserve that subdomain with the addresses on the new blogging platform. For example, the parking API post has two different URIs.

Posterous :

www.gregtracy.com/adding-parking-to-the-madison-api-homebrew

Blogger :

blog.gregtracy.com/2012/01/adding-parking-to-madison-api-homebrew.html

Migrating content

Content migration is a tedious process. But the current option for getting Posterous content to Blogger is well documented on the web.

1. Import Posterous content into Wordpress 
2. Export Wordpress to an XML file
3. Use wordpress2blogger to transform the XML file for Blogger import
4. Import XML file into Blogger

This flow is documented in detail here. Lifehacker has also documented a process for migrating content using the auto-post feature inside Posterous, but it is tedious.

It doesn't necessarily end there, however. There are two glaring problems that make this process linger on forever. First, the content is poorly formatted so you'll find yourself editing posts in Blogger to clean up formatting. Second, multimedia sometimes doesn't import correctly or references files on Posterous or Wordpress so you need to upload that content to Blogger.

Someone will eventually build a migrator by mashing up the Posterous API with the Blogger API to make this all go away. In a different life, I might have taken the time to do it. But until someone does, the content migration is a little messy.

Building the redirector

This was the piece I was most interested in because it's simple and fun to build. 

1. Use the Posterous API to get a list of all post URI slugs and their post dates
2. Construct a slug map for Posterous URIs to Blogger URIs utilizing the dates and slugs
3. Build an app that takes any Posterous URI slug and redirects the page to the Blogger URI
4. Setup DNS so your old blog points to your app

Posterous listing

I wrote the following nodejs app to grab the full list of public posts from Posterous.

Blogger listing

If you can get access to the Blogger API, which requires a special request to the Blogger team, you can do the same thing I did with Posterous. The challenge is finding the ways Blogger changes slugs. For example, they remove "the" and "a" from most post slugs. 

Finding these differences is tedious so I actually recommend making sure you get the mapping correct for your top ten blog posts and then just crowd source the rest. Keep track of the page misses with logging (see below for my solution) and then update the map as you find mistakes. 

In the end, the map looks something like this:

{ 'awesome-post-title-wins : { date : '/2012/11/' , blogger_slug : 'awesome-post-title' }

With those details, you can get from any legacy slug - /awesome-post-title-wins - to any Blogger URI - /2012/11/awesome-post-title

Redirector application

I chose to use Google App Engine to host my redirector, but you could use anything. The logic is very simple. Just lookup the inbound request in your slug map and redirect to the Blogger URI. It looks a lot like the following.

My full implementation, including the slug map can be found on github. My implementation also includes a miss tracker which is what I've used for understanding which redirects are failing. I'm persisting the routes and a counter for the number of occurrences to understand what's missing. I can find all of the misses using the datastore viewer in the App Engine dashboard.

The incidentals

There are a couple of miscellaneous resources that you'll need to add to your map file as well. 

• RSS feed
• favicon.ico
• robots.txt
• Apple touch icons
• Posterous "tags" (Blogger "labels")

All of those can be redirected, but creating static files for your favicon and apple touch icons allow you to personalize that content better on your new domain.

Domain mapping

Now that you have an application that redirects content to Blogger, you need to map that application to your old domain. In my case, I updated the DNS for www.gregtracy.com to point to my app. The combination of App Engine and Google Apps makes this very easy.

Inside your App Engine dashboard, select "Application Settings" and use "Add Domain" to map the application to your subdomain. These steps walk you all the way through DNS setup.

That's it. What details did I miss?  http://blog.gregtracy.com

Cancer stole one of the good ones today

Around 1,500 people die from cancer every day.
 
But today that number feels different. Today, my own sister became a statistic.

Marla Tracy Pak was diagnosed with glioblastoma multiforme last August. Fought an ugly and losing battle for 10 months, and left us today, leaving a big void in the TracyPak Tribe.

I was asked to do a PechKucha talk the day I got the news last August. I called an audible and instead of talking about civic hacking, I talked about human qualities that make families, tribes and communities better. The qualities that my sister had.

I neglected to share that presentation last year, but here are the highlights...

1. Be kind to every person you interact with and talk to strangers like you’ve met them before.


2. Have empathy.


3. Support the things your friends are passionate about regardless of your opinion.


4. Celebrate nostalgia within your tribe.


5. Spoil your nieces, nephews and neighbors’ kids.


6. Converse with others with infinite optimism in your voice.


7. Send surprise packages to people.


8. Call an out of town friend or sibling and setup a long-weekend visit just because you can.


9. Make a choice to relate to people rather than drawing contrasts.

Success in your personal relationships defines who you are. We should all be a little bit more like Marla.

Spam blockers and colors

The Astronomy Picture of the Day email service I created several years ago has been getting pummeled by spam critters. I was getting up to five signups a day with bogus email addresses. Most of them came from Hotmail, AOL and Yahoo. With the new pricing structure of Google App Engine, it was beginning to cost me in the pocketbook so I had to address it.

The solution I came up with was part spam filter and part fundraiser. I decided to draw more attention to the PayPal donate button by requiring new signups to type the color of the button into one of the form fields. A couple of interesting things happened.

First, the rate of donations didn't change. I've had one modest donation in two weeks. 

Second, the spam ended. Overnight. It was fantastic.

Most interestingly, the little test showed fun insight into how we all see colors. When I originally wrote the test, the answer was "yellow". At the very last minute I decided that I should throw in "orange" as a possible answer as well and it was a good thing I did. Since deploying that change 15 days ago, there have been 51 signups - 26 orange and 25 yellow. I don't know why, but his fascinates me. It seems obvious to me that it is yellow. :)

What color do you see?

Adding parking to the Madison API homebrew

It started with this tweet.

... and then I found myself on the couch during New Year's Eve with a really bad movie. Laptop out...  in the wee hours of the New Year, I released a new web service for Madison. It provides a simple JSON wrapper around Madison's recently released data for parking availability on the isthmus. I added it to the list of services available in the SMSMyBus API.

http://smsmybus.com/api/v1/getparking
I was greeted in the morning by a lot of great feedback so I hooked it up to the SMSMyBus interfaces - SMS, xmpp and email - to create an app for it. For example, you can find the percentage of open spaces in downtown lots by texting 'parking' to the service...

... but what I really hope for, is that others take this and build something creative with it.

Although it's great to get so much positive feedback from the Madison community on these little projects, it's really (really) important to recognize that the work I've done with the API is going in the wrong direction in many ways. I'd rather be building the apps on top of services created by Madison. Eventually, this great city of ours needs to stop building websites and start producing the core feeds that I've done. That's the foundation. That is the government as a platform.

It's been two months since I canceled my data plan

Nearly two months ago, I wrote that I had canceled the data plan on my mobile phone. I'm reporting back that it has been a smashing success. I love being connected... to the people around me rather than the interwebs.

If you're disciplined about keeping the phone in your pocket, good for you. If you're not, this might be an option for you.

The experiment was given an extra boost when two weeks into the experiment I promptly lost my non-smart phone. Brilliant move in retrospect. I avoided the nervous, fidgety grabbing of the phone to look to see if a new message arrived. One might say that I went cold turkey, and it likely made the whole transition easier. So easy, in fact, I often questioned whether I needed any kind of phone at all!

That's not to say that the data plan isn't missed. My expectations were pretty spot on. I miss Google Maps and posting photos for the family. Here are the few notes I've jotted down along the way...

The pot

Let's get it right out there... I miss having reading material when I'm on the pot. If you have a smart phone, you know what I'm talking about.

My calendar

I miss access to my calendar... I haven't missed big meetings, but I have missed those non-vital but still important events that aren't necessarily on my radar every day. I've looked into the Google API to get access via SMS and will try to implement this one.

Twitter

I'd like to find a better desktop tool for Twitter. Now that I see less of it throughout the day, it would be nice to find a tool that helps me catchup on some feeds I don't want to miss.

Email

Email becomes less important - which is good. Everyone talks about the email tax where you can't control inbound email. I've found that by sending less and reading less frequently, I've been able to lower the burden of managing email. It now comes in well controlled bursts - those blocks of time that I dedicate to my inbox. I've also become ruthless with unsubscribe options.

Better focus

I haven't quantified this, but it feels like I have better focus through less distraction. If you're not pulling your phone out all the time to check in on your online life, you've improved your chances of focusing on a specific task whether it's work, a game with the kids or cleaning up around the house.

You all have become more annoying

I'm now way more annoyed by people who choose their phones over me. Whether it's to take a phone call, respond to a text message and check the sports scores, it's nothing short of annoying. I don't know how much I did this to other people two months ago, but I'm glad I don't do it anymore.

Revisiting Google App Engine's pricing changes

This post revisits my earlier evaluation of Google App Engine's post-preview pricing changes and how it affected my project, SMSMyBus. As I noted, the app was projected to cost between $6 and $7 dollars per day under the new platform pricing.

Since that post, I’ve been rolling out small, incremental changes to optimize the code and combat all of the known issues. I’m thrilled to report that I have the price down to $0/day. And I’m once again impressed by the snappy and reliable App Engine platform.

Looking back on the changes, I can say that I was doing some bad things, some abusive things, and App Engine was making some bad choices as well. But the end result proves that if its developers optimize and do smart things, they are rewarded. App Engine remains a great solution for my transit API service.

Here’s a history of the changes over the last two months that got me down to $0/day...

Platform Configuration

1. Instance allocation
The new pricing model charges applications based on their use of instances (hardware resources where your application is running) rather than CPU utilization. A key to keeping your instance cost down is to simply reduce the number of instances that are spinning. Duh. So I grabbed the instance slider in the application settings and yanked it to the left. This doesn't prevent scaling, it just limits my billing for normal traffic flow.

2. Delete data
App Engine data storage (for your database) costs $0.008/GByte-day. Doesn’t sound too expensive, but I had been storing every single API call I had ever gotten. I thought it would be useful for API developers and for analytics. My drive to $0 outweighed that, however, so I deleted all of the history data and got under the free quota for storage.

Application Configuration

3. Memcached the application's route listings
I was surprised to find that I wasn’t doing this already, but there it was. I have a data structure that maps bus routes and bus stops to scheduling data on the Metro website and it never changes. In some cases - like the static calls from the kiosk clients - I was looking up route listing details in the datastore once every minute!! Fail. I used memcache to keep the common queries in memory and avoid the extra datastore reads.

4. Limit access during off hours
One thing that never changes is when the Metro service is running. There are five+ hours a day where the buses aren’t on the street. But some clients are still asking for data. I stubbed out most of the API during these off hours before the code ever gets close to making a datastore or memcache call.

These four changes brought me down to $0.70 per day. Bam!

Algorithm Changes

5. Asynchronous screen grabs
If you don’t know, behind the API curtain is an ugly screen scraping task that extracts the arrival estimates from the Metro website. So when a client requests arrival data for a stop, the app goes off and requests multiple web pages, machine-reads the information and aggregates all of the results.

The original implementation of the SMS interface did this by creating multiple tasks (one for each route traveling through the respective stop). When a task ran, it stored the results in the datastore. An aggregator task would read those results out of the datastore and piece together the response to the caller.

When the API was created, I couldn’t use background tasks because I had to respond with results in the same HTTP context. That’s when I discovered the great feature, asynchronous url fetch. This essentially let me grab all of the different Metro web pages at the same time. But when I implemented this, I continued to use the datastore as the mechanism for storing and retrieving results. This was just lazy. Under the old pricing, I wasn’t incented to change it other then the fact that it was a bit slow.
Under the new pricing model, this solution was very expensive. The API is continuously running this aggregation algorithm - constantly writing and reading to the datastore for model instances that have a lifespan of under a minute!

I rolled out a change that removed the use of the datastore and instead sorted the aggregated results in memory. This had a dramatic effect on my API quota for datastore reads and writes as well as overall performance and latency for my users. Especially the write operations, where you get penalized by an order of magnitude for this type of behavior because index updates work against your API quota as well.

6. Dogfood
After optimizing the API, I realized that the original SMSMyBus apps (SMS, chat, email and phone interfaces for the Metro) were now the long pole. Those apps were implemented before the API existed so they weren’t benefiting from the API optimizations. Solution... re-implement to use the SMSMyBus API.

It should have been done long ago simply as a validation exercise of the API methods. Credit to the eligence and simplicity of the API - this port was simple and only took a couple of hours.

These two changes brought me down to $0.10/day. Badda-bing.

AppStats

7. Run Appstats on all application interfaces
The last stop on the optimization train was Appstats. A truly great tool in the App Engine toolbox. In just a matter of minutes, you can find the hidden datastore operations that are dragging you down. In my case, it led me to one area that wasn’t being memcached at all. And it revealed an area that was simply using the memcache incorrectly! Love this tool...

This change brought me down $0.00/day. Winning.

Results
App Engine remains a great platform for developers that don’t abuse it and take the time to optimize their applications.

The SMSMyBus API now serves over 6,000 transit requests per day. It’s fast, reliable and flat out fun to use. I’m as proud as ever that I brought this to Madison.

Next step... find a way to fund my SMS users. :)