Catspaw's Guide to the Inevitably Insane

It's a lovely summer day on the intartubes, when young Timmy decided that he wanted to change his website's look and feel to see if he could get more business. Timmy hired a web designer to change the background color, move some of the elements around, design a delicious banner, and soon Timmy had quite a lovely website. All of a sudden, Timmy's sales started to drop. Oh noes! What happened? Did people not like his banner? Could they not find something on the page that they needed to find? What went wrong?!

What should Timmy do?

Naive approach: Timmy could just roll back the changes (you did save a copy, right, Timmy?) and go back to the website the way that it was. Presumably his sales would go back up. But he'd lose the beautiful new website that he paid good money for someone else to design.

Trial and error approach: Timmy could try removing the new changes, one by one, until he starts to make money again. That would let him find the culprit.

Science approach: If Timmy were a true scientist, he would have thought of doing an A/B experiment. He could have shown half of the visitors to his website (the A group) one of the visual changes and shown the other half of his vistitors (the B group) his original website without any changes. Then he could have seen how they performed differently. If both groups decreased in sales, he'd know that it wasn't his layout's fault but maybe people just aren't as interested in buying Timmy's Snowshoes in June.

It turns out that Google does this all the time. Believe it or not, you're helping us to test a new feature right now. The background color of ads on the top of Google's search results page used to be blue and now it's yellow. We showed some people the blue ads and some people the yellow ads, and watched how they behaved in aggregate. The yellow group won.

One of our VPs, Marissa Mayer, talked about our A/B testing at the Google I/O conference last week, and showed examples of some of the things that my team is working on right now.

For example, in the picture below, we varied the whitespace between the top of the page and the blue bar across the page. Look around where the logo is -- see how the spacing is slightly different above and below the logo? (It's subtle!) It turns out that users perform more searches when they're shown the top configuration.

The impact of this subtle change is small, but when you remember how much traffic Google gets, even things that cause a 1%, 0.5% or even 0.1% change can make a big difference. And because of the amount of traffic we get, we can get very accurate results from our tests with very tight confidence intervals, which is needed in order to detect changes this small.

These Ads UI changes are just one part of Ads Quality at Google which was featured in today's New York Times. I haven't found a video of Marissa's talk on Google's A/B testing yet, but I'll post it once I do since it's a good illustration of how my job these days is as more about science and research than about just making things pretty.

Do you use Google Reader? If so, log in to your Google Reader account and type the Konami Code from nintendo days past:

Sweeeeeeeeeeeeet.

Next week I have an all-day event scheduled to participate in a vendor taste testing for the new cafe that I'm helping to theme. How do you do an all-day taste test? Wouldn't you get, you know, full? I guess you could admire food for its other characteristics like appearance, healthiness, stuff like that. But 8 hours of food evaluation seems like a long (and lovely) day.

I'm doing a rush trip to Toronto, leaving this afternoon, to attend a friend's wedding this weekend. I've hit this weird stage in flying where it no longer takes any real mental effort. For example, I'm leaving for the airport in about 40 mins and I have yet to pack, I'm not sure which terminal I'm leaving from, and I'm spending the time writing a blog entry. I long ago made a pack.txt file on my laptop that contains everything I need to remember to bring on trips, so I don't have to worry about forgetting anything. And since it's such a rush visit, I think I'll just bring my work backpack rather than any Real luggage.

Now that Dancing Matt has released his latest video...

...I wish I were going somewhere a little more awesome (not that you're not all awesome, Toronto readers!). Though it's cool that I've been to 17 of the places that he dances in during this video -- London, Kyrgyzstan, Austin, Tokyo, Florida, Chicago, Seoul, New York, Panama, Mexico City, San Francisco, Vancouver, Washington DC, Paris, Montreal, Los Angeles, and Seattle. How many have you been to?

I finally got around to installing some insanecats spam control, so hopefully the number of comments has gone waaaaaaaaaay down. I also went through the archives (no, not manually) and removed spammy comments, erring on the side of keeping things that could have gone either way.

If your comment gets marked as spam, I give no indication of it except for the fact that it doesn't show up. No doubt a few times a year, this will drive some of the regulars crazy because you'll submit your comment and nothing will happen. That's what you get for sounding like a spammer, spammer.

Also, every time I look at the insanecats code, I cry a little inside. I should really rewrite it one day...

One of my officemates was complaining about a wardrobe malfunction he was experiencing and I pulled out a sewing kit that I stole from the last hotel I was at. "Here", I said, as I grabbed a needle and thread and offered a solution.

"I thought you were all anti-wife and kids", he laughed, meaning that I participated in Women in Engineering events at Google.

"All feminists should know how to sew and cook", I responded with a grin, "or else we'd be dependent on males to do things for us. That's not very empowering."

To be honest, I stole the philosophy from Dr J's father, who, if I recall the story correctly, taught his son how to cook so that he wouldn't have to be dependent on a woman. Not exactly coming at it from the same angle, but it made sense to me.

Lack of knowledge is never power.

Yesterday I gathered in a room with three other engineers, three Google chefs, and a handful of people from the benefits team, and we spent all day eating.

Being part of the team that has to taste their way through a day is similar to being one of the Googlers who used to sample candidate masseuses' wares: you spend a whole lot of time just thinking "okay, this rocks".

Of course, we're not just looking for tastiness, or even just presentation. "Googleyness" is an attribute which can be present in food just as much as in the engineers we hire. Was it local? Organic? Sustainably farmed/fished? Were there enough vegetarian/vegan options? What about gluten-free? Would the chefs enjoy making this food?

One of my favourite parts of the day was following behind our Google executive chefs as they tasted food from each table and listening to the types of things that they looked for in their food. They'd point to some random ingredient in some random dish and say, "tell me about this." As the sous-chef would explain where the ingredient came from (down to the farm) and how it was prepared, the executive chef was able to infer a whole bunch of information about the cost and quality of the ingredient.

Eight hours later, with a very full belly, I've developed a new-found appreciation for all of the work and philosophy that goes into our food here at Google. Thanks to the chef team for inviting us along! More photos here...