Checksum Arcanius

Today marks the completion of my 4th year of employment at Facebook. From eager anticipation and the first week, though my graduation from bootcamp and my first faceversary, to two years of management experience and now a chance to go back to engineering, it has truly been a roller-coaster ride in every sense of the metaphor — fears and thrills, ups and downs, highs and lows.

I celebrated the day by helping my Ultimate Frisbee team, the Feral Cows, break seed, even if only for a day. I think tomorrow will  be a somewhat less restrained celebration.

Though I don’t know what the future holds, for now I’m sticking with Facebook. To many more highs and not too many lows!

Today marks one full year that I’ve been working at Facebook. It has been the most consuming, thrilling, and educational work experience I could imagine.

Looking back at what I’ve already done — from hacking on MySQL to presenting at a conference to a packed house; from taking ownership of core modules to helping decide the direction of Facebook’s infrastructure; from pushing code to 300 then 400 and now over 500 million active users to debugging site issues at an unprecedented scale… every experience has helped me grow as a person and an engineer.

Facebook surrounded me with some of the brightest people in the world to work on some of the most interesting technical challenges anywhere, while providing the support to make it possible and the autonomy to make it my own. I could not have asked for a better job.

Here’s looking forward to what I’ll be able to accomplish in my next year.

Facebook’s announcements at f8 earlier today have made socializing any website trivial — instantly. You don’t even need to know how to program. Just add an iframe — one line of html — and you can make your website have an instant social presence. I hacked in the widget just now on my site in a matter of minutes.

This is the one line I added to my blog’s template, in single.php:

<iframe src="http://www.facebook.com/widgets/like.php?show_faces=1&amp;href=<?php
the_permalink() ?>" frameborder="0" scrolling="no" height="61"></iframe>

My first day at my first MySQL conference was a riotous success. I attended the “State of the Dolphin” keynote followed by talks given by Tim O’Reilly and Facebook’s own Mark Callaghan, who also won a MySQL Community Member of the Year Award during the opening talks. Congrats to Mark!

After the Keynotes, I synced up with other Facebookers at our expo hall booth, and then I went to Domas Mituzas’ talk on “High Concurrency MySQL”. The ballroom couldn’t hold all the people who wanted to watch — there was actually a line outside the door of people listening in on his talk! Although I wouldn’t suggest Domas give up his day job to write slides full-time, he had a great presentation overall that kept the audience interested and engaged.

Next, I attended a presentation on Sqoop by my two-time TA at the UW and now Cloudera co-founder and presenter extraordinaire, Aaron Kimball. Sqoop is a SQL-to-Hadoop translation layer that automates many of the steps of shuttling data from OLTP stores to HDFS for analytics. It is open source and Aaron is it’s primary developer. You can check out the code on github, or use it as part of Cloudera’s Hadoop Distribution.

After lunch, I went to a presentation by Lars Thalmann on new MySQL replication features in 5.1 and 5.5. Lead replication developer Mats Kindall was also there to answer questions. It’s good to see that MySQL is making progress on replication, but it is still woefully limited in a number of ways: not crashproof, single-threaded, and difficulty in replicating to non-MySQL data stores are all weak points of MySQL’s replication system today. These are all on the roadmap, but from the answers to my questions, I got the impression that these ideas are still mostly bullet points on a slide rather than almost-features in MySQL.

Make no mistake, these features are hard to add — I’ve dabbled around in the area myself — and it took Mark a concerted effort to port rpl_transaction_enabled from our 5.0 patch to Facebook’s 5.1 patch. Still, I hope MySQL takes the rpl_transaction_enabled patch and  into 5.1 or 5.5 officially, because in any large deployment, it is incredibly useful to not manually intervene when a slave crashes.

After the replication talk, I went back to the expo hall to talk with people, then I hacked on MySQL in the afternoon. Could there possibly be a better venue for this? Two (small) diffs later, and I was back into the expo hall socializing/recruiting for Facebook. The night ended well with a trip to In-and-Out.

The 2010 O’Reilly MySQL Conference starts tomorrow in Santa Clara. Facebook’s Database Engineering team (which includes me!) will be there along with some of our Operations team and our one-man Performance team. Each team will be giving a talk at the conference:

On Tuesday, the Database Performance Team will be presenting on “High Concurrency MySQL.” Domas is an interesting, animated fellow, and I imagine that his talk will be quite entertaining as well as informative.

On Wednesday, the Database Operations Team will be speaking about Database Operations at Scale. Our DB Ops guys are some of the best in the business; they keep our database tiers, which are often under enormous pressure from growth and changing requirements, running remarkably well.

On Thursday, Mark Callaghan, Ryan Mack, and I will be presenting our talk on High-throughput MySQL (we claim that Domas stole our title rather than the other way around). Mark Callaghan is one of the leading advocates for MySQL at Facebook and in the MySQL community. Working with him and the original Ryan (as I call Ryan Mack, who preceded me on this team) has been nothing short of an extraordinary opportunity for me to learn from the best.

On Friday, I graduated Facebook bootcamp, a six-week onboarding program that is designed to get new engineers at Facebook up to speed quickly. Overall, I think bootcamp is a good program — it certainly beat the socks off of my onboarding experience at Amazon. Facebook moves even faster than Amazon, yet the onboarding sessions were up-to-date because they were owned and presented by engineers, not relegated to some out-of-date wiki (although, overall, Amazon’s wiki is considerably better than Facebook’s).

At any rate, the best parts of bootcamp for me were two excellent onboarding sessions: one on JavaScript a few weeks ago, and one on  Git last week. The engineers presenting had good presentations, but more importantly, they took time to answer my questions very thoroughly. It is very exciting to be working with people who have such deep knowledge and are so readily available.

One of the tasks during bootcamp is to figure out what team to join out of the teams that are hiring. I found I enjoyed most of the tasks I worked on, so I had trouble narrowing it down initially. In the end, it really came down to working on Scribe, an open-source distributed logging system that Facebook created, or working with the Databases team on what essentially amounts to MySQL hacking. As hard as the choice was to make, it was a good choice to have because I’m certain either team would have been an interesting place to work. However, in the end I had to make a choice, and today I started with the Databases team.

For now, my task is to get up to speed on Drizzle, a stripped-down branch of MySQL. I certainly have my work cut out for me, but I wouldn’t have it any other way.

Friday marked the end of my first work week at Facebook, my new employer. As far as first weeks go, it started off with a bang when I got a new Macbook Pro and an iPhone 3Gs on the first day. I’ve grown to like the iPhone pretty well (the Twitter and Facebook apps, particularly), but I still am having my struggles with the Macbook.

I love the touchpad and the transparent terminals, and the back lit keyboard is sexy and functional, but there are also a lot of keys I miss, especially when I’m away from the extended keyboard at my main desk — home, end, page up and down, and delete (forward) being the leading examples. The function key on the laptop is also misplaced (I prefer control on the outside, as is the case on every reasonable keyboard known to man). The menu-bar-atop-primary-display decision is, quite frankly, terrible. There are also lots of other little annoyances: half of what “should be” control-key is command-key (ie, copy and paste), while the other half is still control-key (ie, changing tabs in Firefox, everything in the terminal).

The end all is that I am seriously considering swapping the Macbook for a Thinkpad. I haven’t made that decision yet, but it may happen next week. Or I may decide to tough it out, since I’ve already learned how to deal with most of the quirks.

Apart from the Macbook blues, the first week went fairly well. The first day was the normal mostly-government-mandated HR stuff, and then on day two I got to check out all of Facebook’s codebase and set up a sandbox. With some onboarding sessions, good documentation, and some help from a knowledgeable returning intern, I was actually able to be a little productive in my first week, a vastly different experience than I had at Amazon last summer.

Also, the food has been uniformly quite good to excellent, so living cheaply (except for the exorbitant rent at my too-nice Palo Alto pad), will be possible. At least, I won’t be spending too much money on groceries or restaurants.

Last but not least, there was the news that had to make this the best week to start at Facebook ever:

1. Facebook hit 300 million active users, another unprecedented milestone in social networking
2. Facebook became, for the first time, cash flow positive, a year ahead of their own projections

So overall, a good week to start, I would have to say.