Yesterday I noticed that the search assistant component of our client was consistently chewing up 25% of my CPU – not a good thing. It wasn't having any noticeable impact on my machine, but still, some thread or other had apparently spiraled out of control. This sort of bug doesn't show up very often, and they're terribly difficult to reproduce, so I suspected we were going to have to debug it in place.
So this is where it gets embarrassing for me. As Zango's CTO, you'd think I could do this in my sleep. Isn't that what CTO's do, debug code? But it's been so long since I've written any production code, that I had to ask our lead client developer to come over to my desk, and have him walk me through grabbing the right label from TFS, finding the right symbols file, and navigating through the call stack on a couple hundred suspicious looking threads.
(For those of you who care: turns out that the binary tree we use to search for keywords quickly had gotten corrupt somehow, and a particular child node was pointing back to one or another of its parent nodes: as our lead dev put it, "It's no longer a tree; it's a graph." He's still back at his desk trying to figure out what happened. It's presumably a rare timing issue that didn't show up in our QA, and we'll get it fixed for the next release.)
But my poverty of C++ debugging skills got me thinking: what does it take to be a CTO? Apparently it's not coding skills, as I've never done production C++, and I haven't written a line of production C# or even SQL for years now. I'm a lousy project manager: I'm not organized enough, and I'm too easily distracted. I'm a reasonable systems architect, but I'm very glad we've got people who are a lot smarter than me doing that now. E. F. Codd would roll over in his relational grave if he saw the database schemas I used to churn out. I haven't done calculus since high school, I've never taken a class on statistics, and I've never implemented even the barest Quicksort algorithm.
But even with all of this, Zango's Tech department is doing pretty well. We've been turning out solid products on time for years now. Last year we pushed out some sixty different projects, most of them on time and under budget. (OK, 10.0 was neither, but trust me, it stood out.) Our uptime is right at three nine's ("99.9%", for those of you not in the industry). We've got a world-class BI infrastructure, hourly loads into our multi-terabyte data warehouse, and better analytics than companies ten times our size. Despite the fact that Zango's tech department has 90 people spread across five offices, three countries, two continents, and ten time zones, we're less political than any organization I've ever worked for. We get along with each other, we get good work done, and we have fun doing it.
So why does it all work?
It comes down to the people here. When Keith roped me into helping him start Zango back in 1999, we wanted to create a company that was fun to work for, that didn't get hung up on politics. We wanted to work with people who had big brains and small egos. And it's almost a cliché these days, but we set out to hire people who were better than we were. (I wish that were harder to do.)
I often brag that I don't do anything anymore – which leaves my wife wondering why I leave for the office at 6:00 am and often don't get home until 8:00 in the evening. But at some level, it's true. We've got phenomenal people here, who work hard, care enormously about what they do, and who deserve all the credit for what Zango has become.
In other words, if you want to be a good CTO, hire good people.