Month: January 2012

HDFS on Isilon – the new hotness

Today, EMC Isilon announced our introduction of HDFS support. Chuck’s blog post does a good job of explaining why our HDFS implementation is a leg up on everything else out there (in short: no single point of failure). This is so much hotness that at the internal EMC 2011 year-end-review meeting, the CEO had one slide dedicated to this. It was one of, at most, 3 slides mentioning specific products.

This is the project I helped with at the end of last year, both from a performance and a functional perspective (and the one I got such great affirmation on).

Announcements like this are another reason why I love working at Isilon. No one ever got excited when a new version of ITIM was released.

Interviewing at Isilon, part 3: Who we’re looking for

Right now we’re hiring lots of people, particularly in engineering. This includes developers, testers, project managers, and writers among others. I’m on the Test Core team and end up on interview loops and phone screens. In fact, every week I do at least 2 of them.

So what are we looking for? Well, let me tell you what we’re looking for in test. Although I can’t speak for the development org (heck, I’m probably not suppose to be speaking for test at large) I would assume their list is similar.

  • Test experience. This is a big one in the test organization, obviously. In Test Core, everyone ends up testing everything at some point, but each person owns a functional area they’re responsible for. When new features are in development, a tester is assigned to that feature and works hand-in-hand with development to ensure it is adequately tested. That person will write up a test plan (with help and feedback from the test team and review by development) and may execute it themselves or hand it off to someone else. Individuals with existing test experience, even if it has nothing to do with the storage industry, are awesome. New just-out-of-college hires won’t have this experience, and that’s ok. Heaven knows Texas A&M certainly didn’t do a good job of teaching me anything about test. Which leads us to…
  • Critical eyes. Testers are advocates for the customer. The best part of our job is that we get to break things. Horribly break things. And get paid to do it. Individuals with a critical eye, even if they don’t have prior test experience, are great. If you’re tenacious, have attention to detail, I’ll even go so far as to say a bit OCD about getting things Just Right, we want you.
  • Unix experience. OneFS runs on top of FreeBSD so folks who have unix experience of some sort (Linux, FreeBSD, Solaris, AIX, doesn’t matter — even HPUX) are a plus. Granted, here in Seattle many folks have been tainted (I almost said poisoned) by Microsoft and have nothing but Windows experience. We recognize that there are great candidates out there who may not have unix experience, so this isn’t a hard-and-fast rule. But ideally: we want folks who aren’t afraid at a command line and who can dance a jig with standard unix commands. Bonus points if you grok NFS.
  • Storage experience. We’re a storage company, if you have storage experience that’s a plus. Again, we don’t expect many people to have it so this is a bonus, not a requirement. Even having a high understanding of file systems (ext2/3/4, NTFS, etc — we don’t really care) is great. Experience with NAS, SAN, and/or RAID super bonus.
  • Scripting experience. We don’t hire people to do manual testing and while we’re not requiring testers to have been former developers, we need folks with some level of scripting / automation experience. We’re a python shop (don’t get me started, I’m personally a perl guy) but if you know any type of structured programming language that’s likely good enough. For a data point, I didn’t have any practical python experience and they still hired me.
  • Passion. We want people who are passionate. Excitement about the job with a willingness and ability to learn can counteract shortcomings just about anywhere else. I’d much rather hire someone who is excited about Isilon and what we’re doing but doesn’t have any test experience than someone who has years of test experience and is just looking for a job.
  • Culture fit. This one is ooey-gooey subjective but important. We’re going to have to work with you ~8+ hours every weekday and not want to kill you at the end of a month. Isilon is a fun and challenging company. We value diversity, both in skills, cultures, sexual orientation, genders, and personalities. If that’s something that you’re not comfortable with — don’t even walk in the door. Isilon managers don’t have the time or desire to micromanage their employees, and that’s a good thing. That means we need folks who can be given a task or direction and will go run with it. And when you get stuck, because we all do at some point, can speak up and ask questions. Importantly, we’re not looking for sheep. If you disagree with how something is being done and think you have a better way, we want you to speak up and advocate for your idea. That said, if after a discussion the group decides to not go with your idea, don’t pout and go home.
  • Performance experience. This is not a requirement, but is double-plus bonus. We’re looking for another performance person. Someone who has experience with measuring the performance of a software product, generating benchmarks, and diving into diagnostic data to find the bottleneck. This person must not be afraid of poking and writing code, kernel, userspace, even test automation. I don’t have a good job description for this, mostly because it’s a jack-of-all-trades position since performance is a holistic investigation not single point problem. You’ll be working closely with me — I’ll let you decide if that’s an incentive or disincentive!

I’m but one voice in the test organization, not a hiring manager, and we hire and interview as a team. So while the above list reflects what I’m looking for in a candidate I interview, that isn’t to say it’s the gold standard everyone use. However, if you can’t see a faint reflection of yourself in the list above – Isilon may not be the place for you.

If, on the flip side, you resonate with the rambling bulleted list above, send me your resume — we’d love to talk with you. Isilon email addresses are in the form first.last@isilon.com (and anyone worth their salt should be able to ascertain what my first and last name is :).

See also: Interviewing at Isilon, part 1: Why you should and Interviewing at Isilon, part 2: What we do.

Interviewing at Isilon, part 2: What we do

Lately I’ve interviewed and phone screened more and more people who have no idea what Isilon does or why they want to work here.

Part of that I understand. Isilon sells to enterprises, not consumers, so we don’t have a great presence even within tech. Heck, most people don’t even know what our parent company EMC does and they’re a much larger company (hint: it’s primarily storage). But there’s no excuse for not googling the company you’re interviewing with.

If you’re interviewing at Isilon for any job, particularly any position in engineering, go read “Big data meets big storage: an in-depth look at Isilon’s scale-out storage solution” at Ars Technica. It gives a fantastic overview of what makes our system awesome — a peek into our architecture and secret sauce. You’re by no means required to understand it all, but it’ll give you a base knowledge of our technology, a leg up on the interview, and points in your favor for being ahead of the game.

See also: Interviewing at Isilon, part 1: Why you should and Interviewing at Isilon, part 3: Who we’re looking for.

Interviewing at Isilon, part 1: Why you should

Isilon is hiring. In fact, we’re doubling the size of our test teams alone this year, not to mention the hiring of developers, writers, and PMs. And you want to work for us for several reasons.

  • We’re working on cool technology. Petabytes of data (“big data” in today’s marketing lingo) meet distributed computing. Don’t buy more storage than you need right now. Need more space? Need more performance? Just add more nodes — our filesystem grows seamlessly with you.
  • Our products enable cool things. Isilon gear is used to make movies, help genomic researchers, serve and process news content, and power the company ultimately behind your Google and Bing maps, among others. Our gear is so vital to some of our customers that I can’t even mention their name for fear of tipping their hand to their competitors.
  • Isilon is full of smart people. From developers to testers, doc folks to sales engineers, PMs to hardware designers – our folks are crazy smart. Every day they solve hard problems with creativity, not brute force. It’s not your average joe that can double the write throughout of your hardware with a software update!
  • We work hard. People here are passionate about what they’re doing. We take initiative, aren’t micromanaged, and motivated by the challenge and our peers. That said…
  • We have fun. Our game room has a keg and a ping-pong table. We have free soft drinks and coffee (some of you laugh that this is a perk and not an industry standard — you’ve obviously not worked at IBM). We have grass-roots ordering of Isilon-branded beer steins, shot glasses, bamboo laptop covers, and even snuggies. Yes, I said snuggies. We have an unofficial Isilon Drinking Team that meets for a pint and often karaoke.

We have all the fun and hard work of a startup but with a big-company paycheck and job security. Come join us!

See also: Interviewing at Isilon, part 2: What we do and Interviewing at Isilon, part 3: Who we’re looking for.

Updated blog design

I spent way too much time tonight updating the design of my LJ blog (which is pulled into my main home page in case you’re reading this there — that design hasn’t changed). I’m never satisfied with a pre-packaged theme — I always have to tweak it. I’m mostly happy with the end result although it’s possible it may get a few minor updates over the weekend.

It appears that some pages are being served with the old theme. I hope this is transitory as its entirely on LJ’s side of things.

Development leadership failure

Last night I did some dev work for DP. Mostly some code cleanup (heaven knows we need it) but also rolling out some committed code to production. I’ve made a concerted effort to get committed-but-not-released code deployed — some of which has been waiting for, literally, years.

Even worse, we have reams of code updates sitting uncommitted (and slowly suffering from bitrot) in volunteers’ sandboxes waiting for code review. In the case of Amy’s new quizzes, for almost 5(!!!!) years. In other cases volunteers have done a crazy amount of legwork to address architectural issues that remain unimplemented due to no solid commitment that if they did the work it would be reviewed, committed, and deployed — like Laurent’s site localization effort.

These are clear systematic failures by development leadership, ie: me. It’s obvious why even when the project attracts developers, we can’t retain them.

The first step is to get through the backlog of outstanding work. I have Laurent’s localization work almost finished. This will allow the site to be translated into other languages — I think Portuguese and French are already done. Next up is getting Amy’s new quizzes pushed out. She’s done a marvelous job of keeping her code up to date with HEAD based on my initial work last night. Now to get them committed and rolled out. Then a site-wide change on our include()s required to get full site localization implemented.

After all that, we need to address how to better keep code committed and rolled out. I think we as a team suffer from “don’t commit until it’s perfect, then wait until it’s simmered before rolling it out”. Where “simmered” means “sitting in CVS with no active testing done on it”. We need to move to a more flexible check-in criteria or a more liberal roll-out. There’s no good reason why the bar is so crazy high on both ends of that.

But first – the backlog.

Out at the gym

This morning I somewhat intentionally outed myself at the gym. Chris was curious what a day guest pass costs at Rain, so on my way out of the gym today I asked the front desk attendant who was out on the floor. She asked if I had someone I wanted to join me and, after a brief hesitation1, I replied “my boyfriend” — within earshot of some of the regulars.2

I’m totally out everywhere else in my life, but for some reason I’m always hesitant to be out at the gym. I can’t quantify exactly why that is3, although it’s likely due to not wanting to make anyone uncomfortable in the locker room. Don’t misunderstand, I’m there to work out, get cleaned up, and leave, not hit on, or leer at, anyone. Still, there’s no point giving a homophobe a target.

Then again, it’s entirely possible that everyone at the gym knew I was gay already.

1 The fact that I hesitated, even briefly, bothers me more than anything. I need to puzzle out exactly why that is (both the hesitation and why it bothers me).

2 I think she was relieved to figure out which team I bat for. I think she’s been flirting with me for the past few months. I’m terrible recognizing when gay men are flirting with me, much less straight women, so I could be totally off here.

3 In some gyms (Golds on Cap Hill or 24 downtown) being out is more likely to get you hit on, rather than hit. Even at Rain it’s not like I’m afraid of being beaten up.

Microsoft supports gay marriage in WA State; Where’s EMC Isilon?

Yesterday Microsoft publicly declared their backing of the two WA marriage equality bills (SB 6239 and HB 2516). Their reasoning:

As other states recognize marriage equality, Washington’s employers are at a disadvantage if we cannot offer a similar, inclusive environment to our talented employees, our top recruits and their families. Employers in the technology sector face an unprecedented national and global competition for top talent. Despite progress made in recent years with domestic partnership rights, same-sex couples in Washington still hold a different status from their neighbors. Marriage equality in Washington would put employers here on an equal footing with employers in the six other states that already recognize the committed relationships of same-sex couples – Connecticut, Iowa, Massachusetts, New Hampshire, New York and Vermont. This in turn will help us continue to compete for talent.

In short: it’s good business sense.

I was recently asked by my mother if I would ever move back to Texas. The answer was a resounding no. I made a promise to myself years ago that I would never move to a state where I had fewer rights than I did where I was currently located1. Colorado was a step up from Texas. Washington was an even bigger step up from Colorado. Top talent has a choice on who they work for, and what state that is in.

Which makes me say: hey EMC Isilon – where’s your voice? EMC is based in Massachusetts, so you should be well familiar with marriage equality. This is a perfect time for Isilon to get more visibility that we’re a WA company who values our employees and supports equality. Because after all, it just makes good business sense.

1 Texas’ rabid conservativeism being a close second.

Update: It took them 8 months, but they did it!

Feelin’ Mavericky

The code name for the next version of OneFS is Maverick and everyone in my office is heads down to move that mountain. I also can’t stop hearing Sarah Palin in my head every time someone says it.1

Like everyone else in my office – I’m swamped. So mentally swamped that I’ve started emailing myself at random times of the day with things I need to do. Like just now on the bus ride home. I’m mere days away from cascading lists complete with full dependency chains.

I gotta get the reigns on this horse before it runs all over me. There’s no way I’ll survive 6 more months of this with any semblance of sanity. Otherwise I’ll be as Mavericky as Sarah Palin!

1 Well, technically I can’t stop hearing Shaun mimicking Tina Fay mimicking Sarah Palin.

Romney and the Religious Right

The Religious Right, specifically the Evangelicals, are in a tizzy. Their worst fear is that Romney will get the Republican nomination and they’ll be left with two unpalatable choices: a Democrat who is a Christian1 or a Republican who is a cultist2.

Lest you think I’m joking, let me reassure you that I know from first-hand experience. I was raised in a small town in the Texas panhandle. I attended church in that town from when I was born until I left for college at 16. About 12 years or so of that was at the Methodist church and the other half was at the First Baptist church (part of the Southern Baptist Convention). And despite the greater Methodist church being more accepting and liberal – Methodist churches in small towns are as conservative as you can get. So I can speak with authority what conservative small town evangelicals are taught and, presumably, believe.

Both churches taught that Mormons3 weren’t Christians but were, in fact, a cult. These churches only begrudgingly conceded that Catholics are Christian if pressed4. But Mormons? They’re in the same bucket as Jehovah Witnesses, wiccans, and satan worshipers.

So that brings us back to Romney. A group of evangelical leaders met this past weekend to decide which non-Romney candidate they’ll support. Note that it wasn’t which Republican candidate, but which non-Romney candidate. Their choice? Rick Santorum. Keep in mind that Santorum is Catholic, not Protestant. They’re that desperate to keep Romney out of the White House. Their only non-Mormon, non-Catholic choices were Rick Perry and Ron Paul, neither of which are serious competitors to Romney. So why Santorum instead of Gingrich, who is also Catholic? My guess is that the twice-divorced, cheated-on-two-wives Gingrich isn’t the “family values” candidate they were looking for.

What does this mean for those of us not falling off the right side of sanity? Grab some popcorn, throw a couple of bucks Romney’s way before the primary, and watch the evangelicals squirm from now through November.

Update: And just in case you don’t believe me.

1 Assuming they really believe he’s a Christian as he professes to be. Many of them still believe he’s a Muslim. Sadly I’m not joking.

2 I don’t believe this. Or rather I believe that most all religious groups are cults. Either way, I hold that Mormons are no worse off than anyone else.

3 Anyone seen the new Mormon ad campaign? The buses around here have large, 3-panel ads with a person in each panel with the phrase “I’m a Mormon” in them. Sadly. Every time I see one my mind parses it as “I’m a moron”. Mormons aren’t morons, and I don’t think they are. But pure #FAIL on the ad campaign.

4 If cornered they usually say something like “As long as [the Catholic] accepted Jesus as their savior, then they’re a Christian. If they haven’t and are just listening to the pope, they’re not”.