Ciao EMC Isilon, hello BlackSky

Friday, June 17th will be my last day at EMC Isilon. It’s been a fun 5.5 years but for a variety of reasons it’s time to move on to new adventures. I’ve had the pleasure of working with some amazing people on some superbly-interesting projects and I wish them all the best on the cool stuff they’re working on next. (To Infinity and beyond!)

Monday, June 27th I start work at BlackSky Global (BSG) where I’ll have the pleasure of working with Jane Vail and Eric Rybczynski again. I’m being brought on to build up their validation team1 and drive their end-to-end testing alongside Eric.

BSG began courting me last August but that was just two months into my sabbatical and I decided the sabbatical wasn’t something I could give up. I am unbelievably fortunate that they were unable to fill the position and still have a place for me now.

I’m excited to be working at my first startup and at a company solving complex problems in space, [say it with me] the final frontier.

An unexpected bonus is that BSG is only a 10-minute walk from my house down a long flight of stairs, making my morning commute a bus-free breeze. The walk back home, on the other hand, is going to be a 15-minute straight-up Stairmaster workout. Buns of steel, here I come!


1 This is in no way soliciting existing EMC (and soon, Dell) employees to join us, unless today’s date is after 2017/06/17.

Enabling teams to warp factor 10

One of my strengths is enabling teams. Seeing spots where some documentation here or a tool there will help move the team forward and increase their velocity. This is something that is hard to hire for, or call out that you do well on a resume, but is really important for building effective and efficient teams.

Being successful at it requires the ability to step back and look beyond the immediate technical problems to see where people are struggling or wasting time. It also requires good communication skills to ask the right questions and engage the right players to solve them. And, quite frankly, sometimes it involves working outside of an established process and asking for forgiveness instead of permission.

For me, this is just an extension of performance. Instead of optimizing a piece of software or hardware it’s optimizing teams.

What does enablement look like?

What does it actually mean to “enable teams”? How does this business-speak translate into the real world? How can we get a team to warp factor 10?

What follows are some examples of how I’ve enabled teams by creating tools, writing documentation, and leading teams. Keep in mind that through all of this I was officially a consulting-level validation engineer in the performance team.

Enabling your team

The EMC Isilon performance team is one of the most geographically-distributed teams we have with developers in multiple US timezones (including Hawaii) and countries (including Denmark, Italy, and France). Our master SVN server, however, is located in Seattle. Our remote developers were having a heck of a time doing SVN branch merges on their local workstations across the WAN — the WAN latency being the bottleneck. Merges could take hours or even timeout and fail altogether.

To fix this I took an older hardware client with a 10GigE network interface, threw in some spare SSDs (we are a storage company after all), and put it up in the Seattle lab for remote devs to use for their merges. This significantly decreased the time it took to do branch merges, not to mention reducing the headache of doing it. This was so successful that I put similar clients in our remote labs for other developer-related activities that take place in those locations. These remote shared workstations have since become popular with other teams inside the engineering organization as well.

Take away: Enabling teams can be a very low-level activity, it doesn’t always have to be grand and visionary. Something as simple as playing the IT guy can significantly improve the day-to-day life of your team.

Enabling your team by enabling others

The EMC Isilon OneFS operating system is built on top of FreeBSD. OneFS 8.0 upgrades the underlying FreeBSD version from 7 to 10. The development process used to do the upgrade prevented developers from testing their incremental code changes in the standard Isilon regression framework until almost code-complete and provided no way to leverage any existing test automation. A prior effort had a similar testability problem and was ultimately unsuccessful. Moving to FreeBSD 10 was important to the organization as a whole, but particularly so for the performance team as we kept hitting limitations of FreeBSD 7 when trying to address performance problems.

Seeing the success of the FreeBSD 10 effort as vital to unblocking my team’s goals I volunteered to help. I created a lightweight isolated virtual framework built on top of VMware Workstation that allowed developers to run a small set of regression tests on their branch to validate it before they merged it. Validation engineers were able to work alongside developers to add tests to this framework as code was developed. This approach enabled the validation team to test the code as it was dropped in instead of waiting until the end when the new product was capable of being run inside the standard Isilon framework.

Take away: Enabling your team may mean taking a step back, seeing how it depends on others, and assisting them directly.

Enabling your team by enabling the larger organization

In mid-2014 EMC Isilon started pushing towards a more DevOps-centric model and I jumped at the opportunity to help the organization build better products faster. For internal purposes, we have been using virtual nodes running on VMware Workstation for many, many years. These enable developers to quickly spin up nodes on their workstations, test something out, and break them down without waiting for physical hardware to become available or waiting for the hardware to reimage. The downside is that these only ran on a developer’s workstation, require beefy workstations, and limit developers to 3-node clusters.

Seeing how underutilized virtual nodes were, and yet how powerful they are for both development and test, I volunteered to lead the team to make virtual nodes more accessible. I did so because I believed I had a unique understanding of the interdependencies involved and the skills to make it successful. Management agreed and I ended up equal parts team lead, project manager, developer, and tester for the DuctTape team. Together we made virtual clusters running on centrally-hosted ESXi systems a reality. Now both developers and testers can spin up a virtual cluster of any size running almost any build at the push of a button. This capability was delivered just in time for the final testing of the FreeBSD 10 code upgrade (mentioned above) before it could be run on hardware rigs. It was later incorporated into our standard regression wheel to supplement the physical hardware rigs already in use, further increasing our test concurrency.

Take away: Enabling your team can mean taking a huge step back, looking at organization-wide limitations, evaluating your skill set against those required to address those limitations, and putting yourself out there to try.

Incremental enablement

Developing for Distributed Proofreaders has always been challenging due to the need for very specific (and antiquated) middleware and the use of CVS as source control. Developers struggled to get a working development environment set up to test their changes and then no way to effectively have multiple changes in-flight due to CVS’s poor branch support.

During my 9-month sabbatical I iteratively addressed these by:

  • moving the project to git
  • documenting recommended workflows for the project centered around user branches to have multiple changes in-flight
  • creating a development VM with all supporting middleware already installed and a base set of data populated for testing
  • conducting two git training bootcamps using the VM to get developers up to speed on how to use git in the new development model

These have already paid off and we have several new developers test-driving code changes.

I’m in the process of introducing unit tests for existing classes and establishing them as a best-practice for future classes and refactorings.

Take away: Enabling teams is an incremental process. Once you have one roadblock removed, look afresh and see what the next one is. You don’t have to solve it all at once: incremental progress is progress all the same. And don’t overlook the importance of good documentation!

Not in my job description

None of the above roles were in my “job description” as a performance validation engineer, but I saw problems that I could help fix and fixed them.

In the case of the FreeBSD merge and the DevOps work, I volunteered to take on the role which then became my job for several months. I was able to identify and articulate to management how I was well-qualified to help address the problem and that addressing the problem was how I could best serve the overall business at that time.

I don’t know of anyone whose sole job description is a “team enabler”. Instead, it is everyone’s responsibility to find those areas around them where they can help their teammates work more quickly, efficiently, and thereby increasing the velocity of the team overall.

No one works in isolation

In the modern tech world everyone works on a team of some sort — no one works in isolation. The ability to identify what is tripping up a team and taking the initiative to fix it is an important skill for an individual contributor and a hard requirement for any team lead or manager.

Cultivating the skill of team enablement increases your value to any organization. Being able to highlight it on your resume, discuss it with recruiters, and communicate it during interviews will increase your hireability.

A leave of absence from EMC Isilon

Starting June 15th, I will be taking a one year unpaid leave of absence from EMC Isilon.

The reason for the leave of absence is that I’m burned out. Crispy critter burned out. Although I’m not going to go into detail on exactly why, the 30k-foot version is that I held myself to too-high a standard on some projects that drastically, and negatively, impacted my work-life balance to the point where I was mentally and physically exhausted when I came home from work. Every day. For months.

I’m looking forward to mentally rebooting myself and hopefully learn some skills on how to help prevent this in the future.

That said, it’s incredibly difficult to leave EMC Isilon right now. It’s hard to leave just before the next version of OneFS1 ships. I’m super excited about the work that the entire engineering team has done for this release and would love to see it complete. It’s also hard to leave right when DuctTape is releasing some cool new functionality and significantly increased capacity.

Still, taking some time off is most certainly the right decision for my mental and physical health and I leave the projects I was working on in very capable hands.

1 According to the interwebs, the code-name of the next release is not yet public, and I’m not about to be the one to make it so!

Career levels across Seattle-area tech employers

Disclaimer: These thoughts and opinions are my own and not my employer’s, although if you didn’t know that already you’re clearly not paying attention.

EMC Isilon recently refreshed their career level definitions and it got me thinking of how difficult it is to map levels from one company to another. EMC uses a P1-P7 numbering system, IBM uses a band 6-band 9 numbering system, Microsoft uses levels numbered 59-68, Amazon calls theirs levels L1-L10, etc. How is anyone suppose to make sense of this mess across companies? So I set out to do what any engineer would do: determine if there was a mapping between them.

From my experience at IBM, pre-acquisition Isilon, and now EMC I was able to do some of the mappings myself. I then reached out to friends who have worked for at least one of the three and asked how their current companies mapped to it. The following table includes the results thus-far:

IBM pre-EMC Isilon EMC Microsoft Amazon
P1 – Entry 59/60
band 6 SDE 1 P2 – Intermediate 61 L4 – college hires
band 7 SDE 2 P3 – Senior 62 L5 – mid-career
band 8 SDE 3 P4 – Principal 63 L6 – Senior
band 9 P5 – Consultant 64 L7 – Principal
STSM Staff P6 – Senior Consultant 65/66/67 – Principal L8 – Senior Principal
Distinguished Engineer Distinguished Engineer L7 – Lead /
Distinguished Engineer
68 – Partner L10 – Distinguished Engineer
Fellow Fellow Fellow

Discussion frameworks, not absolutes

The chart above is useful primarily as a framework for discussion. Even within a large company the skills for someone at given level may not map closely to others at that same level. Add cross-company evaluations into the mix and the above is, at best, a guideline.

Why is this important?

People change jobs frequently in the tech industry (no surprise given employees who stay in companies longer than 2 years get paid 50% less). My 10-year career at IBM and 4+ years (so far) at EMC makes me an outlier in a field where most of the resumes I see have people staying just 1-2 years between jobs.

The chart above can be useful when changing jobs across companies. When I left IBM to come work at Isilon this would have come in very handy. Instead of moving from my band 9 position at IBM to at least an SDE3 or Staff position at Isilon, I moved over as an SDE2 — two steps down. Next time I’ll have more data to make a solid lateral (or upward!?) transition.1

Levels !== Pay

Everyone I’ve talked to has confirmed that pay does not directly map to levels. It is common for the upper end of the pay range of level X to overlap greatly with level X+1 and maybe even border the lower end of level X+2.

What about Company X?

I’d love to get other large tech companies like Google and Facebook on here, I just haven’t found anyone willing to share their insight. If you have overlap with one of the above drop me an email and lets fill in the blanks.

1 FWIW, I currently have no intention of leaving EMC Isilon for another job.

LinkedIn – EMC Isilon Experience

I was updating my LinkedIn profile today1 and, after finishing my new write-up for EMC Isilon, noticed that I was drastically over their character limit. I’ve pared it down but thought I’d post the whole thing here since I went to all the effort of writing it to begin with!

1 No, I am not looking for a new job. I just got an itch to update the thing.


I’ve worn many hats so far in my short time at EMC Isilon: performance tester, performance team lead, platforms test lead, and DevOps architect.

Right now I’m leading four teams who together are reinventing how we build OneFS within Isilon engineering. We’re moving to a DevOps model leveraging continuous integration for more timely, focused feedback to developers on the state of their code. I’m personally responsible for the overall architecture of this new system and ensuring that the four teams (Infrastructure as a Service, Test as as Service, Build as a Service, and the Orchestration layer) are all integrating together. I’m the single point of contact for the Engineering organization, both out to my peers and up to management, on the state of the project. I’m also a technical consultant for the Infrastructure as a Service team since I have deep knowledge about the OneFS product as well as our internal infrastructure.

Prior to being a DevOps architect, I was the test team lead for the platforms team as they undertook upgrading the FreeBSD version that OneFS is based off of. This included creating and driving test plans, working with development on how to enable testability from day one, breaking work into smaller pieces for distribution to other team members, and interviewing potential team candidates. The thing I’m most proud of is creating tooling to enable testing of code from development before it could be plugged into the existing infrastructure. This enabled testing of code from before the first code commit to catch code regressions early.

Before the platforms team I worked with the overall performance team. This included all aspects of Isilon product performance (development, test, benchmarking, and support) across all product functional areas (kernel, networking, filesystem, and protocols). During the implementation of the Endurant Cache, one highly respected developer gave me the title of Performance Test Ninja which I’ve gladly kept to this day (’cause come on — it’s a cool title!). I was a team lead and contributor on the creation of the Performance BVT — an extensible performance test suite for automatic performance evaluation to ensure no performance regressions from build-to-build. Passing the PerfBVT (as it is affectionately known) is part of the merge criteria for feature branches before they merge the main code repo. The PerfBVT has caught several major performance regressions before they were ever merged.

Throughout my time at EMC Isilon I’ve reported to the test organization which gives me free reign to have insane focus on quality. I’ve not lost my technical chops either as I’ve muddled around in the FreeBSD TCP stack when troubleshooting 10GigE performance, slogged through kernel traces, and identified code bottlenecks (before tossing the bug to development to fix).

Outside of my “day job” mentioned above, I also have a passion for recruiting, diversity, and training. I’ve given tech talks at colleges, regularly participate in interviews for positions throughout engineering, give Engineering 101 classes for new hires, I’ve been a mentor of Ada Development Academy interns, represented EMC Isilon at GeekGirlCon and the Out & Equal Career Fair, and was on the formative board of the EMC LGBT West Coast employee resource group.

Isilon Anniversary – 4 years

Sunday marks my 4-year anniversary at Isilon. It’s been a fun ride and taken me into roles that I would have never seen myself (BSD10 merge test lead, The Factory whipping boy … I mean architect). I enjoy working with some really remarkable, bright, and passionate people solving challenging problems — they make it all worthwhile. I’ve been a part of several major software releases (Mavericks, Waikiki, Jaws, and Moby) and a whole slew of our new hardware platforms (S200, X200, X400, A100, NL400, S210, X410).

I’ve also enjoyed the diversity work I’ve been able to do, including representing EMC Isilon at the Out & Equal Career Fair, manning a booth at GeekGirlCon, and working with the Ada Developer Academy.

Here’s looking forward to whatever craziness the next year has in store for me!