Ciao EMC Isilon, hello BlackSky

Friday, June 17th will be my last day at EMC Isilon. It’s been a fun 5.5 years but for a variety of reasons it’s time to move on to new adventures. I’ve had the pleasure of working with some amazing people on some superbly-interesting projects and I wish them all the best on the cool stuff they’re working on next. (To Infinity and beyond!)

Monday, June 27th I start work at BlackSky Global (BSG) where I’ll have the pleasure of working with Jane Vail and Eric Rybczynski again. I’m being brought on to build up their validation team1 and drive their end-to-end testing alongside Eric.

BSG began courting me last August but that was just two months into my sabbatical and I decided the sabbatical wasn’t something I could give up. I am unbelievably fortunate that they were unable to fill the position and still have a place for me now.

I’m excited to be working at my first startup and at a company solving complex problems in space, [say it with me] the final frontier.

An unexpected bonus is that BSG is only a 10-minute walk from my house down a long flight of stairs, making my morning commute a bus-free breeze. The walk back home, on the other hand, is going to be a 15-minute straight-up Stairmaster workout. Buns of steel, here I come!


1 This is in no way soliciting existing EMC (and soon, Dell) employees to join us, unless today’s date is after 2017/06/17.

Enabling teams to warp factor 10

One of my strengths is enabling teams. Seeing spots where some documentation here or a tool there will help move the team forward and increase their velocity. This is something that is hard to hire for, or call out that you do well on a resume, but is really important for building effective and efficient teams.

Being successful at it requires the ability to step back and look beyond the immediate technical problems to see where people are struggling or wasting time. It also requires good communication skills to ask the right questions and engage the right players to solve them. And, quite frankly, sometimes it involves working outside of an established process and asking for forgiveness instead of permission.

For me, this is just an extension of performance. Instead of optimizing a piece of software or hardware it’s optimizing teams.

What does enablement look like?

What does it actually mean to “enable teams”? How does this business-speak translate into the real world? How can we get a team to warp factor 10?

What follows are some examples of how I’ve enabled teams by creating tools, writing documentation, and leading teams. Keep in mind that through all of this I was officially a consulting-level validation engineer in the performance team.

Enabling your team

The EMC Isilon performance team is one of the most geographically-distributed teams we have with developers in multiple US timezones (including Hawaii) and countries (including Denmark, Italy, and France). Our master SVN server, however, is located in Seattle. Our remote developers were having a heck of a time doing SVN branch merges on their local workstations across the WAN — the WAN latency being the bottleneck. Merges could take hours or even timeout and fail altogether.

To fix this I took an older hardware client with a 10GigE network interface, threw in some spare SSDs (we are a storage company after all), and put it up in the Seattle lab for remote devs to use for their merges. This significantly decreased the time it took to do branch merges, not to mention reducing the headache of doing it. This was so successful that I put similar clients in our remote labs for other developer-related activities that take place in those locations. These remote shared workstations have since become popular with other teams inside the engineering organization as well.

Take away: Enabling teams can be a very low-level activity, it doesn’t always have to be grand and visionary. Something as simple as playing the IT guy can significantly improve the day-to-day life of your team.

Enabling your team by enabling others

The EMC Isilon OneFS operating system is built on top of FreeBSD. OneFS 8.0 upgrades the underlying FreeBSD version from 7 to 10. The development process used to do the upgrade prevented developers from testing their incremental code changes in the standard Isilon regression framework until almost code-complete and provided no way to leverage any existing test automation. A prior effort had a similar testability problem and was ultimately unsuccessful. Moving to FreeBSD 10 was important to the organization as a whole, but particularly so for the performance team as we kept hitting limitations of FreeBSD 7 when trying to address performance problems.

Seeing the success of the FreeBSD 10 effort as vital to unblocking my team’s goals I volunteered to help. I created a lightweight isolated virtual framework built on top of VMware Workstation that allowed developers to run a small set of regression tests on their branch to validate it before they merged it. Validation engineers were able to work alongside developers to add tests to this framework as code was developed. This approach enabled the validation team to test the code as it was dropped in instead of waiting until the end when the new product was capable of being run inside the standard Isilon framework.

Take away: Enabling your team may mean taking a step back, seeing how it depends on others, and assisting them directly.

Enabling your team by enabling the larger organization

In mid-2014 EMC Isilon started pushing towards a more DevOps-centric model and I jumped at the opportunity to help the organization build better products faster. For internal purposes, we have been using virtual nodes running on VMware Workstation for many, many years. These enable developers to quickly spin up nodes on their workstations, test something out, and break them down without waiting for physical hardware to become available or waiting for the hardware to reimage. The downside is that these only ran on a developer’s workstation, require beefy workstations, and limit developers to 3-node clusters.

Seeing how underutilized virtual nodes were, and yet how powerful they are for both development and test, I volunteered to lead the team to make virtual nodes more accessible. I did so because I believed I had a unique understanding of the interdependencies involved and the skills to make it successful. Management agreed and I ended up equal parts team lead, project manager, developer, and tester for the DuctTape team. Together we made virtual clusters running on centrally-hosted ESXi systems a reality. Now both developers and testers can spin up a virtual cluster of any size running almost any build at the push of a button. This capability was delivered just in time for the final testing of the FreeBSD 10 code upgrade (mentioned above) before it could be run on hardware rigs. It was later incorporated into our standard regression wheel to supplement the physical hardware rigs already in use, further increasing our test concurrency.

Take away: Enabling your team can mean taking a huge step back, looking at organization-wide limitations, evaluating your skill set against those required to address those limitations, and putting yourself out there to try.

Incremental enablement

Developing for Distributed Proofreaders has always been challenging due to the need for very specific (and antiquated) middleware and the use of CVS as source control. Developers struggled to get a working development environment set up to test their changes and then no way to effectively have multiple changes in-flight due to CVS’s poor branch support.

During my 9-month sabbatical I iteratively addressed these by:

  • moving the project to git
  • documenting recommended workflows for the project centered around user branches to have multiple changes in-flight
  • creating a development VM with all supporting middleware already installed and a base set of data populated for testing
  • conducting two git training bootcamps using the VM to get developers up to speed on how to use git in the new development model

These have already paid off and we have several new developers test-driving code changes.

I’m in the process of introducing unit tests for existing classes and establishing them as a best-practice for future classes and refactorings.

Take away: Enabling teams is an incremental process. Once you have one roadblock removed, look afresh and see what the next one is. You don’t have to solve it all at once: incremental progress is progress all the same. And don’t overlook the importance of good documentation!

Not in my job description

None of the above roles were in my “job description” as a performance validation engineer, but I saw problems that I could help fix and fixed them.

In the case of the FreeBSD merge and the DevOps work, I volunteered to take on the role which then became my job for several months. I was able to identify and articulate to management how I was well-qualified to help address the problem and that addressing the problem was how I could best serve the overall business at that time.

I don’t know of anyone whose sole job description is a “team enabler”. Instead, it is everyone’s responsibility to find those areas around them where they can help their teammates work more quickly, efficiently, and thereby increasing the velocity of the team overall.

No one works in isolation

In the modern tech world everyone works on a team of some sort — no one works in isolation. The ability to identify what is tripping up a team and taking the initiative to fix it is an important skill for an individual contributor and a hard requirement for any team lead or manager.

Cultivating the skill of team enablement increases your value to any organization. Being able to highlight it on your resume, discuss it with recruiters, and communicate it during interviews will increase your hireability.

Further thoughts on Dell’s acquisition of EMC

My prior public post about the EMC / Dell acquisition covered impacts to EMC shareholders and employee shareholders. This one is, again, my personal thoughts to discuss some of the broader implications. All of these observations are made from public sources and knowledge of how the tech industry works. I expect anyone familiar with the tech industry to be able to make similar observations / predictions.

Employees

For EMC (and in some respects, Dell) employees, the acquisition is going to be a mixed bag. Employees have no more details of the acquisition than the public at this point, so there’s a lot of uncertainty. So far, the message has been: it’s business as usual until the deal closes — which is at least 7 months away. I expect EMC will see higher-than-usual attrition over the next few months as folks who weren’t really looking before now leave for other opportunities. I wouldn’t be surprised if EMC starts offering some form of carrot to key players incentivizing them to stay. A lot of business obligations can come due in the next 7 months.

By many accounts Dell invested more money in R&D groups after going private. That’s good news for anyone in an engineering group. A friend also pointed out he “[thinks] it will be beneficial for engineering organizations [to be inside by a non-public company]. The less chained by the public market for short term profits and gains, the better the chances of being able to survive flat growth a quarter or two to realign priorities” — which I completely agree with.

Dell and EMC don’t have a whole lot of overlap in products, although there are a few, so there’s unlikely to be layoffs trying to reduce duplication there. There will probably be layoffs as Dell and EMC try to reduce non-engineering duplication in areas like HR, marketing, IT, sales support, and sales to some degree. Dell is going to need to reduce their ~$49 billion debt load after the acquisition (which they intend to do in 18-24 months) and it seems to me that reducing head-count is one way to achieve that. Another way of reducing their debt load would be to divest themselves of some products or product lines, either by selling them to someone else or simply stopping spending money on them by sunsetting the products and laying off the people.

Post-acquisition I would have expected Dell to put a kibosh on the use of Macs which are growing in EMC, but a friend who works for Dell said that other acquisitions which had Macs keep them alongside a Dell laptop for use in front of customers. Given how closely Dell seems tied to Microsoft, I expect its IT organization to push (require?) Windows workstations instead of Linux workstations which are the system of choice in Isilon.

That same friend at Dell says compensation is great and everything is cash or cash awards, so no stock — not a surprise since it’s a privately-owned company. They said it does appear that being private has allowed Dell to do more than play the quarterly-numbers game.

From some of the intangibles, there’s a whole lot of potential upsides and downsides. EMC is a very east-coast company who doesn’t really grok how west-coast companies work. I expect (but don’t know) that Dell being based out of Austin is more aligned with the west-coast mentality which is nice. Both EMC and Dell have a perfect 100 rating on the HRC Corporate Equality Index. Both have both LGBT and Women’s employee resource groups, among others. Dell trumps EMC in offering matching charitable donations but not on-site dry cleaning in MA like EMC — so east coast.

Recruiting

EMC probably isn’t going to be really pushing recruiting during the transition, except for backfills. I expect recruiting for EMC to be challenging until the deal closes. Typically public tech companies include some equity in their offer, either as stock options or RSUs, that vest over a few years. Equity is usually enticing because their worth is tied to the market performance of the company. If the stock goes up, the equity is worth more. This gives the illusion that hard work by an individual can improve the value of the stock.

Dell has assured that the EMC share price is largely fixed around $27 until the deal closes mid-2016. This also fixes the price of any options (which are rendered pretty much worthless if they are issued after today) and RSUs. This makes RSUs more like promises of fixed values than a promise of potential future growth rendering their use in offers less enticing than they were before the acquisition announcement.

It’s going to be hard over the next 12 months to even get people to interview with EMC. Who wants to start working at a company in the middle of a huge acquisition where there will be reorganization and “vision alignment” as soon as the deal closes?

I think overall the acquisition is going to be a positive thing, but the sooner it completes the better it is going to be for everyone.

Thoughts on Dell’s acquisition of EMC from public sources

On Monday, October 12th EMC and Dell announced that Dell was buying EMC. This will make for some interesting times for EMC shareholders and employees. The deal is made more interesting by the fact that, unlike EMC, Dell is not a publicly traded company. Michael Dell (with the help of Silver Lake) took his company private back in 2013.

There has been a lot of news about the acquisition. These are some personal, collated thoughts gathered from public sources (with citations) and in no way reflect the view of my employer (current or future) or any inside information.

Stockholders

From the official press release, EMC shareholders will get $24.05 a share in cash and $9.10 of a tracking stock tied to VMware. One could naively say this values EMC stock at $33.15, but there’s a twist. The twist is the tracking stock, an investment vehicle last seen prior to the dot-com bubble.

In theory, tracking stock gives investors a way to reflect value in another security (in this case VMware stocks owned by Dell) but generally grants no dividend or voting rights. Some analysts expect the tracking stock, which in theory will track the value of VMware stock, to trade from between 5 and 10% discount of that stock. (That link is an excellent read, BTW.) This partially explains why the stock has been trading between $27 and $28 since the acquisition announcement instead of closer to $33.

Also, note that as a private company, Dell won’t pay dividends like EMC, which is a bummer for those of us investors who liked that aspect. VMware doesn’t (yet?) pay dividends so there’s no love lost on that aspect of the tracking stock.

Employee Shareholders

Many (most?) employees are shareholders as well, either through the employee stock purchase program (ESPP) or through equity awards. What will happen to stock options and restricted stock units (RSUs) is unknown at this time.

According to Ars Technica, when Dell went private back in 2013 stock options that were underwater were cancelled. Unvested RSUs retained their vesting schedule but were paid out as cash upon vesting using the stock price when Dell was privatized. The upside is that the RSUs were essentially converted into promises of cash at a fixed value, but that’s a downside too — they aren’t going to be worth more than that fixed value, ever.

Employees & Recruiting

I have additional thoughts on how this impacts current employees and recruiting. While I believe all of them are logical conclusions that anyone familiar with the tech industry could derive, and they reveal no internal information, it’s not worth the risk of EMC legal’s wrath to post them here. So I’m posting them in a separate, locked post for my own edification.

A leave of absence from EMC Isilon

Starting June 15th, I will be taking a one year unpaid leave of absence from EMC Isilon.

The reason for the leave of absence is that I’m burned out. Crispy critter burned out. Although I’m not going to go into detail on exactly why, the 30k-foot version is that I held myself to too-high a standard on some projects that drastically, and negatively, impacted my work-life balance to the point where I was mentally and physically exhausted when I came home from work. Every day. For months.

I’m looking forward to mentally rebooting myself and hopefully learn some skills on how to help prevent this in the future.

That said, it’s incredibly difficult to leave EMC Isilon right now. It’s hard to leave just before the next version of OneFS1 ships. I’m super excited about the work that the entire engineering team has done for this release and would love to see it complete. It’s also hard to leave right when DuctTape is releasing some cool new functionality and significantly increased capacity.

Still, taking some time off is most certainly the right decision for my mental and physical health and I leave the projects I was working on in very capable hands.

1 According to the interwebs, the code-name of the next release is not yet public, and I’m not about to be the one to make it so!

Trying to quit

[This post was non-public when first posted.]

On Monday I gave my 4-weeks notice — it was not well received. Jonathan didn’t try to talk me out of it though. He said knowing me it wouldn’t do any good since I give things lots of thought before acting on them. He did, however, throw out the idea of a leave of absence.

As a rule, a 3-month leave of absence is the most they do which I said was a non-starter. It’s going to take them much more than 3 months to address some of the systematic issues that has me burnt out and leaving. Jonathan and HR are in the process of seeing if EMC will do a year’s leave of absence instead. We’ll see how flexible EMC wants to be.

The most flattering part of all of this is management’s view that my departure is going to be very impactful to the org from a morale perspective. I’m not sure I agree with that, but it’s flattering all the same.

I also discovered this week that 3 other well-respected and long-tenured people are leaving in the next few weeks, which makes me sad for Isilon and a bit guilty for being a 4th.

Career levels across Seattle-area tech employers

Disclaimer: These thoughts and opinions are my own and not my employer’s, although if you didn’t know that already you’re clearly not paying attention.

EMC Isilon recently refreshed their career level definitions and it got me thinking of how difficult it is to map levels from one company to another. EMC uses a P1-P7 numbering system, IBM uses a band 6-band 9 numbering system, Microsoft uses levels numbered 59-68, Amazon calls theirs levels L1-L10, etc. How is anyone suppose to make sense of this mess across companies? So I set out to do what any engineer would do: determine if there was a mapping between them.

From my experience at IBM, pre-acquisition Isilon, and now EMC I was able to do some of the mappings myself. I then reached out to friends who have worked for at least one of the three and asked how their current companies mapped to it. The following table includes the results thus-far:

IBM pre-EMC Isilon EMC Microsoft Amazon
P1 – Entry 59/60
band 6 SDE 1 P2 – Intermediate 61 L4 – college hires
band 7 SDE 2 P3 – Senior 62 L5 – mid-career
band 8 SDE 3 P4 – Principal 63 L6 – Senior
band 9 P5 – Consultant 64 L7 – Principal
STSM Staff P6 – Senior Consultant 65/66/67 – Principal L8 – Senior Principal
Distinguished Engineer Distinguished Engineer L7 – Lead /
Distinguished Engineer
68 – Partner L10 – Distinguished Engineer
Fellow Fellow Fellow

Discussion frameworks, not absolutes

The chart above is useful primarily as a framework for discussion. Even within a large company the skills for someone at given level may not map closely to others at that same level. Add cross-company evaluations into the mix and the above is, at best, a guideline.

Why is this important?

People change jobs frequently in the tech industry (no surprise given employees who stay in companies longer than 2 years get paid 50% less). My 10-year career at IBM and 4+ years (so far) at EMC makes me an outlier in a field where most of the resumes I see have people staying just 1-2 years between jobs.

The chart above can be useful when changing jobs across companies. When I left IBM to come work at Isilon this would have come in very handy. Instead of moving from my band 9 position at IBM to at least an SDE3 or Staff position at Isilon, I moved over as an SDE2 — two steps down. Next time I’ll have more data to make a solid lateral (or upward!?) transition.1

Levels !== Pay

Everyone I’ve talked to has confirmed that pay does not directly map to levels. It is common for the upper end of the pay range of level X to overlap greatly with level X+1 and maybe even border the lower end of level X+2.

What about Company X?

I’d love to get other large tech companies like Google and Facebook on here, I just haven’t found anyone willing to share their insight. If you have overlap with one of the above drop me an email and lets fill in the blanks.

1 FWIW, I currently have no intention of leaving EMC Isilon for another job.

Work dread

Today is the last day of a great 10-day vacation out to Washington DC to visit family — both Daniel’s and my own. We get on a plane tonight to head back to Seattle and I go back to work tomorrow. For the second time in my 15-year career I’m dreading it and I haven’t even sync’d my work email to see how many hundreds of emails are waiting for me.

I’ve been dreading work since last September at the end of my trip to Spain with Jonobie. And that dread has been fairly consistent ever since.

Work has consumed my life. I get up at ~4:30a — 30 minutes before my alarm goes off — and respond to emails from the contractors in India. The plan is to get to the gym by 5:30 for my hour work-out, come home, spend some time with Daniel, and head into the office by 8a. Instead, I skip the gym and spend all morning trying to get caught up with emails and code reviews. I give myself just enough time to shower and eat breakfast and then continue working on the bus ride into the office. I work non-stop, eating lunch at my desk, until 5p when I leave for home. When I get home I’m mentally exhausted and have no energy for much more than eating dinner that Daniel has lovingly cooked, doing the dishes, watching an episode of something, and crashing in bed at 9p.

This is not a life.

LinkedIn – EMC Isilon Experience

I was updating my LinkedIn profile today1 and, after finishing my new write-up for EMC Isilon, noticed that I was drastically over their character limit. I’ve pared it down but thought I’d post the whole thing here since I went to all the effort of writing it to begin with!

1 No, I am not looking for a new job. I just got an itch to update the thing.


I’ve worn many hats so far in my short time at EMC Isilon: performance tester, performance team lead, platforms test lead, and DevOps architect.

Right now I’m leading four teams who together are reinventing how we build OneFS within Isilon engineering. We’re moving to a DevOps model leveraging continuous integration for more timely, focused feedback to developers on the state of their code. I’m personally responsible for the overall architecture of this new system and ensuring that the four teams (Infrastructure as a Service, Test as as Service, Build as a Service, and the Orchestration layer) are all integrating together. I’m the single point of contact for the Engineering organization, both out to my peers and up to management, on the state of the project. I’m also a technical consultant for the Infrastructure as a Service team since I have deep knowledge about the OneFS product as well as our internal infrastructure.

Prior to being a DevOps architect, I was the test team lead for the platforms team as they undertook upgrading the FreeBSD version that OneFS is based off of. This included creating and driving test plans, working with development on how to enable testability from day one, breaking work into smaller pieces for distribution to other team members, and interviewing potential team candidates. The thing I’m most proud of is creating tooling to enable testing of code from development before it could be plugged into the existing infrastructure. This enabled testing of code from before the first code commit to catch code regressions early.

Before the platforms team I worked with the overall performance team. This included all aspects of Isilon product performance (development, test, benchmarking, and support) across all product functional areas (kernel, networking, filesystem, and protocols). During the implementation of the Endurant Cache, one highly respected developer gave me the title of Performance Test Ninja which I’ve gladly kept to this day (’cause come on — it’s a cool title!). I was a team lead and contributor on the creation of the Performance BVT — an extensible performance test suite for automatic performance evaluation to ensure no performance regressions from build-to-build. Passing the PerfBVT (as it is affectionately known) is part of the merge criteria for feature branches before they merge the main code repo. The PerfBVT has caught several major performance regressions before they were ever merged.

Throughout my time at EMC Isilon I’ve reported to the test organization which gives me free reign to have insane focus on quality. I’ve not lost my technical chops either as I’ve muddled around in the FreeBSD TCP stack when troubleshooting 10GigE performance, slogged through kernel traces, and identified code bottlenecks (before tossing the bug to development to fix).

Outside of my “day job” mentioned above, I also have a passion for recruiting, diversity, and training. I’ve given tech talks at colleges, regularly participate in interviews for positions throughout engineering, give Engineering 101 classes for new hires, I’ve been a mentor of Ada Development Academy interns, represented EMC Isilon at GeekGirlCon and the Out & Equal Career Fair, and was on the formative board of the EMC LGBT West Coast employee resource group.

Isilon Anniversary – 4 years

Sunday marks my 4-year anniversary at Isilon. It’s been a fun ride and taken me into roles that I would have never seen myself (BSD10 merge test lead, The Factory whipping boy … I mean architect). I enjoy working with some really remarkable, bright, and passionate people solving challenging problems — they make it all worthwhile. I’ve been a part of several major software releases (Mavericks, Waikiki, Jaws, and Moby) and a whole slew of our new hardware platforms (S200, X200, X400, A100, NL400, S210, X410).

I’ve also enjoyed the diversity work I’ve been able to do, including representing EMC Isilon at the Out & Equal Career Fair, manning a booth at GeekGirlCon, and working with the Ada Developer Academy.

Here’s looking forward to whatever craziness the next year has in store for me!