Ciao EMC Isilon, hello BlackSky

Friday, June 17th will be my last day at EMC Isilon. It’s been a fun 5.5 years but for a variety of reasons it’s time to move on to new adventures. I’ve had the pleasure of working with some amazing people on some superbly-interesting projects and I wish them all the best on the cool stuff they’re working on next. (To Infinity and beyond!)

Monday, June 27th I start work at BlackSky Global (BSG) where I’ll have the pleasure of working with Jane Vail and Eric Rybczynski again. I’m being brought on to build up their validation team1 and drive their end-to-end testing alongside Eric.

BSG began courting me last August but that was just two months into my sabbatical and I decided the sabbatical wasn’t something I could give up. I am unbelievably fortunate that they were unable to fill the position and still have a place for me now.

I’m excited to be working at my first startup and at a company solving complex problems in space, [say it with me] the final frontier.

An unexpected bonus is that BSG is only a 10-minute walk from my house down a long flight of stairs, making my morning commute a bus-free breeze. The walk back home, on the other hand, is going to be a 15-minute straight-up Stairmaster workout. Buns of steel, here I come!


1 This is in no way soliciting existing EMC (and soon, Dell) employees to join us, unless today’s date is after 2017/06/17.

Enabling teams to warp factor 10

One of my strengths is enabling teams. Seeing spots where some documentation here or a tool there will help move the team forward and increase their velocity. This is something that is hard to hire for, or call out that you do well on a resume, but is really important for building effective and efficient teams.

Being successful at it requires the ability to step back and look beyond the immediate technical problems to see where people are struggling or wasting time. It also requires good communication skills to ask the right questions and engage the right players to solve them. And, quite frankly, sometimes it involves working outside of an established process and asking for forgiveness instead of permission.

For me, this is just an extension of performance. Instead of optimizing a piece of software or hardware it’s optimizing teams.

What does enablement look like?

What does it actually mean to “enable teams”? How does this business-speak translate into the real world? How can we get a team to warp factor 10?

What follows are some examples of how I’ve enabled teams by creating tools, writing documentation, and leading teams. Keep in mind that through all of this I was officially a consulting-level validation engineer in the performance team.

Enabling your team

The EMC Isilon performance team is one of the most geographically-distributed teams we have with developers in multiple US timezones (including Hawaii) and countries (including Denmark, Italy, and France). Our master SVN server, however, is located in Seattle. Our remote developers were having a heck of a time doing SVN branch merges on their local workstations across the WAN — the WAN latency being the bottleneck. Merges could take hours or even timeout and fail altogether.

To fix this I took an older hardware client with a 10GigE network interface, threw in some spare SSDs (we are a storage company after all), and put it up in the Seattle lab for remote devs to use for their merges. This significantly decreased the time it took to do branch merges, not to mention reducing the headache of doing it. This was so successful that I put similar clients in our remote labs for other developer-related activities that take place in those locations. These remote shared workstations have since become popular with other teams inside the engineering organization as well.

Take away: Enabling teams can be a very low-level activity, it doesn’t always have to be grand and visionary. Something as simple as playing the IT guy can significantly improve the day-to-day life of your team.

Enabling your team by enabling others

The EMC Isilon OneFS operating system is built on top of FreeBSD. OneFS 8.0 upgrades the underlying FreeBSD version from 7 to 10. The development process used to do the upgrade prevented developers from testing their incremental code changes in the standard Isilon regression framework until almost code-complete and provided no way to leverage any existing test automation. A prior effort had a similar testability problem and was ultimately unsuccessful. Moving to FreeBSD 10 was important to the organization as a whole, but particularly so for the performance team as we kept hitting limitations of FreeBSD 7 when trying to address performance problems.

Seeing the success of the FreeBSD 10 effort as vital to unblocking my team’s goals I volunteered to help. I created a lightweight isolated virtual framework built on top of VMware Workstation that allowed developers to run a small set of regression tests on their branch to validate it before they merged it. Validation engineers were able to work alongside developers to add tests to this framework as code was developed. This approach enabled the validation team to test the code as it was dropped in instead of waiting until the end when the new product was capable of being run inside the standard Isilon framework.

Take away: Enabling your team may mean taking a step back, seeing how it depends on others, and assisting them directly.

Enabling your team by enabling the larger organization

In mid-2014 EMC Isilon started pushing towards a more DevOps-centric model and I jumped at the opportunity to help the organization build better products faster. For internal purposes, we have been using virtual nodes running on VMware Workstation for many, many years. These enable developers to quickly spin up nodes on their workstations, test something out, and break them down without waiting for physical hardware to become available or waiting for the hardware to reimage. The downside is that these only ran on a developer’s workstation, require beefy workstations, and limit developers to 3-node clusters.

Seeing how underutilized virtual nodes were, and yet how powerful they are for both development and test, I volunteered to lead the team to make virtual nodes more accessible. I did so because I believed I had a unique understanding of the interdependencies involved and the skills to make it successful. Management agreed and I ended up equal parts team lead, project manager, developer, and tester for the DuctTape team. Together we made virtual clusters running on centrally-hosted ESXi systems a reality. Now both developers and testers can spin up a virtual cluster of any size running almost any build at the push of a button. This capability was delivered just in time for the final testing of the FreeBSD 10 code upgrade (mentioned above) before it could be run on hardware rigs. It was later incorporated into our standard regression wheel to supplement the physical hardware rigs already in use, further increasing our test concurrency.

Take away: Enabling your team can mean taking a huge step back, looking at organization-wide limitations, evaluating your skill set against those required to address those limitations, and putting yourself out there to try.

Incremental enablement

Developing for Distributed Proofreaders has always been challenging due to the need for very specific (and antiquated) middleware and the use of CVS as source control. Developers struggled to get a working development environment set up to test their changes and then no way to effectively have multiple changes in-flight due to CVS’s poor branch support.

During my 9-month sabbatical I iteratively addressed these by:

  • moving the project to git
  • documenting recommended workflows for the project centered around user branches to have multiple changes in-flight
  • creating a development VM with all supporting middleware already installed and a base set of data populated for testing
  • conducting two git training bootcamps using the VM to get developers up to speed on how to use git in the new development model

These have already paid off and we have several new developers test-driving code changes.

I’m in the process of introducing unit tests for existing classes and establishing them as a best-practice for future classes and refactorings.

Take away: Enabling teams is an incremental process. Once you have one roadblock removed, look afresh and see what the next one is. You don’t have to solve it all at once: incremental progress is progress all the same. And don’t overlook the importance of good documentation!

Not in my job description

None of the above roles were in my “job description” as a performance validation engineer, but I saw problems that I could help fix and fixed them.

In the case of the FreeBSD merge and the DevOps work, I volunteered to take on the role which then became my job for several months. I was able to identify and articulate to management how I was well-qualified to help address the problem and that addressing the problem was how I could best serve the overall business at that time.

I don’t know of anyone whose sole job description is a “team enabler”. Instead, it is everyone’s responsibility to find those areas around them where they can help their teammates work more quickly, efficiently, and thereby increasing the velocity of the team overall.

No one works in isolation

In the modern tech world everyone works on a team of some sort — no one works in isolation. The ability to identify what is tripping up a team and taking the initiative to fix it is an important skill for an individual contributor and a hard requirement for any team lead or manager.

Cultivating the skill of team enablement increases your value to any organization. Being able to highlight it on your resume, discuss it with recruiters, and communicate it during interviews will increase your hireability.

Further thoughts on Dell’s acquisition of EMC

My prior public post about the EMC / Dell acquisition covered impacts to EMC shareholders and employee shareholders. This one is, again, my personal thoughts to discuss some of the broader implications. All of these observations are made from public sources and knowledge of how the tech industry works. I expect anyone familiar with the tech industry to be able to make similar observations / predictions.

Employees

For EMC (and in some respects, Dell) employees, the acquisition is going to be a mixed bag. Employees have no more details of the acquisition than the public at this point, so there’s a lot of uncertainty. So far, the message has been: it’s business as usual until the deal closes — which is at least 7 months away. I expect EMC will see higher-than-usual attrition over the next few months as folks who weren’t really looking before now leave for other opportunities. I wouldn’t be surprised if EMC starts offering some form of carrot to key players incentivizing them to stay. A lot of business obligations can come due in the next 7 months.

By many accounts Dell invested more money in R&D groups after going private. That’s good news for anyone in an engineering group. A friend also pointed out he “[thinks] it will be beneficial for engineering organizations [to be inside by a non-public company]. The less chained by the public market for short term profits and gains, the better the chances of being able to survive flat growth a quarter or two to realign priorities” — which I completely agree with.

Dell and EMC don’t have a whole lot of overlap in products, although there are a few, so there’s unlikely to be layoffs trying to reduce duplication there. There will probably be layoffs as Dell and EMC try to reduce non-engineering duplication in areas like HR, marketing, IT, sales support, and sales to some degree. Dell is going to need to reduce their ~$49 billion debt load after the acquisition (which they intend to do in 18-24 months) and it seems to me that reducing head-count is one way to achieve that. Another way of reducing their debt load would be to divest themselves of some products or product lines, either by selling them to someone else or simply stopping spending money on them by sunsetting the products and laying off the people.

Post-acquisition I would have expected Dell to put a kibosh on the use of Macs which are growing in EMC, but a friend who works for Dell said that other acquisitions which had Macs keep them alongside a Dell laptop for use in front of customers. Given how closely Dell seems tied to Microsoft, I expect its IT organization to push (require?) Windows workstations instead of Linux workstations which are the system of choice in Isilon.

That same friend at Dell says compensation is great and everything is cash or cash awards, so no stock — not a surprise since it’s a privately-owned company. They said it does appear that being private has allowed Dell to do more than play the quarterly-numbers game.

From some of the intangibles, there’s a whole lot of potential upsides and downsides. EMC is a very east-coast company who doesn’t really grok how west-coast companies work. I expect (but don’t know) that Dell being based out of Austin is more aligned with the west-coast mentality which is nice. Both EMC and Dell have a perfect 100 rating on the HRC Corporate Equality Index. Both have both LGBT and Women’s employee resource groups, among others. Dell trumps EMC in offering matching charitable donations but not on-site dry cleaning in MA like EMC — so east coast.

Recruiting

EMC probably isn’t going to be really pushing recruiting during the transition, except for backfills. I expect recruiting for EMC to be challenging until the deal closes. Typically public tech companies include some equity in their offer, either as stock options or RSUs, that vest over a few years. Equity is usually enticing because their worth is tied to the market performance of the company. If the stock goes up, the equity is worth more. This gives the illusion that hard work by an individual can improve the value of the stock.

Dell has assured that the EMC share price is largely fixed around $27 until the deal closes mid-2016. This also fixes the price of any options (which are rendered pretty much worthless if they are issued after today) and RSUs. This makes RSUs more like promises of fixed values than a promise of potential future growth rendering their use in offers less enticing than they were before the acquisition announcement.

It’s going to be hard over the next 12 months to even get people to interview with EMC. Who wants to start working at a company in the middle of a huge acquisition where there will be reorganization and “vision alignment” as soon as the deal closes?

I think overall the acquisition is going to be a positive thing, but the sooner it completes the better it is going to be for everyone.

Thoughts on Dell’s acquisition of EMC from public sources

On Monday, October 12th EMC and Dell announced that Dell was buying EMC. This will make for some interesting times for EMC shareholders and employees. The deal is made more interesting by the fact that, unlike EMC, Dell is not a publicly traded company. Michael Dell (with the help of Silver Lake) took his company private back in 2013.

There has been a lot of news about the acquisition. These are some personal, collated thoughts gathered from public sources (with citations) and in no way reflect the view of my employer (current or future) or any inside information.

Stockholders

From the official press release, EMC shareholders will get $24.05 a share in cash and $9.10 of a tracking stock tied to VMware. One could naively say this values EMC stock at $33.15, but there’s a twist. The twist is the tracking stock, an investment vehicle last seen prior to the dot-com bubble.

In theory, tracking stock gives investors a way to reflect value in another security (in this case VMware stocks owned by Dell) but generally grants no dividend or voting rights. Some analysts expect the tracking stock, which in theory will track the value of VMware stock, to trade from between 5 and 10% discount of that stock. (That link is an excellent read, BTW.) This partially explains why the stock has been trading between $27 and $28 since the acquisition announcement instead of closer to $33.

Also, note that as a private company, Dell won’t pay dividends like EMC, which is a bummer for those of us investors who liked that aspect. VMware doesn’t (yet?) pay dividends so there’s no love lost on that aspect of the tracking stock.

Employee Shareholders

Many (most?) employees are shareholders as well, either through the employee stock purchase program (ESPP) or through equity awards. What will happen to stock options and restricted stock units (RSUs) is unknown at this time.

According to Ars Technica, when Dell went private back in 2013 stock options that were underwater were cancelled. Unvested RSUs retained their vesting schedule but were paid out as cash upon vesting using the stock price when Dell was privatized. The upside is that the RSUs were essentially converted into promises of cash at a fixed value, but that’s a downside too — they aren’t going to be worth more than that fixed value, ever.

Employees & Recruiting

I have additional thoughts on how this impacts current employees and recruiting. While I believe all of them are logical conclusions that anyone familiar with the tech industry could derive, and they reveal no internal information, it’s not worth the risk of EMC legal’s wrath to post them here. So I’m posting them in a separate, locked post for my own edification.

A leave of absence from EMC Isilon – the skinny

The prior post gave a high-level overview of why I’m taking a leave of absence from EMC Isilon, but I wanted to take a moment and write down some of the details of my burn-out for myself and to share with friends.

The downward trajectory of my work-life balance started when I left my cozy position in the performance team to work on BSD10 merge effort in February of last year. When I started on that team there was no test plan and the only tester was a developer who didn’t want to be a tester but was one anyway. The BSD10 merge was said to be critical to the success of the business, yet management wasn’t willing to move any testing resources, experienced or not, over to it. Instead, they pushed us to use contractors which produced crappy code and required extreme hand-holding.1

Despite this, I rallied the test resources we had, created a test plan, worked with the developers on a viable code merge/validation plan, and created test infrastructure to enable automated code testing. I felt like I had moved a mountain. 2

If I had rolled off that effort back to the performance team I probably would have been ok. Instead, in August I put my neck out on The Factory whose mission was to change how we develop our products.

The Factory was charged with architecting whole new systems but had no high-level people with those skills on it — except me (and I’m not an experienced architect). It was tasked with coding up those systems but had no developers — except me (and I’m a tester, not a developer). The code being developed needed to fit into the existing test infrastructure, yet no one with that knowledge was on the team — except me (this one I totally own as me). When I pushed back saying we needed more developers than just me, the only people we were given were from other teams that did not want them and only one of which was a developer. When I pushed again saying that wasn’t enough, we got told to use contractors — who produced poor quality code and only increased the technical debit. All the while management was saying that this effort was critical in keeping Isilon relevant. We were pushed to increase our velocity but given no experienced resources.

I pushed very hard — designing systems, writing code, testing code, herding the team — and we were able to get DuctTape up and running in time to save the BSD10 team’s ass. We enabled them to test their code using virtual nodes since they were not yet running on hardware. Everyone who uses DuctTape loves it. The nodes are fully integrated into our test system and Just Work. I felt like I had moved a mountain again.

For the longest time I was frustrated with the work of some members of the team, until I realized that they are performing at the career level they are at. You don’t expect an intern to architect your internal deployment strategy, why should I expect Consultant-level code from Senior-level people?

And here I am burnt out. Perhaps I would have been less burn out if management had ponied up resources for these teams working on “critical” projects but not staffing them as such. Unless this trend is changed and they invest in the people on the internal infrastructure teams (both more bodies as well as some higher-level people) the entire organization is going to implode.

1 Eventually they brought Ngie back which was the saving grace of the BSD10 merge, their contributions can not be overstated.

2 The BSD10 merge was late merging into HEAD, but came in a very high quality, thanks partially to the work I did early in their effort.

A leave of absence from EMC Isilon

Starting June 15th, I will be taking a one year unpaid leave of absence from EMC Isilon.

The reason for the leave of absence is that I’m burned out. Crispy critter burned out. Although I’m not going to go into detail on exactly why, the 30k-foot version is that I held myself to too-high a standard on some projects that drastically, and negatively, impacted my work-life balance to the point where I was mentally and physically exhausted when I came home from work. Every day. For months.

I’m looking forward to mentally rebooting myself and hopefully learn some skills on how to help prevent this in the future.

That said, it’s incredibly difficult to leave EMC Isilon right now. It’s hard to leave just before the next version of OneFS1 ships. I’m super excited about the work that the entire engineering team has done for this release and would love to see it complete. It’s also hard to leave right when DuctTape is releasing some cool new functionality and significantly increased capacity.

Still, taking some time off is most certainly the right decision for my mental and physical health and I leave the projects I was working on in very capable hands.

1 According to the interwebs, the code-name of the next release is not yet public, and I’m not about to be the one to make it so!

Trying to quit

[This post was non-public when first posted.]

On Monday I gave my 4-weeks notice — it was not well received. Jonathan didn’t try to talk me out of it though. He said knowing me it wouldn’t do any good since I give things lots of thought before acting on them. He did, however, throw out the idea of a leave of absence.

As a rule, a 3-month leave of absence is the most they do which I said was a non-starter. It’s going to take them much more than 3 months to address some of the systematic issues that has me burnt out and leaving. Jonathan and HR are in the process of seeing if EMC will do a year’s leave of absence instead. We’ll see how flexible EMC wants to be.

The most flattering part of all of this is management’s view that my departure is going to be very impactful to the org from a morale perspective. I’m not sure I agree with that, but it’s flattering all the same.

I also discovered this week that 3 other well-respected and long-tenured people are leaving in the next few weeks, which makes me sad for Isilon and a bit guilty for being a 4th.