SDETs in Space!

I’m having a heck of a time finding, much less hiring, SDETs to fill my open Ground Systems and Platforms SDET positions. My gut tells me that the job descriptions just aren’t exciting enough to get people’s attention. What does a “Ground Systems SDET” do? What “Platform”?

Lets see if I can’t explain them a bit better.

Talk-To-A-Satellite SDET

Ground systems refer to all of the hardware and software components here on the ground that work together to talk to a satellite in orbit. That’s everything from the software the satellite operators use to issue high-level commands, to the systems that relay that information to our ground stations around the world, to the services on the ground station that control and task the radio chain, to the services that move the antennas, to the entire telemetry pipeline back to the missions operations center allowing the operators to know the health of the satellite and ground system components.

Everything that goes to the satellite or comes from it goes through our ground systems. These systems have to work reliably to support our growing constellation.

Making sure they all work together is where the Ground Systems SDET comes in. You’re the first line of defense in making sure that all the awesome code our devs are slinging actually cling together and make a functional system. You get to play with our satellite-on-the-table (aka: Flatsat) in our staging environment to make sure what is being built works, and then see that be deployed to our production systems and task Pathfinder-1 (and soon Pathfinder-2!) in space.

If that sounds interesting and you either live or are willing to relocate to Seattle, WA, take a look at the Ground Systems SDET position and toss me your resume!

Satellite-Picture-Selling SDET

We’ve taken all of these pictures of the Earth from space, how do we sell them to people? Well, you need an intuitive interface for customers to see all of the images you have in your catalog, buy them, and task new pictures to be taken. That’s our Platform that ends up tasking the satellites in space through our ground systems.

There are a lot of factors in play when you start talking about satellite imagery. How cloudy was it when the picture was taken? What angle was the picture taken at? Where, exactly, was the picture and how does it map onto the earth? When was it taken? Is the customer allowed to see an image over this country?

And you can’t just show them a grid view of the images. You need to place those images onto an interface that makes sense, such as a map of the Earth, and oriented such that they align up correctly.

The interface needs to scale with the ever-growing number of users as well as the ever-growing number of images in the catalog. It also needs to have good access time to our customers around the globe while maintaining security restrictions on what geographies have access to what images.

Making sure all of this works is the role of the Platform SDET. As the devs craft javascript and RESTful backend code at a break-neck speed, you’re the one that ensures cohesion and functionality. Oh sure, their new gee-whiz feature looks great in demos, but how does it scale? What did they break adding that new feature? You’re one of the first to see new images from the satellite as they make their way into our catalog and enable customers to fully realize the power of our satellite constellation.

Interested? If you live in the Herndon, VA area or are willing to relocate, take a look at our Platform SDET position and apply!

Resume tips when applying for an SDET job

I’ve been a Software Development Engineer in Test (SDET) for 15 years. I’ve interviewed many people and seen many resumes. Now, as a manager with 3 open SDET positions1, I see a lot of resumes every day. Here are some tips when applying for an SDET job.

Be a tester; have test experience

I can’t tell you how many resumes I’ve seen for developers applying for an SDET position. Pretty much by definition, developers aren’t testers. That isn’t to say that developers can’t test, or testers can’t develop, but the skill set and approach of the two are very different. If you are applying for an SDET position and the words ‘test’ or ‘validate’ do not appear prominently in your resume, you’re probably applying for the wrong job.

Demonstrate test methodologies

Your resume should communicate that you understand test methodologies and that you come from a culture of test. What type of testing have you done (unit, functional, system, integration, regression, performance)? Do you understand the software development process? Call out automation that you’ve done and the bugs exposed or time saved. Tell me how much you love breaking things and representing the customer.

Have development skills

The ‘D’ in SDET is for development — you need to have some development experience. If a company is hiring SDETs it’s because they are looking at people to write and develop automated tests and tools for automated tests to use. SDETs are not hired to be manual testers or push buttons, they’re hired to write automated tests to break the heck out of things. I want to see some software development experience on a resume and know what languages you are familiar with.

Write tests or tools/frameworks?

If you are an SDET that likes to write test automation frameworks, call that out on your resume as some SDET jobs are looking for more tools- and framework- oriented people. If you like to write automated tests and actively break things, call that out on your resume.

Most of the time I’m hiring for an SDET, I’m looking for them to write automated tests, not frameworks. Both are perfectly valid skills, but make sure it’s clear where your skill set lies and be upfront with how that matches with the position.

Make your platform experience clear

Just because you are a rockstar SDET on Windows doesn’t mean you necessarily have the skills needed to be a rockstar SDET for an embedded-Linux device. Similarly, if you have extensive experience writing system integration tests for Linux distros doesn’t mean you know a hill of beans about the same for Windows 10.

Your resume should be very clear about which platforms you have experience with and which ones you’ve simply used.

Overall resume tips

These tips aren’t specific to SDET positions, but they’re worth calling out:

  • Any link you put on your resume is fair game to be viewed.
  • If you have a LinkedIn profile, assume I will search for it and look at it. I will not find you on Facebook or do a Google search for your name, but I assume your LinkedIn profile is there to present your professional image to the world and it should not contradict your resume.
  • If at all possible, provide your resume in PDF format. That’s the only way to guarantee that the way it looks on your screen is the same as it looks on mine. There can be huge differences when I open up your .docx resume on my Fedora desktop in LibreOffice with no Microsoft fonts compared to how you meant for it to be viewed on your Windows machine in Office.
  • If you’re not already friends with a technical writer, make friends with one and ask them to look over your resume in exchange for buying them lunch; their feedback will make your resume stand out from the crowd. Tech writers are amazing people and knowing them will enrich your career and your life.
  • Spell check your resume. The number of resumes with horrendously misspelled words is, frankly, sad. I don’t expect it to be perfect, but misspelling ‘validation’ is not going to impress me (and yes, that’s happened).


Fellow SDETs and test managers, what do you look for on a resume? What do you like to call out on yours?

1 This post contains my own opinions and does not reflect those of my employer, because everyone is paranoid around lawyers.

Ciao EMC Isilon, hello BlackSky

Friday, June 17th will be my last day at EMC Isilon. It’s been a fun 5.5 years but for a variety of reasons it’s time to move on to new adventures. I’ve had the pleasure of working with some amazing people on some superbly-interesting projects and I wish them all the best on the cool stuff they’re working on next. (To Infinity and beyond!)

Monday, June 27th I start work at BlackSky Global (BSG) where I’ll have the pleasure of working with Jane Vail and Eric Rybczynski again. I’m being brought on to build up their validation team1 and drive their end-to-end testing alongside Eric.

BSG began courting me last August but that was just two months into my sabbatical and I decided the sabbatical wasn’t something I could give up. I am unbelievably fortunate that they were unable to fill the position and still have a place for me now.

I’m excited to be working at my first startup and at a company solving complex problems in space, [say it with me] the final frontier.

An unexpected bonus is that BSG is only a 10-minute walk from my house down a long flight of stairs, making my morning commute a bus-free breeze. The walk back home, on the other hand, is going to be a 15-minute straight-up Stairmaster workout. Buns of steel, here I come!

1 This is in no way soliciting existing EMC (and soon, Dell) employees to join us, unless today’s date is after 2017/06/17.

Enabling teams to warp factor 10

One of my strengths is enabling teams. Seeing spots where some documentation here or a tool there will help move the team forward and increase their velocity. This is something that is hard to hire for, or call out that you do well on a resume, but is really important for building effective and efficient teams.

Being successful at it requires the ability to step back and look beyond the immediate technical problems to see where people are struggling or wasting time. It also requires good communication skills to ask the right questions and engage the right players to solve them. And, quite frankly, sometimes it involves working outside of an established process and asking for forgiveness instead of permission.

For me, this is just an extension of performance. Instead of optimizing a piece of software or hardware it’s optimizing teams.

What does enablement look like?

What does it actually mean to “enable teams”? How does this business-speak translate into the real world? How can we get a team to warp factor 10?

What follows are some examples of how I’ve enabled teams by creating tools, writing documentation, and leading teams. Keep in mind that through all of this I was officially a consulting-level validation engineer in the performance team.

Enabling your team

The EMC Isilon performance team is one of the most geographically-distributed teams we have with developers in multiple US timezones (including Hawaii) and countries (including Denmark, Italy, and France). Our master SVN server, however, is located in Seattle. Our remote developers were having a heck of a time doing SVN branch merges on their local workstations across the WAN — the WAN latency being the bottleneck. Merges could take hours or even timeout and fail altogether.

To fix this I took an older hardware client with a 10GigE network interface, threw in some spare SSDs (we are a storage company after all), and put it up in the Seattle lab for remote devs to use for their merges. This significantly decreased the time it took to do branch merges, not to mention reducing the headache of doing it. This was so successful that I put similar clients in our remote labs for other developer-related activities that take place in those locations. These remote shared workstations have since become popular with other teams inside the engineering organization as well.

Take away: Enabling teams can be a very low-level activity, it doesn’t always have to be grand and visionary. Something as simple as playing the IT guy can significantly improve the day-to-day life of your team.

Enabling your team by enabling others

The EMC Isilon OneFS operating system is built on top of FreeBSD. OneFS 8.0 upgrades the underlying FreeBSD version from 7 to 10. The development process used to do the upgrade prevented developers from testing their incremental code changes in the standard Isilon regression framework until almost code-complete and provided no way to leverage any existing test automation. A prior effort had a similar testability problem and was ultimately unsuccessful. Moving to FreeBSD 10 was important to the organization as a whole, but particularly so for the performance team as we kept hitting limitations of FreeBSD 7 when trying to address performance problems.

Seeing the success of the FreeBSD 10 effort as vital to unblocking my team’s goals I volunteered to help. I created a lightweight isolated virtual framework built on top of VMware Workstation that allowed developers to run a small set of regression tests on their branch to validate it before they merged it. Validation engineers were able to work alongside developers to add tests to this framework as code was developed. This approach enabled the validation team to test the code as it was dropped in instead of waiting until the end when the new product was capable of being run inside the standard Isilon framework.

Take away: Enabling your team may mean taking a step back, seeing how it depends on others, and assisting them directly.

Enabling your team by enabling the larger organization

In mid-2014 EMC Isilon started pushing towards a more DevOps-centric model and I jumped at the opportunity to help the organization build better products faster. For internal purposes, we have been using virtual nodes running on VMware Workstation for many, many years. These enable developers to quickly spin up nodes on their workstations, test something out, and break them down without waiting for physical hardware to become available or waiting for the hardware to reimage. The downside is that these only ran on a developer’s workstation, require beefy workstations, and limit developers to 3-node clusters.

Seeing how underutilized virtual nodes were, and yet how powerful they are for both development and test, I volunteered to lead the team to make virtual nodes more accessible. I did so because I believed I had a unique understanding of the interdependencies involved and the skills to make it successful. Management agreed and I ended up equal parts team lead, project manager, developer, and tester for the DuctTape team. Together we made virtual clusters running on centrally-hosted ESXi systems a reality. Now both developers and testers can spin up a virtual cluster of any size running almost any build at the push of a button. This capability was delivered just in time for the final testing of the FreeBSD 10 code upgrade (mentioned above) before it could be run on hardware rigs. It was later incorporated into our standard regression wheel to supplement the physical hardware rigs already in use, further increasing our test concurrency.

Take away: Enabling your team can mean taking a huge step back, looking at organization-wide limitations, evaluating your skill set against those required to address those limitations, and putting yourself out there to try.

Incremental enablement

Developing for Distributed Proofreaders has always been challenging due to the need for very specific (and antiquated) middleware and the use of CVS as source control. Developers struggled to get a working development environment set up to test their changes and then no way to effectively have multiple changes in-flight due to CVS’s poor branch support.

During my 9-month sabbatical I iteratively addressed these by:

  • moving the project to git
  • documenting recommended workflows for the project centered around user branches to have multiple changes in-flight
  • creating a development VM with all supporting middleware already installed and a base set of data populated for testing
  • conducting two git training bootcamps using the VM to get developers up to speed on how to use git in the new development model

These have already paid off and we have several new developers test-driving code changes.

I’m in the process of introducing unit tests for existing classes and establishing them as a best-practice for future classes and refactorings.

Take away: Enabling teams is an incremental process. Once you have one roadblock removed, look afresh and see what the next one is. You don’t have to solve it all at once: incremental progress is progress all the same. And don’t overlook the importance of good documentation!

Not in my job description

None of the above roles were in my “job description” as a performance validation engineer, but I saw problems that I could help fix and fixed them.

In the case of the FreeBSD merge and the DevOps work, I volunteered to take on the role which then became my job for several months. I was able to identify and articulate to management how I was well-qualified to help address the problem and that addressing the problem was how I could best serve the overall business at that time.

I don’t know of anyone whose sole job description is a “team enabler”. Instead, it is everyone’s responsibility to find those areas around them where they can help their teammates work more quickly, efficiently, and thereby increasing the velocity of the team overall.

No one works in isolation

In the modern tech world everyone works on a team of some sort — no one works in isolation. The ability to identify what is tripping up a team and taking the initiative to fix it is an important skill for an individual contributor and a hard requirement for any team lead or manager.

Cultivating the skill of team enablement increases your value to any organization. Being able to highlight it on your resume, discuss it with recruiters, and communicate it during interviews will increase your hireability.

Dear Recruiter

Dear Recruiter,
Thank you for the unsolicited message regarding employment at your company1. As you know, hiring in the tech industry is very competitive and high-performing individuals have many opportunities. To save us both time, please answer the following questions to see if your company is a good fit for me.

  1. Are you recruiting for Microsoft, Amazon, or Oracle? If so, your company is not a good fit.
  2. Would I be required to use a Windows desktop and/or develop/test software primarily for Windows? If so, you are not a good fit. Macs are great, Linux is even better.
  3. Where are your offices located? I don’t enjoy commuting, so if you’re not in Seattle, preferably downtown, you’re unlikely to be a good fit. I’m not currently interested in moving from Seattle, so if you are further afield, you are not a good fit although I might entertain working remotely for the right opportunity.
  4. What is your company’s relationship with the open source community? Are you consumers of open source projects? Do you contribute back to open source projects?
  5. Is your company on the HRC Corporate Equality Index? If so, what is your score? If you are not listed on the HRC Corporate Equality Index, please send me a copy of your corporate nondiscrimination and inclusion policies.
  6. What are you doing to enhance diversity and inclusion in your organization, specifically hiring and retention of women, LGBT, and minorities?
  7. How long is your paid maternity leave? Do you offer paid paternity leave? If so, how long?
  8. What are your vacation and sick policies?

If, after reviewing the above, you think your company would still be of interest, please contact me again with the requested information and we’ll talk further.

— Casey

1 Whether that be from finding me on LinkedIn (good job) or sending an email to my work address (really?!).

Cutting the sabbatical short

Six months ago today I left for my sabbatical (which EMC HR insists on calling an unimaginative “leave of absence”) with the intent of being gone for a full year. Then while I was on vacation in Europe, Dell announced they were buying EMC which put a kink into my plan (and not a in a good way).

The deal is expected to close sometime between May and October, and yes that’s a huge window. From the terms of my leave of absence I have a pretty strong incentive to either be back in the office or to quit before that happens.

Today I went into the office and had a day full of meetings with various managers and coworkers to get a sense of how things are going and where I would fit in should I come back. The meetings were quite useful and well worth the visit.

We’re still working out the details on when and where exactly, but it looks like I’ll probably be going back to EMC Isilon sometime in April. Hope they’re ready for me by then!

My longest vacation yet

15 years ago today I started work at IBM in Austin. Between then and now my life has revolved around my job. In fact, the longest time I’ve ever had off from work was around 12 days traveling overseas or driving across the country with Daniel. When I changed jobs from IBM to Isilon, I took zero days off in-between. Zero.

All that to say I’m very much looking forward to having some time off of work.

I’m also scared shitless.

One week from today I’ll start my sabbatical and have a year off from working. In many ways I consider this a retirement trial run: can I keep myself active, engaged, and happy without of a job? Do I have an identity outside of work?

I can’t wait to find out.