Bye SFI, hello ExtraHop

After much deliberation and soul-searching I’m changing jobs. Next Monday, November 26th is my last day at Spaceflight Industries. I will then have 3 glorious weeks of vacation before I start my new job at ExtraHop as a lead on their performance team.

I gave notice a month ago but I wanted to stay at SFI to support my team through our first commercial satellite launch — a launch that was suppose to take place today but is now delayed (the challenges of planning around rocket launches was one factor in my decision to leave the aerospace industry).

I’ve learned a great deal during my 2.5 years at Spaceflight Industries. I’ve worked with some brilliant and hardworking people, whom I will miss, and together we solved some really challenging problems in ingenious ways. I appreciate that SFI was willing to take a chance on me being a manager and giving me the flexibility to explore what that looked like for me.

That said, I’m looking forward to stepping back into an individual contributor position. While I’m told I was a good people manager it didn’t feed my soul and I found it really draining. I’ve had some really great managers over the past 18 years and attempting to live up to the high standards I set for myself was exhausting. I’m not ruling out going back into it in the future, but for now I’m excited to sink my teeth into some gnarly technical problems and to sling some code with the rest of the performance team.

I’m also looking forward to working, albeit indirectly, with the esteemed Jeena Khan and her team of writers! Frankly, I’m not certain ExtraHop knows what they’ve gotten themselves into with Jeena and I working together again. The building might not be able to contain our mutual enthusiasm!

Constellation orchestration with Gemini

This is a company blog post I wrote about Gemini, the cloud-based constellation orchestration software my team and I created at Spaceflight Industries. I’m duplicating it here from the original that was posted on 2018/11/12 for posterity.

Constellation Orchestration using the Cloud

Since the launch of Pathfinder-1 two years ago, the BlackSky ground and control team has been working on Gemini, our internal name for our next-generation cloud-based constellation orchestration system. We’ve taken operator interactions with our first demonstration satellite Pathfinder-1 combined with lessons learned from our first-generation software and redesigned the system from the ground-up for fully-automated operations of our Global satellites. From the very beginning, Gemini was designed to scale up with our constellation.

Designed for fully-automated operations

The initial checkout of the satellite post-launch begins with our satellite operators. Satellite constellation operators use Gemini for manual commanding of Global satellites during launch and early operations to confirm the satellite is healthy in orbit. After checkout is complete, the operators take a step back and the satellite is handed over to Gemini automation. Gemini is responsible for orchestrating the tasking and downlink from the satellite, engaging the groundstations around the world to communicate with the satellite during contact passes, create and upload satellite mission tasking scripts, manage telemetry & health logs, and alert operators to any anomalous telemetry. The automation is designed to protect the satellite but as additional safeguard Gemini alerts operators in the event of anomalous behavior so that they can intervene if needed.

In addition, Gemini also:

  • plans images and tasks them across the entire constellation
  • orchestrates connectivity with multiple satellites around our world-wide network of groundstations
  • manages the radio chain & antenna tracking
  • propagates satellite and equipment telemetry in sub-seconds from groundstations to operator dashboards during contact passes
  • monitors the entire system in real-time and alerts on anomalies
  • provides infrastructure for our image processing pipeline, code-named Obscura internally, that does georeferencing and orthorectification and more
  • exposes web-based UIs to operators for manual satellite commanding in addition to insight into automated activities and constellation health

Cross-team development and validation

Gemini development was a collaborative effort using input from many cross-company teams to ensure that we could test the system in the same way we expected to use it while in space (as they say in aerospace: test what you fly, fly what you test). The Gemini development team worked closely with operators to design a system that provided the control and insight they needed for successful satellite operations. Our development team worked hand-in-hand with flight software and hardware AI&T teams to validate all radio, commanding, and telemetry interfaces. An agile development approach allowed operators and other stakeholders to request features and resolve issues through an iterative testing and release process.

Our validation team created multi-satellite constellations using virtual satellites — a novelty in the aerospace industry — to ensure our system scalability. They also created automated deployments and tests to run nightly against our physical test satellite (Flatsat) to validate end-to-end radio equipment functionality and full-system integration. This innovative testing showcases the robustness of our constellation automation ahead of launch and allows the cross-functional team to evaluate the space to ground system while still on Earth.

Under the hood

Gemini was built leveraging technologies and practices that, while common in many software development shops, are new to aerospace. Our microservices architecture runs on EC2 instances running CoreOS in Amazon GovCloud and in CoreOS virtual machines on top of VMware ESXi hosts in our groundstations around the world, allowing a unified architecture across these disparate environments. Microservices are coded in Python 3.6, primarily with asyncio/aiohttp, with a smattering of node.js and are deployed via Docker containers.

To handle the firehose of critical telemetry, both from the satellite as well as the groundstation systems, we propagate telemetry in real-time using Redis pubsub then store it in KairosDB/Cassandra and expose it to operators in Grafana dashboards. WebSockets are used for real-time service alerts and messages making them available nearly instantaneously to the user. Our Polymer-based operations UI allows for tight coupling between the microservice source of the data and the operator interface all while being presented together as a single cohesive interface. Using encapsulated web components allows quick deployment of new features and easy integration with third party tools.

We use the HashiCorp stack (Terraform, Consul, Vault, and Nomad) to manage our infrastructure as code, Gitlab for source management, and Pants/Concourse for builds.

Launch ready

We’re excited to put Gemini to work when the rubber meets the road with the upcoming Global launches!

2 years at Spaceflight Industries

Today is my 2-year anniversary at Spaceflight Industries.

Coincidentally today I am operating as an Engineering Lead for our 4th mission rehearsal in preparation for commanding Global-1 when it launches in a few months. In the last 18 months my team has built Gemini, a ground & control system, from the ground up (pun intended) to task a 20+ earth-imaging satellite constellation from our groundstations around the world. The system provides satellite operators with real-time telemetry on the state of the spacecraft during a contact pass.

It’s amazing to think about what we’ve accomplished since I’ve been here and I’m excited what the next several months have in store!

My questions for new direct reports

My management mantra has always been “what would I like my manager to do in this position?”. That gave rise to the following set of questions that I ask every new person who reports to me, either as a transfer or new-hire, to start off on the right foot.

  • What would you prefer your core work hours to be?
    I’m not monitoring when my reports are in and out of the office every day (far from it), but knowing if they are a morning or evening person helps me know how they work best and when to start getting worried if they don’t show up and I haven’t heard from them.
  • During those core hours, what hours would you like to have meetings?
    Are there certain days of the week or times of the day you would prefer to not have meetings?

    I view one of my primary objectives as a manager to buffer my folks from interruptions. One way I can do that is to make sure I’m scheduling meetings at times that are good for the employee. For example, if they prefer to eat lunch at 11a I’ll try my best not to schedule a meeting with them then. I also try to enforce meeting-free Thursdays to give a solid block of Maker time and enable people to work from home.
  • How often would you like to have one-on-ones?
    Setting up reoccurring 1:1s are important, as is knowing how frequently the person wants to meet. We may have a discussion if their desired frequency is the right amount, but most people know how often they want to check in with their manager.
  • How do you like to communicate? (Slack/email/in-person/phone/etc)
    I think this is one of the most important questions. Part of buffering folks from interruptions is buffering them from my interruptions too. If someone prefers email to Slack, I’ll drop them a more coherent email rather than a train-of-thought IM. If someone would rather me stop by their desk to ask something rather than send an IM (and I have a couple of folks who prefer this), I’m happy to oblige.

Thus far these questions have been well-received and knowing the answer has improved my ability to effectively manage my employees and communicate with them.

What questions do you ask your direct reports or wish your manager would ask you?

Promotion: Software Development Manager

Over the past 8 months I’ve gradually taken over responsibility for our Ground & Control segment here at Spaceflight Industries, making sure we’re on-track to support our upcoming launch. Apparently I worked myself into a new role. Today I was promoted to Software Development Manager over Ground & Control.

It’s been insanely exciting watching our next-generation ground systems come online. The team is doing some highly innovative and industry-leading work on automated satellite commanding, whole-constellation planning, ultra-low-latency telemetry propagation, distributed system monitoring & alerting, and more. We learned a ton from our first iteration that is operating Pathfinder-1 and we have significantly improved on it now that we better understand the problem space1.

I’m stoked to be a part of this team as we prepare for Global-1 and beyond!

1 Ba dum bum.

Work-life questions to ask tech recruiters

When I was looking for a job over a year ago I had a list of questions for tech recruiters about the company’s work environment, some of which seemed to catch them off-guard. I continue to refine these questions as I discover what environments I work best in.

These may or may not match things you care about, but perhaps they’ll spark some ideas on what is important to you.

What workstation hardware is provided and is that flexible?
If you’re a Mac aficionado and they stick you with a Windows box, are you going to be happy? If you are use to working with a laptop but they only provide desktops, is that OK? What if you function best with both, is that an option? How many monitors are provided and how big are they? You’re going to be spending hours and hours in front of whatever they give you, so make sure it’s something you want, they’re flexible in getting you want you want, or they’re at least OK with you bringing your own hardware.

Do you provide standing desks?
I’ve used a standing desk for 6 years now and couldn’t go back to sitting down all day. If this is something important to you, ask.

Do you have an open floor plan, cubicles, or offices?
Spaceflight is the first company I’ve ever worked in with an open floor plan and I hate it. It’s loud and disruptive. In the future this is going to be one of the factors I consider when looking for something else.

Can I access my personal email?
Shockingly, some companies block IMAP/POP3/SMTP and/or webmail sites for their employees, preventing them from using their personal email. Yes, you really have to ask this question.

Do you have a man-in-the-middle for HTTPS requests?
This question blew recruiters away. They couldn’t believe that a company would distrust their employees enough to snoop on their secure traffic for banking and other things. Except this is exactly what EMC did to their employees. All corporate-provided systems included an EMC CA. Their snooping appliance used that CA to sign certs provided to your browser every time it made an HTTPS request. For those of us in engineering who installed their Linux OS from scratch on Day 1 and didn’t have it, the web browsers would rightfully complain loudly that the certs were invalid and your traffic was being snooped on. Chrome would go so far as to refuse to connect to Google services when presented with a cert that wasn’t signed by a Google CA.

Can I bring and use my personal devices?
What is the official company policy on bringing and using your personal devices (laptops, tablets, cell phone) while at work? Can you work from the devices?

What is your work-from-home policy?
Are employees allowed to periodically work from home? Does the company provide adequate resources to make that possible?

What is the real vacation policy?
I’m way too old to start a job with just 2 weeks of vacation. Sorry, not going to happen. If the company refuses to budge, ask if they are OK with unpaid leave. On the flip side, if the company policy is “unlimited vacation”, what does this really mean in practice? Because if you give me unlimited vacation I’m likely to take a 4-6 weeks worth of vacation over the course of a year, usually in one or two day increments, while still making sure my work is getting done and my team is taken care of. If that’s not OK I need to know up front.

Where is the office located and are there existing plans to move?
Long commutes do not fit into my work-life balance and I will not work for an employer where I have to waste 2 hours of my day getting to and from work. For instance, I live in Seattle and will not take a job on the east side (that might change when the light rail gets completed, we’ll see). Knowing where the company is located is important to me. Knowing if there are existing plans to move the company is equally important.

 

These are just a small set of the questions to think about (I covered some more in my Dear Recruiter post two years ago) but don’t hesitate to ask them. We spent an exorbitant amount of our lives at work and we need to be happy there too.

1 year at Spaceflight Industries

Today, June 27th, marks my 1-year anniversary at Spaceflight Industries1 (SFI) and it’s hard to believe it’s already been a year. I was brought on to build up their validation team and was promoted to Validation Manager three months later. Spaceflight is the first company to successfully convince me to be a manager and so far it has stuck.

It’s been great to work alongside such a passionate and knowledgeable set of engineers, both software and aerospace. Getting to work with Jane and Eric again, in particular, is fantastic. I’ve had to learn a whole different set of terminology and skills being a part of New Space. IBM and EMC have absolutely nothing on aerospace’s use of obscure initialisms or their fascination with waterfall development.

I was excited to be at SFI on September 25th when we launched Pathfinder-1 and subsequently confirmed communication with it & downloaded our first images. It is thrilling to task a satellite to take a picture of the other side of the Earth and get the photo back in less than 2 hours.

The team is hard at work as we take lessons learned from Pathfinder-1 & our first-generation ground systems and build out our next-generation satellite constellation & ground systems. It’s been a fantastic year full of challenges and fun and I look forward to what lies ahead!


1 nee BlackSky — same company, just some marketing/branding changes.