Bye SFI, hello ExtraHop

After much deliberation and soul-searching I’m changing jobs. Next Monday, November 26th is my last day at Spaceflight Industries. I will then have 3 glorious weeks of vacation before I start my new job at ExtraHop as a lead on their performance team.

I gave notice a month ago but I wanted to stay at SFI to support my team through our first commercial satellite launch — a launch that was suppose to take place today but is now delayed (the challenges of planning around rocket launches was one factor in my decision to leave the aerospace industry).

I’ve learned a great deal during my 2.5 years at Spaceflight Industries. I’ve worked with some brilliant and hardworking people, whom I will miss, and together we solved some really challenging problems in ingenious ways. I appreciate that SFI was willing to take a chance on me being a manager and giving me the flexibility to explore what that looked like for me.

That said, I’m looking forward to stepping back into an individual contributor position. While I’m told I was a good people manager it didn’t feed my soul and I found it really draining. I’ve had some really great managers over the past 18 years and attempting to live up to the high standards I set for myself was exhausting. I’m not ruling out going back into it in the future, but for now I’m excited to sink my teeth into some gnarly technical problems and to sling some code with the rest of the performance team.

I’m also looking forward to working, albeit indirectly, with the esteemed Jeena Khan and her team of writers! Frankly, I’m not certain ExtraHop knows what they’ve gotten themselves into with Jeena and I working together again. The building might not be able to contain our mutual enthusiasm!

Constellation orchestration with Gemini

This is a company blog post I wrote about Gemini, the cloud-based constellation orchestration software my team and I created at Spaceflight Industries. I’m duplicating it here from the original that was posted on 2018/11/12 for posterity.

Constellation Orchestration using the Cloud

Since the launch of Pathfinder-1 two years ago, the BlackSky ground and control team has been working on Gemini, our internal name for our next-generation cloud-based constellation orchestration system. We’ve taken operator interactions with our first demonstration satellite Pathfinder-1 combined with lessons learned from our first-generation software and redesigned the system from the ground-up for fully-automated operations of our Global satellites. From the very beginning, Gemini was designed to scale up with our constellation.

Designed for fully-automated operations

The initial checkout of the satellite post-launch begins with our satellite operators. Satellite constellation operators use Gemini for manual commanding of Global satellites during launch and early operations to confirm the satellite is healthy in orbit. After checkout is complete, the operators take a step back and the satellite is handed over to Gemini automation. Gemini is responsible for orchestrating the tasking and downlink from the satellite, engaging the groundstations around the world to communicate with the satellite during contact passes, create and upload satellite mission tasking scripts, manage telemetry & health logs, and alert operators to any anomalous telemetry. The automation is designed to protect the satellite but as additional safeguard Gemini alerts operators in the event of anomalous behavior so that they can intervene if needed.

In addition, Gemini also:

  • plans images and tasks them across the entire constellation
  • orchestrates connectivity with multiple satellites around our world-wide network of groundstations
  • manages the radio chain & antenna tracking
  • propagates satellite and equipment telemetry in sub-seconds from groundstations to operator dashboards during contact passes
  • monitors the entire system in real-time and alerts on anomalies
  • provides infrastructure for our image processing pipeline, code-named Obscura internally, that does georeferencing and orthorectification and more
  • exposes web-based UIs to operators for manual satellite commanding in addition to insight into automated activities and constellation health

Cross-team development and validation

Gemini development was a collaborative effort using input from many cross-company teams to ensure that we could test the system in the same way we expected to use it while in space (as they say in aerospace: test what you fly, fly what you test). The Gemini development team worked closely with operators to design a system that provided the control and insight they needed for successful satellite operations. Our development team worked hand-in-hand with flight software and hardware AI&T teams to validate all radio, commanding, and telemetry interfaces. An agile development approach allowed operators and other stakeholders to request features and resolve issues through an iterative testing and release process.

Our validation team created multi-satellite constellations using virtual satellites — a novelty in the aerospace industry — to ensure our system scalability. They also created automated deployments and tests to run nightly against our physical test satellite (Flatsat) to validate end-to-end radio equipment functionality and full-system integration. This innovative testing showcases the robustness of our constellation automation ahead of launch and allows the cross-functional team to evaluate the space to ground system while still on Earth.

Under the hood

Gemini was built leveraging technologies and practices that, while common in many software development shops, are new to aerospace. Our microservices architecture runs on EC2 instances running CoreOS in Amazon GovCloud and in CoreOS virtual machines on top of VMware ESXi hosts in our groundstations around the world, allowing a unified architecture across these disparate environments. Microservices are coded in Python 3.6, primarily with asyncio/aiohttp, with a smattering of node.js and are deployed via Docker containers.

To handle the firehose of critical telemetry, both from the satellite as well as the groundstation systems, we propagate telemetry in real-time using Redis pubsub then store it in KairosDB/Cassandra and expose it to operators in Grafana dashboards. WebSockets are used for real-time service alerts and messages making them available nearly instantaneously to the user. Our Polymer-based operations UI allows for tight coupling between the microservice source of the data and the operator interface all while being presented together as a single cohesive interface. Using encapsulated web components allows quick deployment of new features and easy integration with third party tools.

We use the HashiCorp stack (Terraform, Consul, Vault, and Nomad) to manage our infrastructure as code, Gitlab for source management, and Pants/Concourse for builds.

Launch ready

We’re excited to put Gemini to work when the rubber meets the road with the upcoming Global launches!

2 years at Spaceflight Industries

Today is my 2-year anniversary at Spaceflight Industries.

Coincidentally today I am operating as an Engineering Lead for our 4th mission rehearsal in preparation for commanding Global-1 when it launches in a few months. In the last 18 months my team has built Gemini, a ground & control system, from the ground up (pun intended) to task a 20+ earth-imaging satellite constellation from our groundstations around the world. The system provides satellite operators with real-time telemetry on the state of the spacecraft during a contact pass.

It’s amazing to think about what we’ve accomplished since I’ve been here and I’m excited what the next several months have in store!

Promotion: Software Development Manager

Over the past 8 months I’ve gradually taken over responsibility for our Ground & Control segment here at Spaceflight Industries, making sure we’re on-track to support our upcoming launch. Apparently I worked myself into a new role. Today I was promoted to Software Development Manager over Ground & Control.

It’s been insanely exciting watching our next-generation ground systems come online. The team is doing some highly innovative and industry-leading work on automated satellite commanding, whole-constellation planning, ultra-low-latency telemetry propagation, distributed system monitoring & alerting, and more. We learned a ton from our first iteration that is operating Pathfinder-1 and we have significantly improved on it now that we better understand the problem space1.

I’m stoked to be a part of this team as we prepare for Global-1 and beyond!

1 Ba dum bum.

1 year at Spaceflight Industries

Today, June 27th, marks my 1-year anniversary at Spaceflight Industries1 (SFI) and it’s hard to believe it’s already been a year. I was brought on to build up their validation team and was promoted to Validation Manager three months later. Spaceflight is the first company to successfully convince me to be a manager and so far it has stuck.

It’s been great to work alongside such a passionate and knowledgeable set of engineers, both software and aerospace. Getting to work with Jane and Eric again, in particular, is fantastic. I’ve had to learn a whole different set of terminology and skills being a part of New Space. IBM and EMC have absolutely nothing on aerospace’s use of obscure initialisms or their fascination with waterfall development.

I was excited to be at SFI on September 25th when we launched Pathfinder-1 and subsequently confirmed communication with it & downloaded our first images. It is thrilling to task a satellite to take a picture of the other side of the Earth and get the photo back in less than 2 hours.

The team is hard at work as we take lessons learned from Pathfinder-1 & our first-generation ground systems and build out our next-generation satellite constellation & ground systems. It’s been a fantastic year full of challenges and fun and I look forward to what lies ahead!


1 nee BlackSky — same company, just some marketing/branding changes.

SDETs in Space!

I’m having a heck of a time finding, much less hiring, SDETs to fill my open Ground Systems and Platforms SDET positions. My gut tells me that the job descriptions just aren’t exciting enough to get people’s attention. What does a “Ground Systems SDET” do? What “Platform”?

Lets see if I can’t explain them a bit better.

Talk-To-A-Satellite SDET

Ground systems refer to all of the hardware and software components here on the ground that work together to talk to a satellite in orbit. That’s everything from the software the satellite operators use to issue high-level commands, to the systems that relay that information to our ground stations around the world, to the services on the ground station that control and task the radio chain, to the services that move the antennas, to the entire telemetry pipeline back to the missions operations center allowing the operators to know the health of the satellite and ground system components.

Everything that goes to the satellite or comes from it goes through our ground systems. These systems have to work reliably to support our growing constellation.

Making sure they all work together is where the Ground Systems SDET comes in. You’re the first line of defense in making sure that all the awesome code our devs are slinging actually cling together and make a functional system. You get to play with our satellite-on-the-table (aka: Flatsat) in our staging environment to make sure what is being built works, and then see that be deployed to our production systems and task Pathfinder-1 (and soon Pathfinder-2!) in space.

If that sounds interesting and you either live or are willing to relocate to Seattle, WA, take a look at the Ground Systems SDET position and toss me your resume!

Satellite-Picture-Selling SDET

We’ve taken all of these pictures of the Earth from space, how do we sell them to people? Well, you need an intuitive interface for customers to see all of the images you have in your catalog, buy them, and task new pictures to be taken. That’s our Platform that ends up tasking the satellites in space through our ground systems.

There are a lot of factors in play when you start talking about satellite imagery. How cloudy was it when the picture was taken? What angle was the picture taken at? Where, exactly, was the picture and how does it map onto the earth? When was it taken? Is the customer allowed to see an image over this country?

And you can’t just show them a grid view of the images. You need to place those images onto an interface that makes sense, such as a map of the Earth, and oriented such that they align up correctly.

The interface needs to scale with the ever-growing number of users as well as the ever-growing number of images in the catalog. It also needs to have good access time to our customers around the globe while maintaining security restrictions on what geographies have access to what images.

Making sure all of this works is the role of the Platform SDET. As the devs craft javascript and RESTful backend code at a break-neck speed, you’re the one that ensures cohesion and functionality. Oh sure, their new gee-whiz feature looks great in demos, but how does it scale? What did they break adding that new feature? You’re one of the first to see new images from the satellite as they make their way into our catalog and enable customers to fully realize the power of our satellite constellation.

Interested? If you live in the Herndon, VA area or are willing to relocate, take a look at our Platform SDET position and apply!

Pathfinder-1 images are here!

I’m excited to finally share some of the images Pathfinder-1 has taken in its first 53 days in orbit! We’ve had access to these internally for several weeks now and it’s super-exciting to finally share them publicly.

See the BlackSky blog post Hello Beautiful! for the full story and first set of released images. More images to come on the BlackSky twitter feed!

Next up: Pathfinder-2 launches early next year.