This is a company blog post I wrote about Gemini, the cloud-based constellation orchestration software my team and I created at Spaceflight Industries. I’m duplicating it here from the original that was posted on 2018/11/12 for posterity.
Constellation Orchestration using the Cloud
Since the launch of Pathfinder-1 two years ago, the BlackSky ground and control team has been working on Gemini, our internal name for our next-generation cloud-based constellation orchestration system. We’ve taken operator interactions with our first demonstration satellite Pathfinder-1 combined with lessons learned from our first-generation software and redesigned the system from the ground-up for fully-automated operations of our Global satellites. From the very beginning, Gemini was designed to scale up with our constellation.
Designed for fully-automated operations
The initial checkout of the satellite post-launch begins with our satellite operators. Satellite constellation operators use Gemini for manual commanding of Global satellites during launch and early operations to confirm the satellite is healthy in orbit. After checkout is complete, the operators take a step back and the satellite is handed over to Gemini automation. Gemini is responsible for orchestrating the tasking and downlink from the satellite, engaging the groundstations around the world to communicate with the satellite during contact passes, create and upload satellite mission tasking scripts, manage telemetry & health logs, and alert operators to any anomalous telemetry. The automation is designed to protect the satellite but as additional safeguard Gemini alerts operators in the event of anomalous behavior so that they can intervene if needed.
In addition, Gemini also:
- plans images and tasks them across the entire constellation
- orchestrates connectivity with multiple satellites around our world-wide network of groundstations
- manages the radio chain & antenna tracking
- propagates satellite and equipment telemetry in sub-seconds from groundstations to operator dashboards during contact passes
- monitors the entire system in real-time and alerts on anomalies
- provides infrastructure for our image processing pipeline, code-named Obscura internally, that does georeferencing and orthorectification and more
- exposes web-based UIs to operators for manual satellite commanding in addition to insight into automated activities and constellation health
Cross-team development and validation
Gemini development was a collaborative effort using input from many cross-company teams to ensure that we could test the system in the same way we expected to use it while in space (as they say in aerospace: test what you fly, fly what you test). The Gemini development team worked closely with operators to design a system that provided the control and insight they needed for successful satellite operations. Our development team worked hand-in-hand with flight software and hardware AI&T teams to validate all radio, commanding, and telemetry interfaces. An agile development approach allowed operators and other stakeholders to request features and resolve issues through an iterative testing and release process.
Our validation team created multi-satellite constellations using virtual satellites — a novelty in the aerospace industry — to ensure our system scalability. They also created automated deployments and tests to run nightly against our physical test satellite (Flatsat) to validate end-to-end radio equipment functionality and full-system integration. This innovative testing showcases the robustness of our constellation automation ahead of launch and allows the cross-functional team to evaluate the space to ground system while still on Earth.
Under the hood
Gemini was built leveraging technologies and practices that, while common in many software development shops, are new to aerospace. Our microservices architecture runs on EC2 instances running CoreOS in Amazon GovCloud and in CoreOS virtual machines on top of VMware ESXi hosts in our groundstations around the world, allowing a unified architecture across these disparate environments. Microservices are coded in Python 3.6, primarily with asyncio/aiohttp, with a smattering of node.js and are deployed via Docker containers.
To handle the firehose of critical telemetry, both from the satellite as well as the groundstation systems, we propagate telemetry in real-time using Redis pubsub then store it in KairosDB/Cassandra and expose it to operators in Grafana dashboards. WebSockets are used for real-time service alerts and messages making them available nearly instantaneously to the user. Our Polymer-based operations UI allows for tight coupling between the microservice source of the data and the operator interface all while being presented together as a single cohesive interface. Using encapsulated web components allows quick deployment of new features and easy integration with third party tools.
We use the HashiCorp stack (Terraform, Consul, Vault, and Nomad) to manage our infrastructure as code, Gitlab for source management, and Pants/Concourse for builds.
Launch ready
We’re excited to put Gemini to work when the rubber meets the road with the upcoming Global launches!
That isn’t to say that my life is perfect. My knees and back periodically act up, generally because of something I did at the gym. My metabolism has slowed down a bit so I can no longer eat with abandon. I expect this year will finally be the year I need to get reading glasses (I can’t complain after 40 years without glasses or contacts). The stress over the last two years about the shit-show that is our country hasn’t helped either. But on the whole my problems are of the champagne variety.
Digital Ramblings 