Accessing Ubuntu desktop UI over SSH+VNC

During this pandemic I’m working from home on my Mac laptop and accessing things on my Ubuntu 18.04-based Linux desktop in the office. For most things this is fine via SSH or sshfs, but there are times you just need access to the desktop UI to get things done.

Specifically I had a 500 MB OVA that I needed to upload to an ESXi system — both of which are in the office. I could have downloaded the OVA to my laptop over the VPN, then uploaded it back over the VPN to ESXi but that is both slow, tedious, and wasteful. Instead after a bit of googling I figured out how to get a VNC client on my Mac securely accessing my work Xwindows display and do it all local to the office:

On your desktop, install x11vnc:

sudo apt install x11vnc

On your home computer, open an SSH tunnel and start the viewer on your remote system (below as $HOSTNAME):

ssh -t -L 5900:localhost:5900 $HOSTNAME 'x11vnc -localhost -display :0'

Then start a VNC viewer on your home computer (on MacOS I recommend RealVNC) and connect to localhost:5900

Security advisory: when accessing your desktop like this your computer is unlocked and accessible by keyboard and mouse to users who wander by your desk. Granted, in a pandemic when everyone is working from home is this really a problem? Lock your computer when you’re done as if you were walking away from your desk and you’ll be fine.

Preparing your software engineering team for a pandemic

We would be naive if we didn’t consider the possibility that the Coronavirus might flare up into a full pandemic over the next few months. Here are some things you can do to ensure your engineering team keeps humming along if that happens. If the pandemic doesn’t happen at least we’ll be better prepared for next flu season.

For individual contributors

If you’re sick, stay home and take care of yourself.
Your colleagues want you to feel better and also for you to keep your germs to yourself.

Don’t come back to work until you’re fever-free for 24 hours.
If you feel like you can work, work from home rather than come back too soon and risk a relapse or sharing your illness.

Note: the CDC goes further and says to stay home until you’ve been fever- and symptom-free for 24 hours without the help of medication.

Be prepared to work from home.
Whether it’s because you’re sick, you’re taking care of a loved one who is sick, or your kid’s school is closed, be prepared to work effectively from home.

Make sure you have the tools (computer, monitor, keyboard, etc) and access (VPN, routing to AWS resources, etc) you need to do your job effectively. Don’t wait until you’re sick to figure it all out, do so now while you have the energy to tackle some bumps along the way.

Talk with your manager about your team’s WFH policies, procedures, and best practices.

Be prepared for others to work from home.
Add call-in info to all of your meetings and actually call into them. Consider ways for those working from home to participate in stand-ups and other activities (my team does daily chat-based standups, for example).

For managers

Allow & enable your employees to work from home.
Even if your engineering org prefers to have employees in the office, allow your employees more latitude to work from home if a pandemic happens. Some employees will need to stay at home and take care of their children if schools close.

Be sure to provide employees with the resources they need to work effectively from home. That might include computer hardware, VPN software, head sets, etc.

Have WFH policies, procedures, and best practices.
Ensure your employees have clear expectations for when they work from home. Be clear if you have different expectations for when employees are working from home vs working in the office such as daily status reports or check-ins. Working remotely may be challenging for some individuals who may need more structure.

Be prepared for a lot of people to work from home.
Ensure your infrastructure will handle many more people than usual working from home at the same time.

Factor potential sick time into your planning and sprints.
When doing planning, be sure to add in some buffer for people who might be out sick. You may need to take on fewer stretch goals.

And…. ?

What did I miss? What are you doing to ensure your engineering team is prepared in case Coronavirus goes full pandemic?

Further reading

Bye SFI, hello ExtraHop

After much deliberation and soul-searching I’m changing jobs. Next Monday, November 26th is my last day at Spaceflight Industries. I will then have 3 glorious weeks of vacation before I start my new job at ExtraHop as a lead on their performance team.

I gave notice a month ago but I wanted to stay at SFI to support my team through our first commercial satellite launch — a launch that was suppose to take place today but is now delayed (the challenges of planning around rocket launches was one factor in my decision to leave the aerospace industry).

I’ve learned a great deal during my 2.5 years at Spaceflight Industries. I’ve worked with some brilliant and hardworking people, whom I will miss, and together we solved some really challenging problems in ingenious ways. I appreciate that SFI was willing to take a chance on me being a manager and giving me the flexibility to explore what that looked like for me.

That said, I’m looking forward to stepping back into an individual contributor position. While I’m told I was a good people manager it didn’t feed my soul and I found it really draining. I’ve had some really great managers over the past 18 years and attempting to live up to the high standards I set for myself was exhausting. I’m not ruling out going back into it in the future, but for now I’m excited to sink my teeth into some gnarly technical problems and to sling some code with the rest of the performance team.

I’m also looking forward to working, albeit indirectly, with the esteemed Jeena Khan and her team of writers! Frankly, I’m not certain ExtraHop knows what they’ve gotten themselves into with Jeena and I working together again. The building might not be able to contain our mutual enthusiasm!

Constellation orchestration with Gemini

This is a company blog post I wrote about Gemini, the cloud-based constellation orchestration software my team and I created at Spaceflight Industries. I’m duplicating it here from the original that was posted on 2018/11/12 for posterity.

Constellation Orchestration using the Cloud

Since the launch of Pathfinder-1 two years ago, the BlackSky ground and control team has been working on Gemini, our internal name for our next-generation cloud-based constellation orchestration system. We’ve taken operator interactions with our first demonstration satellite Pathfinder-1 combined with lessons learned from our first-generation software and redesigned the system from the ground-up for fully-automated operations of our Global satellites. From the very beginning, Gemini was designed to scale up with our constellation.

Designed for fully-automated operations

The initial checkout of the satellite post-launch begins with our satellite operators. Satellite constellation operators use Gemini for manual commanding of Global satellites during launch and early operations to confirm the satellite is healthy in orbit. After checkout is complete, the operators take a step back and the satellite is handed over to Gemini automation. Gemini is responsible for orchestrating the tasking and downlink from the satellite, engaging the groundstations around the world to communicate with the satellite during contact passes, create and upload satellite mission tasking scripts, manage telemetry & health logs, and alert operators to any anomalous telemetry. The automation is designed to protect the satellite but as additional safeguard Gemini alerts operators in the event of anomalous behavior so that they can intervene if needed.

In addition, Gemini also:

  • plans images and tasks them across the entire constellation
  • orchestrates connectivity with multiple satellites around our world-wide network of groundstations
  • manages the radio chain & antenna tracking
  • propagates satellite and equipment telemetry in sub-seconds from groundstations to operator dashboards during contact passes
  • monitors the entire system in real-time and alerts on anomalies
  • provides infrastructure for our image processing pipeline, code-named Obscura internally, that does georeferencing and orthorectification and more
  • exposes web-based UIs to operators for manual satellite commanding in addition to insight into automated activities and constellation health

Cross-team development and validation

Gemini development was a collaborative effort using input from many cross-company teams to ensure that we could test the system in the same way we expected to use it while in space (as they say in aerospace: test what you fly, fly what you test). The Gemini development team worked closely with operators to design a system that provided the control and insight they needed for successful satellite operations. Our development team worked hand-in-hand with flight software and hardware AI&T teams to validate all radio, commanding, and telemetry interfaces. An agile development approach allowed operators and other stakeholders to request features and resolve issues through an iterative testing and release process.

Our validation team created multi-satellite constellations using virtual satellites — a novelty in the aerospace industry — to ensure our system scalability. They also created automated deployments and tests to run nightly against our physical test satellite (Flatsat) to validate end-to-end radio equipment functionality and full-system integration. This innovative testing showcases the robustness of our constellation automation ahead of launch and allows the cross-functional team to evaluate the space to ground system while still on Earth.

Under the hood

Gemini was built leveraging technologies and practices that, while common in many software development shops, are new to aerospace. Our microservices architecture runs on EC2 instances running CoreOS in Amazon GovCloud and in CoreOS virtual machines on top of VMware ESXi hosts in our groundstations around the world, allowing a unified architecture across these disparate environments. Microservices are coded in Python 3.6, primarily with asyncio/aiohttp, with a smattering of node.js and are deployed via Docker containers.

To handle the firehose of critical telemetry, both from the satellite as well as the groundstation systems, we propagate telemetry in real-time using Redis pubsub then store it in KairosDB/Cassandra and expose it to operators in Grafana dashboards. WebSockets are used for real-time service alerts and messages making them available nearly instantaneously to the user. Our Polymer-based operations UI allows for tight coupling between the microservice source of the data and the operator interface all while being presented together as a single cohesive interface. Using encapsulated web components allows quick deployment of new features and easy integration with third party tools.

We use the HashiCorp stack (Terraform, Consul, Vault, and Nomad) to manage our infrastructure as code, Gitlab for source management, and Pants/Concourse for builds.

Launch ready

We’re excited to put Gemini to work when the rubber meets the road with the upcoming Global launches!

2 years at Spaceflight Industries

Today is my 2-year anniversary at Spaceflight Industries.

Coincidentally today I am operating as an Engineering Lead for our 4th mission rehearsal in preparation for commanding Global-1 when it launches in a few months. In the last 18 months my team has built Gemini, a ground & control system, from the ground up (pun intended) to task a 20+ earth-imaging satellite constellation from our groundstations around the world. The system provides satellite operators with real-time telemetry on the state of the spacecraft during a contact pass.

It’s amazing to think about what we’ve accomplished since I’ve been here and I’m excited what the next several months have in store!

My questions for new direct reports

My management mantra has always been “what would I like my manager to do in this position?”. That gave rise to the following set of questions that I ask every new person who reports to me, either as a transfer or new-hire, to start off on the right foot.

  • What would you prefer your core work hours to be?
    I’m not monitoring when my reports are in and out of the office every day (far from it), but knowing if they are a morning or evening person helps me know how they work best and when to start getting worried if they don’t show up and I haven’t heard from them.
  • During those core hours, what hours would you like to have meetings?
    Are there certain days of the week or times of the day you would prefer to not have meetings?

    I view one of my primary objectives as a manager to buffer my folks from interruptions. One way I can do that is to make sure I’m scheduling meetings at times that are good for the employee. For example, if they prefer to eat lunch at 11a I’ll try my best not to schedule a meeting with them then. I also try to enforce meeting-free Thursdays to give a solid block of Maker time and enable people to work from home.
  • How often would you like to have one-on-ones?
    Setting up reoccurring 1:1s are important, as is knowing how frequently the person wants to meet. We may have a discussion if their desired frequency is the right amount, but most people know how often they want to check in with their manager.
  • How do you like to communicate? (Slack/email/in-person/phone/etc)
    I think this is one of the most important questions. Part of buffering folks from interruptions is buffering them from my interruptions too. If someone prefers email to Slack, I’ll drop them a more coherent email rather than a train-of-thought IM. If someone would rather me stop by their desk to ask something rather than send an IM (and I have a couple of folks who prefer this), I’m happy to oblige.

Thus far these questions have been well-received and knowing the answer has improved my ability to effectively manage my employees and communicate with them.

What questions do you ask your direct reports or wish your manager would ask you?

Promotion: Software Development Manager

Over the past 8 months I’ve gradually taken over responsibility for our Ground & Control segment here at Spaceflight Industries, making sure we’re on-track to support our upcoming launch. Apparently I worked myself into a new role. Today I was promoted to Software Development Manager over Ground & Control.

It’s been insanely exciting watching our next-generation ground systems come online. The team is doing some highly innovative and industry-leading work on automated satellite commanding, whole-constellation planning, ultra-low-latency telemetry propagation, distributed system monitoring & alerting, and more. We learned a ton from our first iteration that is operating Pathfinder-1 and we have significantly improved on it now that we better understand the problem space1.

I’m stoked to be a part of this team as we prepare for Global-1 and beyond!

1 Ba dum bum.

Work-life questions to ask tech recruiters

When I was looking for a job over a year ago I had a list of questions for tech recruiters about the company’s work environment, some of which seemed to catch them off-guard. I continue to refine these questions as I discover what environments I work best in.

These may or may not match things you care about, but perhaps they’ll spark some ideas on what is important to you.

What workstation hardware is provided and is that flexible?
If you’re a Mac aficionado and they stick you with a Windows box, are you going to be happy? If you are use to working with a laptop but they only provide desktops, is that OK? What if you function best with both, is that an option? How many monitors are provided and how big are they? You’re going to be spending hours and hours in front of whatever they give you, so make sure it’s something you want, they’re flexible in getting you want you want, or they’re at least OK with you bringing your own hardware.

Do you provide standing desks?
I’ve used a standing desk for 6 years now and couldn’t go back to sitting down all day. If this is something important to you, ask.

Do you have an open floor plan, cubicles, or offices?
Spaceflight is the first company I’ve ever worked in with an open floor plan and I hate it. It’s loud and disruptive. In the future this is going to be one of the factors I consider when looking for something else.

Can I access my personal email?
Shockingly, some companies block IMAP/POP3/SMTP and/or webmail sites for their employees, preventing them from using their personal email. Yes, you really have to ask this question.

Do you have a man-in-the-middle for HTTPS requests?
This question blew recruiters away. They couldn’t believe that a company would distrust their employees enough to snoop on their secure traffic for banking and other things. Except this is exactly what EMC did to their employees. All corporate-provided systems included an EMC CA. Their snooping appliance used that CA to sign certs provided to your browser every time it made an HTTPS request. For those of us in engineering who installed their Linux OS from scratch on Day 1 and didn’t have it, the web browsers would rightfully complain loudly that the certs were invalid and your traffic was being snooped on. Chrome would go so far as to refuse to connect to Google services when presented with a cert that wasn’t signed by a Google CA.

Can I bring and use my personal devices?
What is the official company policy on bringing and using your personal devices (laptops, tablets, cell phone) while at work? Can you work from the devices?

What is your work-from-home policy?
Are employees allowed to periodically work from home? Does the company provide adequate resources to make that possible?

What is the real vacation policy?
I’m way too old to start a job with just 2 weeks of vacation. Sorry, not going to happen. If the company refuses to budge, ask if they are OK with unpaid leave. On the flip side, if the company policy is “unlimited vacation”, what does this really mean in practice? Because if you give me unlimited vacation I’m likely to take a 4-6 weeks worth of vacation over the course of a year, usually in one or two day increments, while still making sure my work is getting done and my team is taken care of. If that’s not OK I need to know up front.

Where is the office located and are there existing plans to move?
Long commutes do not fit into my work-life balance and I will not work for an employer where I have to waste 2 hours of my day getting to and from work. For instance, I live in Seattle and will not take a job on the east side (that might change when the light rail gets completed, we’ll see). Knowing where the company is located is important to me. Knowing if there are existing plans to move the company is equally important.

 

These are just a small set of the questions to think about (I covered some more in my Dear Recruiter post two years ago) but don’t hesitate to ask them. We spent an exorbitant amount of our lives at work and we need to be happy there too.

1 year at Spaceflight Industries

Today, June 27th, marks my 1-year anniversary at Spaceflight Industries1 (SFI) and it’s hard to believe it’s already been a year. I was brought on to build up their validation team and was promoted to Validation Manager three months later. Spaceflight is the first company to successfully convince me to be a manager and so far it has stuck.

It’s been great to work alongside such a passionate and knowledgeable set of engineers, both software and aerospace. Getting to work with Jane and Eric again, in particular, is fantastic. I’ve had to learn a whole different set of terminology and skills being a part of New Space. IBM and EMC have absolutely nothing on aerospace’s use of obscure initialisms or their fascination with waterfall development.

I was excited to be at SFI on September 25th when we launched Pathfinder-1 and subsequently confirmed communication with it & downloaded our first images. It is thrilling to task a satellite to take a picture of the other side of the Earth and get the photo back in less than 2 hours.

The team is hard at work as we take lessons learned from Pathfinder-1 & our first-generation ground systems and build out our next-generation satellite constellation & ground systems. It’s been a fantastic year full of challenges and fun and I look forward to what lies ahead!


1 nee BlackSky — same company, just some marketing/branding changes.

SDETs in Space!

I’m having a heck of a time finding, much less hiring, SDETs to fill my open Ground Systems and Platforms SDET positions. My gut tells me that the job descriptions just aren’t exciting enough to get people’s attention. What does a “Ground Systems SDET” do? What “Platform”?

Lets see if I can’t explain them a bit better.

Talk-To-A-Satellite SDET

Ground systems refer to all of the hardware and software components here on the ground that work together to talk to a satellite in orbit. That’s everything from the software the satellite operators use to issue high-level commands, to the systems that relay that information to our ground stations around the world, to the services on the ground station that control and task the radio chain, to the services that move the antennas, to the entire telemetry pipeline back to the missions operations center allowing the operators to know the health of the satellite and ground system components.

Everything that goes to the satellite or comes from it goes through our ground systems. These systems have to work reliably to support our growing constellation.

Making sure they all work together is where the Ground Systems SDET comes in. You’re the first line of defense in making sure that all the awesome code our devs are slinging actually cling together and make a functional system. You get to play with our satellite-on-the-table (aka: Flatsat) in our staging environment to make sure what is being built works, and then see that be deployed to our production systems and task Pathfinder-1 (and soon Pathfinder-2!) in space.

If that sounds interesting and you either live or are willing to relocate to Seattle, WA, take a look at the Ground Systems SDET position and toss me your resume!

Satellite-Picture-Selling SDET

We’ve taken all of these pictures of the Earth from space, how do we sell them to people? Well, you need an intuitive interface for customers to see all of the images you have in your catalog, buy them, and task new pictures to be taken. That’s our Platform that ends up tasking the satellites in space through our ground systems.

There are a lot of factors in play when you start talking about satellite imagery. How cloudy was it when the picture was taken? What angle was the picture taken at? Where, exactly, was the picture and how does it map onto the earth? When was it taken? Is the customer allowed to see an image over this country?

And you can’t just show them a grid view of the images. You need to place those images onto an interface that makes sense, such as a map of the Earth, and oriented such that they align up correctly.

The interface needs to scale with the ever-growing number of users as well as the ever-growing number of images in the catalog. It also needs to have good access time to our customers around the globe while maintaining security restrictions on what geographies have access to what images.

Making sure all of this works is the role of the Platform SDET. As the devs craft javascript and RESTful backend code at a break-neck speed, you’re the one that ensures cohesion and functionality. Oh sure, their new gee-whiz feature looks great in demos, but how does it scale? What did they break adding that new feature? You’re one of the first to see new images from the satellite as they make their way into our catalog and enable customers to fully realize the power of our satellite constellation.

Interested? If you live in the Herndon, VA area or are willing to relocate, take a look at our Platform SDET position and apply!