Searching for evidence of beauty you cannot see

[I]n this pandemic haunted world in which we now live, some of y’all are in dark places too, some of you for the first time. And while I don’t know what it is like to be you, I do want to let you know that it is important to realize that your reality is not the only reality, and the world is still a beautiful place and worth fighting for, and that sometimes the best you can do is to search for the evidence of the beauty you cannot see, and then rest in it until the darkness passes.

Hugh Hollowell, Life is So Beautiful #54

Accepting the new normal

I think I’ve moved into the final stage of grief: acceptance.

For 8 weeks I’ve been grieving, among many things, that my old life is over. Going to the gym at 5:30a every morning. Running with good friends on Sundays and laughing with them as we share lunch together. Going to shows at ACT and listening to the Seattle Symphony. Eating out at amazing restaurants. Hugging friends so dear to you that they’re family. Board game nights and shared dinners with loved ones.

That life is over for the foreseeable future for very valid reasons.

This is the new normal.

Now I’m focused on switching from “surviving until this is over” to “living in the mid- and post-pandemic world”. It’s hard and sad. It’s also necessary for me to drag myself out of “I’m alive out of obligation” back into “living again”.

Accessing Ubuntu desktop UI over SSH+VNC

During this pandemic I’m working from home on my Mac laptop and accessing things on my Ubuntu 18.04-based Linux desktop in the office. For most things this is fine via SSH or sshfs, but there are times you just need access to the desktop UI to get things done.

Specifically I had a 500 MB OVA that I needed to upload to an ESXi system — both of which are in the office. I could have downloaded the OVA to my laptop over the VPN, then uploaded it back over the VPN to ESXi but that is both slow, tedious, and wasteful. Instead after a bit of googling I figured out how to get a VNC client on my Mac securely accessing my work Xwindows display and do it all local to the office:

On your desktop, install x11vnc:

sudo apt install x11vnc

On your home computer, open an SSH tunnel and start the viewer on your remote system (below as $HOSTNAME):

ssh -t -L 5900:localhost:5900 $HOSTNAME 'x11vnc -localhost -display :0'

Then start a VNC viewer on your home computer (on MacOS I recommend RealVNC) and connect to localhost:5900

Security advisory: when accessing your desktop like this your computer is unlocked and accessible by keyboard and mouse to users who wander by your desk. Granted, in a pandemic when everyone is working from home is this really a problem? Lock your computer when you’re done as if you were walking away from your desk and you’ll be fine.

Granny Dot died today

My maternal grandmother Dorothy Birkelbach — Dot to her friends and Granny Dot to her grandchildren — died today. She was an amazing woman who will be missed. I am grateful that I was able to speak with her this morning before she passed to tell her that I love her — I’m confident she never doubted that.

My grandmother was quite the influence on my life. Reading, travel, laughter, adopted families, crafting, baking — I see so much of her life reflected in my own.

She was always an avid reader and I have many books that she gave me over the years. She and my grandfather liked to travel, unlike the rest of my family, and I attribute my joy of seeing other countries and cities to them.

Always quick with a smile and a laugh, she loved fiercely. When I was about three my grandmother met a woman named Wally Watson, her husband Darrel and daughter Renee, at a soaring event. My grandmother “adopted” Wally as her own, which is how I came to know Renee — we’ve been siblings ever since. My Granny was always “Granny” to Renee. If my Granny decided you were family that was it: you were family, loved through and through. My Granny also had her shit list. It took a lot to get on it; it took a lot to get off of it too!

My Granny did ceramics when I was growing up. She always made the trophies for the soaring competitions held in Littlefield: beautiful soaring seagull. She made some for her grandkids too and it stands on our mantle next to a picture of my grandfather in his sailplane. Needless to say she was always interested in my pottery. She was also a crocheter, making me numerous sets of house shoes over the years. Something my husband Daniel continues to this day.

Granny loved to bake. Chocolate sheet cake, cinnamon roles, cookies, you name it. In college, and even years after, I would receive a box of homemade cookies from her every Christmas. Dozens of different cookies. Chocolate chip, oatmeal, cowboy cookies, Marfa Light cookies.

But her chocolate sheet cake is my favorite and the most memorable. Granny always made it when I would bring friends to my grandparents’ lake house. I never saw her make it, it always just seemed to appear when we walked in the door. Years later I would get the following recipe from her:

Granny Dot’s Chocolate Cake

Bake a boxed chocolate cake mix (Duncan Hines, Betty Crocker, etc)

5 minutes before the cake is done, start the icing:
Melt and bring to boil 1 stick margarine, 4 T cocoa, and 6 T milk. Take from stove and add 1 box (16 oz; 3.5 C) powdered sugar, 1 t vanilla, and 1 cup chopped pecans. Mix well and spread over hot cake.

Imagine my surprise to discover it started with box of cake mix! But that was my Granny, not a pretentious bone in her body. I made a chocolate cake last night in her honor.

But the most cherished memory of my grandmother was back in 2009. My youngest brother was getting married and my brother and parents said that my (now ex-) husband Benjamin was not welcome at the wedding. They didn’t want us to be a “distraction”. I told my family that I would not be attending the wedding if my husband were not welcome. I had not yet come out to my grandparents, but it was clear I needed to do so before the wedding.

I called Granny Dot and told her that I was gay and why I wasn’t going to be at the wedding. I forget the details of how that conversation went, because I was a nervous wreck, but I remember it going well. I received this letter from her a few days after the wedding:

Dearest Casey,

It was so good to hear from you. The news you told me was no surprise to me – I had suspected the situation for a long time. I know you didn’t wake up one day and decide to be gay – it is an inborn thing and is natural to you but the average person views it as unnatural, because it doesn’t follow the norm. They need to stop and evaluate all the couples who co-habitate. In my opinion there is not much difference.

Of course I had looked forward to your children to love, but my love for you has never changed and never will. You are a part of me and Papa and we have always been proud of you. Just be happy and people will learn to accept the situation. I wish you every happiness.

I love you –
Granny

I love you too Granny.

Preparing your software engineering team for a pandemic

We would be naive if we didn’t consider the possibility that the Coronavirus might flare up into a full pandemic over the next few months. Here are some things you can do to ensure your engineering team keeps humming along if that happens. If the pandemic doesn’t happen at least we’ll be better prepared for next flu season.

For individual contributors

If you’re sick, stay home and take care of yourself.
Your colleagues want you to feel better and also for you to keep your germs to yourself.

Don’t come back to work until you’re fever-free for 24 hours.
If you feel like you can work, work from home rather than come back too soon and risk a relapse or sharing your illness.

Note: the CDC goes further and says to stay home until you’ve been fever- and symptom-free for 24 hours without the help of medication.

Be prepared to work from home.
Whether it’s because you’re sick, you’re taking care of a loved one who is sick, or your kid’s school is closed, be prepared to work effectively from home.

Make sure you have the tools (computer, monitor, keyboard, etc) and access (VPN, routing to AWS resources, etc) you need to do your job effectively. Don’t wait until you’re sick to figure it all out, do so now while you have the energy to tackle some bumps along the way.

Talk with your manager about your team’s WFH policies, procedures, and best practices.

Be prepared for others to work from home.
Add call-in info to all of your meetings and actually call into them. Consider ways for those working from home to participate in stand-ups and other activities (my team does daily chat-based standups, for example).

For managers

Allow & enable your employees to work from home.
Even if your engineering org prefers to have employees in the office, allow your employees more latitude to work from home if a pandemic happens. Some employees will need to stay at home and take care of their children if schools close.

Be sure to provide employees with the resources they need to work effectively from home. That might include computer hardware, VPN software, head sets, etc.

Have WFH policies, procedures, and best practices.
Ensure your employees have clear expectations for when they work from home. Be clear if you have different expectations for when employees are working from home vs working in the office such as daily status reports or check-ins. Working remotely may be challenging for some individuals who may need more structure.

Be prepared for a lot of people to work from home.
Ensure your infrastructure will handle many more people than usual working from home at the same time.

Factor potential sick time into your planning and sprints.
When doing planning, be sure to add in some buffer for people who might be out sick. You may need to take on fewer stretch goals.

And…. ?

What did I miss? What are you doing to ensure your engineering team is prepared in case Coronavirus goes full pandemic?

Further reading

Investing with your values

Since the Pulse nightclub shooting in Orlando in 2016, I’ve made a concerted effort to not invest in ETFs or mutual funds that hold stock in gun manufacturers, and to divest in funds who do. (I also buy my anti-NRA membership every year). But knowing if a given fund invests in the likes of American Outdoor Brands (AOBC) and Sturm Ruger & Co (RGR) can be difficult.

Enter gunfreefunds.org, one of several values-based websites run by As You Sow. There you can put in a stock symbol and see if the fund invests in gun manufacturers, and if so which ones.

As You Sow has other values-based websites too:

One tip is that when you look up a stock on one, scroll down to the Sustainability Scorecard and you can see the results from all their sites together, regardless of which one you are on. That’s way better than having to look up a fund in multiple places to see how it rates.

As with anything, this is just additional data to consider and should only be a part of your decision making process.

Oh, and while it should go without saying: I am not a financial planner, this is not financial advice, and stay hydrated.

Migrating Distributed Proofreaders to Unicode

When Distributed Proofreaders started in 2000, Project Gutenberg only accepted eBooks in ASCII, later flexing to ISO-8859-1. pgdp.net has always only supported ISO-8859-1 (although practically this was really Windows-1252) which we refer to simply as Latin-1. This character set was enforced not only for the eBooks themselves, but also for the user interface. While the DP codebase has long supported arbitrary character sets, in many places it was assumed these were always using 1-byte-per-character encodings.

But the world is much bigger than Latin-1 and there are millions of books in languages that can’t be represented with 1-byte-per-character encodings. Enter Unicode and UTF-8, which Project Gutenberg started accepting a while back.

There has been talk of moving pgdp.net to Unicode and UTF-8 for many years but the effort involved is daunting. At a glance:

  • Updating the DP code to support UTF-8. The code is in PHP which doesn’t support UTF-8 natively. Most PHP functions treat strings as an array of bytes regardless of the encoding.
  • Converting our hundreds of in-progress books from ISO-8859-1 to UTF-8.
  • Finding monospace proofreading fonts that support the wide range of Unicode glyphs we might need.
  • Updating documentation and guidelines.
  • Educating hundreds of volunteers on the changes.

In addition, moving to Unicode introduces the possibility that proofreaders will insert incorrect Unicode characters into the text. Imagine the case where a proofreader inserts a κ (kappa) instead of a k. Because visually they are very similar this error may end up in the final eBook. Unicode contains literally millions of characters that wouldn’t belong in our books.

There has been much hemming and hawing and discussions for probably at least a decade, but little to no progress on the development side.

Unicode, here we come

Back in February 2018 I started working on breaking down the problem, doing research on our unknowns, making lists of lists, and throwing some code against the wall and seeing what stuck.

This past November, almost 2 years later, we got to what I believe is functionally code-complete. Next June, we intend to roll out the update to convert the site over to Unicode. Between now and then we are working to finish testing the new code, address any shortcomings, and update documentation. The goal is to have the site successfully moved over to Unicode well before our 20th anniversary in October 2020.

Discoveries and Decisions

Some interesting things we’ve learned and/or decided as part of this effort.

Unicode

Rather than open up the floodgates and allow proofreaders to input any Unicode character into the proofreading interface, we’re allowing project managers to restrict what Unicode characters they want their project to use. Both the UI and the server-side normalization enforce that only those characters are used. This allows the code to support the full set of Unicode characters but reduces the possibility of invalid characters in a given project.

For our initial roll-out, we will only be allowing projects to use the Latin-1-based glyphs in Unicode. So while everything will be in Unicode, proofreaders will only be able to insert glyphs they are familiar with. This will give us some time to ensure the code is working correctly and that our our documentation is updated before allowing other glyphsets to be used.

When you start allowing people to input Unicode characters, you have to provide them an easy way to select characters not on their keyboard. Our proofreaders come to us on all kinds of different devices and keyboards. We put a lot of effort into easing how proofreaders find and use common accented characters in addition to the full array of supported characters for a given project. One of our developers created a new, extensible, character picker for our editing interface in addition to coding up javascript that converts our custom diacritical markup into the desired character.

Unicode has some real oddities that we continue to stumble across. It took me weeks to wrap my head around how the Greek polytonic oxia forms get normalized down to the monotonic tonos forms and what that meant for our books which all predate the 1980s change from polytonic to monotonic. I also can’t believe I summed up weeks of discussions and research in that one sentences.

Fonts

DP has our own proofreading font: DPCustomMono2. It is designed to help proofreaders distinguish between similar characters such as: l I 1. Not surprisingly, it only supports Latin-1 glyphs. We are still evaluating how to broaden it to support a wider set. That said, fonts render glyphs, not character sets, so the font will continue to work for projects that only use the Latin-1 character set.

We were able to find two other monospace fonts with very broad Unicode support: Noto Sans Mono and DejaVu Sans Mono. Moreover, both of these can be provided as a web font (see this blog post for the former, and FontSquirrel for the latter), ensuring that all of our proofreaders have access to a monospace Unicode-capable font. Note that a prior version of Noto Sans Mono, called Noto Mono, is deprecated and you should use the former instead.

Most browsers do sane glyph substitution. Say you have the following font-family style: ‘font-family: DPCustomMono2, Noto Sans Mono;’ and both fonts are available to your browser as web fonts. If you use that styling against text and there is a glyph to render that doesn’t exist in DPCustomMono2, the browser will render that one glyph in Noto Sans Mono and keep going rather than render a tofu. This is great news as it means we can provide one of our two sane wide-coverage Unicode monospace fonts as a fallback font-family ensuring that we will always have some monospace rendering of all characters on a page.

MySQL

Modern versions of MySQL support two different UTF-8 encodings: utf8mb3 (aka utf8) & utf8mb4. The former only supports 3-byte UTF-8 characters whereas utf8mb4, introduced in MySQL v5.5.3, supports 4-byte UTF-8 characters. We opted for utf8mb4 to get the broadest language support and help future-proof the code (utf8mb3 is now deprecated).

An early fear was that we would need to increase the size of our varchar columns to handle the larger string widths needed with UTF-8. While this was true in MySQL 4.x, in 5.x MySQL varchar sizes represents the size of the string regardless of encoding.

PHP

PHP continues to be a pain regarding UTF-8, but it wasn’t as bad as we feared. It turns out that although most of the PHP string functions operate only on byte arrays, assuming 1-byte characters most of the places where we use them that’s just fine.

For other places, we found portable-utf8 to be a great starting point. Some of the functions aren’t particularly performant and it’s incredibly annoying that so many of them call utf8_clean() every time they are used, but it was super helpful in moving in the right direction.

mb_detect_encoding() is total crap, as I mentioned in one recent blog post, but we hacked around that.

Operations that move InnoDB tables out of the system tablespace

Distributed Proofreaders has a very large InnoDB-backed table that was created many years ago on a MySQL version that only supported the system tablespace (ibdata1). We’ve since upgraded to 5.7 which supports file-per-table tablespaces and we have innodb_file_per_table=ON.

With file-per-table tablespaces enabled, some table operations will move tables from the system tablespace to their own per-file tablespace and given the size of the table in question it was important to understand which ones would cause this to happen.

My research lead me to the Online DDL Operations doc which was the key. Any operation that says it Rebuilds Table will move an InnoDB table out of the system tablespace, regardless if In Place says “Yes”.

For example, the following will keep the table where it is:

  • Creating, dropping, or renaming non-primary-key indexes
  • Renaming a column
  • Setting a column default value
  • Dropping a column default value

And these will rebuild the table and move it to its own tablespace:

  • Adding or dropping primary keys
  • Adding or dropping a column
  • Reordering columns
  • Changing column types
  • Converting a character set

The above are not an exhaustive list, see the Online DDL Operations documentation for your version of MySQL for the definitive list.

It’s important to know that OPTIMIZE TABLE will also move an InnoDB table out of the system tablespace.

If there’s an operation you want to perform that will rebuild the table but you want to keep the table in the system tablespace, you can temporarily set innodb_file_per_table=OFF (it’s a dynamic variable), do the operation, and then turn it back ON.

And for the curious, if you have a table already in its per-file tablespace and set innodb_file_per_table=OFF, making changes that will rebuild the table won’t move it to the system tablespace. It looks like you have to drop and recreate the table to do that.