python, bytecode, and read-only containers

Upon first access, python compiles .py code into bytecode and stores it in .pyc files. Subsequent uses of those python sources are read from the .pyc files without needing to re-compile. This make startup time, but not runtime, faster.

But what about read-only filesystems? If python is running on a read-only filesystem no .pyc files are written and every use of a .py file involves compiling the code afresh. Everything works, the startup time of a script is just a little slower.

Read-only filesystems within a container are a security best practice in production environments. Often in kubernetes deployment manifests you might see something like:

securityContext:
readOnlyRootFilesystem: true

And if you’re running python only once within the container, the container has a little bit of overhead at startup as it compiles into bytecode and then everything is in memory and off it goes. But if you’re running python multiple times, or want to make even the single run start faster, we can pre-compile the code when we build the container with the built-in python compileall library.

# Compile code on sys.path (includes system packages and CWD):
RUN python -c "import compileall; compileall.compile_path(maxlevels=10)"

# Compile code in a directory:
RUN python -m compileall /path/to/code

This moves the compilation overhead to the container build where it happens once, and out of the startup.

Thanks to Itamar Turner-Trauring at https://pythonspeed.com/ for their excellent Production-ready Docker packaging for Python slide deck with this gem.

poetry auth via .netrc

poetry, the python package manager, provides several ways of authenticating against a repository. What isn’t explicitly documented, because it’s an implicit dependency, is that poetry can also use the ~/.netrc file for authentication when fetching packages.

poetry uses requests under the covers, and requests falls back to the ~/.netrc file. This is the same fallback method for pip for the same reason.

There are several (probably bad) reasons why someone would want to do this vs one of the explicit methods given by poetry. One that comes to mind is needing to install python packages from a private repository from inside a docker container by simply volume mapping the host’s ~/.netrc file to have poetry use the right creds.

This approach probably won’t work when publishing packages — caveat emptor.

While I’m not suggesting that this is a best practice, it’s good to know that it’s an available method in some extreme edge cases.

Accessing Ubuntu desktop UI over SSH+VNC

During this pandemic I’m working from home on my Mac laptop and accessing things on my Ubuntu 18.04-based Linux desktop in the office. For most things this is fine via SSH or sshfs, but there are times you just need access to the desktop UI to get things done.

Specifically I had a 500 MB OVA that I needed to upload to an ESXi system — both of which are in the office. I could have downloaded the OVA to my laptop over the VPN, then uploaded it back over the VPN to ESXi but that is both slow, tedious, and wasteful. Instead after a bit of googling I figured out how to get a VNC client on my Mac securely accessing my work Xwindows display and do it all local to the office:

On your desktop, install x11vnc:

sudo apt install x11vnc

On your home computer, open an SSH tunnel and start the viewer on your remote system (below as $HOSTNAME):

ssh -t -L 5900:localhost:5900 $HOSTNAME 'x11vnc -localhost -display :0'

Then start a VNC viewer on your home computer (on MacOS I recommend RealVNC) and connect to localhost:5900

Security advisory: when accessing your desktop like this your computer is unlocked and accessible by keyboard and mouse to users who wander by your desk. Granted, in a pandemic when everyone is working from home is this really a problem? Lock your computer when you’re done as if you were walking away from your desk and you’ll be fine.

Creating aspell dictionary packages for Ubuntu

There are many aspell dictionary packages available for Ubuntu, but not all of them. If you’re a somewhat esoteric project like Distributed Proofreaders, you may discover that you need things like the Latin aspell dictionary (aspell-la) which I can’t seem to find packaged anywhere.

Installing from source

It’s super easy and perfectly possible to install any of the aspell dictionaries directly. Just fetch the file, configure, make, and make install and you’re golden:

wget https://ftp.gnu.org/gnu/aspell/dict/la/aspell6-la-20020503-0.tar.bz2
tar xvfj aspell6-la-20020503-0.tar.bz2
cd aspell6-la-20020503-0
./configure
make
make install

The quick and dirty works but for systems maintained by multiple people it’s a recipe for disaster without a lot of documentation. How will someone remember that this needs to be done again for the next server upgrade or server migration? In these cases it’s usually best to create a system package and install the package.

Building & installing a package

Building a package for Ubuntu / Debian can be mind-boggling complicated when all you want to do is package up a few files to lay down on the filesystem. Luckily for aspell dictionaries we can easily borrow the template used by the aspell-en package.

Start by finding and downloading the aspell dictionary that you want to install from the list available and extracting it.

wget https://ftp.gnu.org/gnu/aspell/dict/la/aspell6-la-20020503-0.tar.bz2
tar xvfj aspell6-la-20020503-0.tar.bz2

Configure and build it to create the .rws file:

cd aspell6-la-20020503-0
./configure
make

Now head over to the aspell-en package on LaunchPad, to find and download the aspell-en_*.debian.tar.xz file from the Ubuntu version that most closely matches your own, then extract it into the the dictionary directory. This is the source file for the debian/ control directory used to build the aspell-en package, which we’ll use as a template for our own.

# from within aspell6-la-20020503-0/
wget https://launchpad.net/ubuntu/+archive/primary/+files/aspell-en_2017.08.24-0-0.1.debian.tar.xz
tar xvfJ aspell-en_2017.08.24-0-0.1.debian.tar.xz

This contains several files that we don’t need for our simple dictionary, so we can clean things up a bit. Keep in mind that we’re not creating a dictionary for distribution, just for ourselves, so this doesn’t have to be perfect.

cd debian
rm aspell-en.info-aspell changelog copyright extrawords.txt
cp ../COPYING copyright

You’ll need to update some of the files to reference your language, most of these are fairly straightforward:

  • control – Update references to aspell-en to your aspell dictionary; also update Maintainer and Description. You might need to change the debhelper version to whatever is installed on your system (Ubuntu 16.04 uses v9 not v10). If you change this, you should change it in compat too.
  • watch – Update the last line to point to where you got your aspell dictionary from — you probably just need to change the two instances of ‘en’ to your language’s code.

Three files require a little more finessing: installrules, and source/format.

The install file specifies which files should be copied into the package for installation. For reasons that I, frankly, just don’t understand, we need to specify that the .rws file needs to be installed. Your install file should look like this:

*.multi         usr/lib/aspell
*.alias         usr/lib/aspell
*.dat           usr/lib/aspell
*.rws           var/lib/aspell

The rules files is a makefile that does all of the heavy lifting for building the package. The version for aspell-en includes bits that we don’t care about, namely everything related to docs and extrawords, we can remove those and update the DICT_LANG which leaves us with:

#!/usr/bin/make -f

include /usr/share/cdbs/1/rules/debhelper.mk

DICT_LANG := la

DEB_DH_MD5SUMS_ARGS += -Xvar/lib/aspell

install/aspell-$(DICT_LANG)::
        for f in `LC_ALL=C ls *.cwl`; do \
            gzip -9 -n -c "$$f" > "$(DEB_DESTDIR)/usr/share/aspell/"$$f".gz"; \
            WL=`echo $$f | sed 's/\.cwl$$//'`; \
            touch "$(DEB_DESTDIR)/var/lib/aspell/$$WL.rws"; \
            dh_link "var/lib/aspell/$$WL.rws" "usr/lib/aspell/$$WL.rws"; \
            echo "$$WL" >> "$(DEB_DESTDIR)/usr/share/aspell/$(DICT_LANG).contents"; \
        done

        touch $(DEB_DESTDIR)/var/lib/aspell/$(DICT_LANG).compat

        installdeb-aspell

Note that the 8-space indents above should be tabs in your version — this is a makefile!

The final thing to do is change source/format to say we want to use the 1.0 version:

1.0

The last thing to do is to create the changelog file using dch. This file is used by the packager to determine the name and version of the package file. To keep things simple, I recommend sticking with the version from the source file itself, even if that differs from the normal Debian version format.

# from within aspell6-la-20020503-0/
dch --create -v 20020503-0 --package aspell-la

Now all that’s left is building the package:

# from within aspell6-la-20020503-0/
debuild -us -uc

If successful, this will put a aspell-la_20020503-0_all.deb file in the parent directory.

$ ls -1
aspell-la_20020503-0.dsc
aspell-la_20020503-0.tar.gz
aspell-la_20020503-0_all.deb
aspell-la_20020503-0_amd64.build
aspell-la_20020503-0_amd64.changes
aspell6-la-20020503-0
aspell6-la-20020503-0.tar.bz2

You can now install this via:

sudo apt install ./aspell-la_20020503-0_all.deb

Note, the ./ is required, otherwise it will look in the package catalog instead of on disk for the package.

You can test that your new dictionary works via:

$ echo hello | aspell list --lang=la

If that returns with “hello” as misspelled word, it worked. If you have problems, you can remove the package (sudo apt remove aspell-la), futz with some of the files, and try rebuilding it again. Things to watch out for are ensuring you’ve configured and make’d the package and that your changes to the install and rules files are correct.

Installing yaz for PHP on Ubuntu

tl;dr

Here’s how to install yaz on Ubuntu 20.04 with PHP 7.4:

sudo apt install yaz libyaz-dev php-dev php-pear
sudo pecl install yaz

The libyaz-dev package is the important, and oft-overlooked part.

Then add the following line to /etc/php/7.4/apache2/php.ini:

extension=yaz.so

And restart apache:

sudo systemctl restart apache2

Original post from 4 years ago follows.

Numerous sites on the internet have answered the basic question of “how do I install yaz for PHP on Ubuntu”. Which basically boils down to some flavor of:

PHP 5.x

sudo apt-get install yaz
sudo apt-get install pecl      # Ubuntu pre-16.04
sudo apt-get install php-pear  # Ubuntu 16.04 and later
sudo pecl install yaz

Then add the following line to /etc/php5/apache2/php.ini:

extension=yaz.so

PHP 7.0

sudo apt-get install yaz
sudo apt-get install php7.0-dev php7.0-pear
# might just be php-dev and php-pear on your OS (eg: Ubuntu 16.04)
sudo pecl install yaz

Then add the following line to /etc/php/7.0/apache2/php.ini:

extension=yaz.so

But wait, that fails

Sadly, the pecl install will fail with the error:

checking for yaz-config... NONE
configure: error: YAZ not found (missing NONE)
ERROR: `/tmp/pear/temp/yaz/configure --with-yaz' failed

All the search results for this error solve it by downloading the yaz source code and compiling and installing it outside the package manager, which is non-ideal.

The missing piece is that yaz-config is included with the libyaz4-dev package:

sudo apt-get install libyaz4-dev

Interestingly, this yaz install blog post does explicitly calls out the need for the -dev packages, but doesn’t include the error when you don’t have it. Hopefully this blog post will tie the two bits together for future people perplexed by this.

Updates:

  • 2018-06-03: include PHP 7.0 instructions for Ubuntu 16.04.
  • 2020-12-05: include PHP 7.4 instructions for Ubuntu 20.04.

Decreasing bitmap resolution when exporting PDFs in Inkscape

When exporting a document with bitmap images in it, Inkscape won’t downsample them to a lower resolution, despite the “Resolution for rasterization (dpi)” setting in the PDF export dialog. That export dialog setting only applies to bitmaps that have had filters applied against them. See bug 246677.

4 years ago I figured out how to work around this and today I had to relearn it. So as to not repeat this in another 4 years, I’m documenting this here for posterity.

The workaround is to apply an identity filter to each bitmap before saving it. An identity filter is one that doesn’t do anything, but because it is a filter it forces Inkscape to do the downsampling upon export. Because an identity filter doesn’t actually do anything, there isn’t one available in the Filters menu, but we can create one.

To create a reusable identity filter:

  1. Open a new Inkscape file
  2. From the Filters menu, select Filter Editor…
  3. In the Filter Editor pane, click the New button. This will add the filter “filter1” to the Filter list
  4. Double click “filter1” and rename it “Identity”
  5. To the right of the Add Effect: button there is a drop-down. Change it to Color Matrix, and hit the Add Effect: button
  6. Save the document as “Identity filter.svg”
  7. Now put the file where Inkscape can find it:
    • Linux/OS X: ~/.config/inkscape/filters
    • Windows XP: C:\Documents and Settings\[USERNAME]\Application Data\Inkscape\filters
    • Windows Vista and later: C:\Users\[USERNAME]\AppData\Roaming\Inkscape\filters
  8. The ‘filters’ directory may not exist, in which case just create it.
  9. Close and restart Inkscape
  10. This should add the Personal submenu under the Filters menu which should include our Identity filter.

To use the identity filter:

  1. In your Inkscape file, select the bitmaps you want to downsample. Be sure not to select any vector images or text since you don’t want to rasterize that.
  2. From the Filters menu, select Personal > Identity
  3. Now when you save the document as a PDF, adjust the “Resolution for rasterization (dpi)” setting. 90 is a decent number for documents that are being viewed on a computer screen. 150 is probably the smallest you want to go for anything that is being printed.

O_SYNC, where has thy FILE_SYNC gone?

This morning I came across an interesting difference between RHEL 5 and RHEL 6 (and of course their CentOS brethren).

In the RHEL 5 kernel, when an application writes to a file opened with O_SYNC over NFS, those writes come across the wire as FILE_SYNC writes — ie: stable.

In the RHEL 6 kernel, when an application writes to a file opened with O_SYNC over NFS, those writes come across as UNSTABLE immediately followed by a COMMIT. To get the writes to come across the wire as FILE_SYNC, you have to use the O_DIRECT flag (either by itself or in addition to the O_SYNC flag).

This is a rather significant change and has storage-system impacts. Good for both users and implementers to be aware of.

Update: According to this kernel bug the RHEL 6 behavior listed above happens in kernels 2.6.32 to 2.6.38-rc3. Presumably it is fixed (ie: reverted to the RHEL 5 behavior) after 2.6.38-rc3, although I don’t know if its been picked up in the RHEL kernel.

Evolution of Gnome 3 to a flavor of Cinnamon

I want to be on a more recent version of Linux on my desktop here at the office. My Fedora 14 install is getting long in the tooth — particularly in Evolution. In fact, running a more recent version of Evolution is one of my primary reasons of wanting to upgrade: I’m hoping they’ve polished the Exchange integration a bit more.

The problem with Fedora 15 and 16 is the introduction of Gnome 3 and the Gnome Shell, where the Gnome developers decided to dumb-down the shell even further than they did back in Gnome 2. I don’t want it. Lots of people don’t want it.

But that’s ok, because it’s Linux (unlike Microsoft users who are stuck with whatever Redmond gives them — better hope you like Metro!). Cinnamon offers all of the great infrastructure that Gnome 3 provides with the Gnome 2-like interface that people actually like.

So off I go installing Fedora 16 within a VM only to discover that Gnome 3 requires hardware acceleration and VMware doesn’t support hardware acceleration for Linux guests. This means I can’t test out Cinnamon, or even Gnome 3, in a VM before jumping ship.

And so this endeavor comes to a close almost before it got started. With luck maybe VM will start supporting 3D acceleration in Linux sometime soon ’cause I want to play with Gnome 3 and Cinnamon before taking the plunge.

Amazon MP3 Downloader for Fedora 14? Nah, go clamz

I remember futzing for hours a year ago trying to get the Amazon MP3 Downloader (amazonmp3) working on my Fedora 12 (or 13?) instance. I eventually got it working but it’s a pain because the Fedora 11 binary Amazon provides requires a libboost version that isn’t provided with modern Fedora releases.

Now, however, there’s an opensource implementation: clamz. It’s available in most repositories (Fedora 14 for instance). Opening an .amz file will start clamz and download the music to $HOMEDIR/Music by default (you can customize this and more by editing $HOMEDIR/.clamz/config). In other words: it just works.

Getting seahorse to recognize existing SSH keys

I had a doozie of a time getting seahorse (aka: Gnome’s “Passwords and Encryption Keys” program) to recognize my personal SSH keys. The start of the problem was that I’d SSHd my private key (id_rsa) over to the box, but not the public key (id_rsa.pub) which seahorse needs. Not a problem, ssh-keygen -e convienently outputted the public key to the screen which I shoved into id_rsa.pub. The problem was that the key was in the wrong format for seahorse. It looked like this

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "2048-bit RSA, converted by cpeel from "
AAAAB3NzaC1yc2EAAAABIwAAAQEApTseuoEnUBPIQSLWjOnXY6m4idRLJ+WE49/ov0VA5/
S8+fRHosP7WrBnrja8GLqd9jiG3HOSoFT4BgQ1kBJvyKLg88Kyt5V83dmhWhr2TzWHVeo7
gtOmN0pNT7Xl3dxChLvM1LoAolif9PXLTs81CFwUTCWd+DPMa4SX7Ti8MUscy6YNkyB74A
lsbLM/0JEvBSZd+72ZJlG1laKlt8adcp2EXngnqQn0r9d5ws5kF1J3BbkKtKClssuBSsFR
Bckn7xlnXlO8o6lsq/j9cwpGYAoVy+gE/ZaVNUwG4wvZ9hlqY1sWqo9r9R9KjyVkIdXq0V
IxYCudxLYyEtf30pKrGw==
---- END SSH2 PUBLIC KEY ----

instead of looking like

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApTseuoEnUBPIQSLWjOnXY6m4idRLJ+WE49/ov0VA5/S8+fRHosP7WrBnrja8GLqd9jiG3HOSoFT4BgQ1kBJvyKLg88Kyt5V83dmhWhr2TzWHVeo7gtOmN0pNT7Xl3dxChLvM1LoAolif9PXLTs81CFwUTCWd+DPMa4SX7Ti8MUscy6YNkyB74AlsbLM/0JEvBSZd+72ZJlG1laKlt8adcp2EXngnqQn0r9d5ws5kF1J3BbkKtKClssuBSsFRBckn7xlnXlO8o6lsq/j9cwpGYAoVy+gE/ZaVNUwG4wvZ9hlqY1sWqo9r9R9KjyVkIdXq0VIxYCudxLYyEtf30pKrGw== cpeel

(that’s all suppose to be on one line).

A simple massaging and seahorse was happy with it.