VS Code spell check: simplicity wins

Spell Right is a spell checker extension for VS Code. Its README is a little convoluted. Here is how to set it up on a Linux system.

Step 1: set up dictionary files:

cd $HOME/.config/Code/
mkdir Dictionaries && cd Dictionaries
git clone https://github.com/titoBouzout/Dictionaries .

Step 2: In VS Code press Ctrl+P and run

 ext install ban.spellright

Step 3: restart VS Code

Step 4: open the document to spell-check in VS Code. Then click the eye icon (similar to 👁️) in the bottom-right corner of the VS Code status bar. Select the language, press OK. The document is now being spell-checked. Screenshot:

vs code spell check with spell right

Remarks

For my use cases I prefer Spell Right over the more complex Code Spell Checker extension. Spell Right seems to be pretty similar to the simple, unobtrusive experience provided by Sublime’s built-in spell checker (which I was happy with for many years, also using the titoBouzout/Dictionaries).

Setting und using variables in a Makefile: pitfalls

A debug session. I ran into a problem in CI where accessing /var/run/docker.sock from within a container failed with a permission error. I had this specific part working before. So I did a bit of bisecting and added debug output and found the critical difference. In this case I get a permission error (EACCES):

uid=2000(buildkite-agent) gid=0(root) groups=0(root)

In this case not:

uid=2000(buildkite-agent) gid=1001 groups=1001

The difference is in the unix group membership specifics of unix user uid=2000(buildkite-agent) within the specific container. If the user is member of gid=1001 then access is allowed, if the user is member of gid=0 then access is denied.

I found that in the erroneous case I was passing this command line argument to docker run ...:

-u 2000:

Nothing after the colon. This is where the group ID (gid) belongs. docker run ... did not error out. That is no good. This input was treated the same as -u 2000 or -u 2000:0.

But why was there no gid after the colon when I have this in my Makefile?

-u $(shell id -u):${DOCKER_GID_HOST}

Because when this line was run DOCKER_GID_HOST (a Make variable, not an environment variable) was actually not set.

A shell program would catch this case (variable used, but not set) when being used with the -o nounset option, and error out. Make does not have this kind of protection. There is no general protection against using variables that are not set. As far as I know!

Okay, but why was the variable DOCKER_GID_HOST not set when I have

DOCKER_GID_HOST := $(shell getent group docker | awk -F: '{print $$3}')

right before executing docker run? Well, because this is a Makefile. Where you cannot set a variable in one line of a recipe, and use it in the next one (a pattern that we use in almost every other programming environment).

The lines of a recipe are executed in independent shells. The next line’s state is very much decoupled from the previous line’s state, that’s how Makefiles work.

This is probably the most important thing to know about Make, and one of the most common mistakes, and I certainly knew this before, and I certainly made this same mistake before. And I made it again, like probably every single time that I had to do things with Make.

Makefiles are flexible and great, and sometimes they make you waste a great deal of time compared to other development environments, man.

Stack Overflow threads on the matter, with goodies like workarounds, best practices, and generally helpful discussion:

Building Pandas from source: conflicting types for ‘_xgetbv’ (while building BLOSC)

Quick note: I tried building pandas from source (Fedora 30).

During

$ python -m pip install -r requirements-dev.txt

I ran into

blosc/shuffle.c:177:1: error: conflicting types for ‘_xgetbv’

A little more context:

$ python -m pip install -r requirements-dev.txt
 
[...]
 
Installing collected packages: blosc, bottleneck, numexpr, soupsieve, beautifulsoup4, llvmlite, numba, thrift, fastparquet, html5lib, lxml, jdcal, et-xmlfile, openpyxl, pyarrow, PyQt5-sip, pyqt5, tables, python-snappy, s3fs, sqlalchemy, xarray, xlrd, xlsxwriter, xlwt, odfpy, pyreadstat, pandas-sphinx-theme
  Running setup.py install for blosc ... error
    Complete output from command /home/jp/.pyenv/versions/374-pandas-dev/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-fydbjk8m/blosc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-x9bwxqkv/install-record.txt --single-version-externally-managed --compile --install-headers /home/jp/.pyenv/versions/374-pandas-dev/include/site/python3.7/blosc:
    SSE2 detected
    AVX2 detected
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/blosc
 
[...]
 
    c-blosc/blosc/shuffle.c:177:1: error: conflicting types for ‘_xgetbv’
      177 | _xgetbv(uint32_t xcr) {
          | ^~~~~~~
    In file included from /usr/lib/gcc/x86_64-redhat-linux/9/include/immintrin.h:43,
                     from c-blosc/blosc/blosc-common.h:71,
                     from c-blosc/blosc/shuffle.h:18,
                     from c-blosc/blosc/shuffle.c:10:
    /usr/lib/gcc/x86_64-redhat-linux/9/include/xsaveintrin.h:60:1: note: previous definition of ‘_xgetbv’ was here
       60 | _xgetbv (unsigned int __A)
          | ^~~~~~~
    In file included from c-blosc/blosc/shuffle.c:11:
    c-blosc/blosc/shuffle-generic.h:61:13: warning: ‘unshuffle_generic_inline’ defined but not used [-Wunused-function]
       61 | static void unshuffle_generic_inline(const size_t type_size,
          |             ^~~~~~~~~~~~~~~~~~~~~~~~
    c-blosc/blosc/shuffle-generic.h:32:13: warning: ‘shuffle_generic_inline’ defined but not used [-Wunused-function]
       32 | static void shuffle_generic_inline(const size_t type_size,
          |             ^~~~~~~~~~~~~~~~~~~~~~
    error: command 'gcc' failed with exit status 1

The python-blosc documentation says :

Compiler specific optimisations are automatically enabled by inspecting the CPU flags building Blosc. They can be manually disabled by setting the following environmental variables: DISABLE_BLOSC_SSE2 and DISABLE_BLOSC_AVX2.

I ignorantly retried building with AVX2 instructions disabled:

$ DISABLE_BLOSC_AVX2=true pip install blosc
Collecting blosc
  Using cached https://files.pythonhosted.org/packages/6d/3b/2b707cd330a205ba5c69b5e8bfa9c05691442e45ce9ce882c4c8d343e61a/blosc-1.8.1.tar.gz
Installing collected packages: blosc
  Running setup.py install for blosc ... done
Successfully installed blosc-1.8.1
$ python -m pip install -r requirements-dev.txt
...
$ python setup.py build_ext --inplace -j 4
...

That worked. In my development setup I do not care about BLOSC performance, which is why I am OK with that workaround.

By the way, I needed to set up the following dependencies on Fedora 30:

sudo dnf install @development-tools
sudo dnf install libzstd-devel gcc-c++ snappy-devel

FreeNAS: insufficient space to install update (how to replace the USB boot device with a larger one)

I have been running a FreeNAS system at home over the last six years on a self-built machine. In 2013 I started with FreeNAS 9.1.0. I updated conservatively over the years without running into problems (this is rare — kudos to the engineering team behind this!). Recently I tried to install one of the last 11.2 patch releases and ran into the following error:

insufficient space to install update

My 4 GB USB thumb drive (which I have been using as the boot device during all these years) got too small.

Here is how to replace the FreeNAS boot device with a device of larger capacity; without downtime. 

Step 1

Plug in an additional USB thumb drive (8 GB capacity in my case).

Step 2

In the web interface, under System -> Boot Environments choose attach and select the device representing the newly plugged-in USB stick (da2, in my case). Select use all disk space. The message attaching device pops up and disappears shortly thereafter. Behind the scenes this adds the new storage device to the existing ZFS pool freenas-boot.

Step 3

Wait for the “re-silvering” to complete: under the hood, ZFS mirrors (copies) all data from the original USB thumb drive to the new device (which is now also part of the ZFS pool freenas-boot). Quote from the ZFS documentation:

The process of moving data from one device to another device is known as resilvering and can be monitored by using the zpool status command.

I monitored the progress with a shell, periodically invoking the command

zpool status freenas-boot

The output of that command is unambiguous, it either says that re-silvering is currently in progress, or not. In my case, the re-silvering took more than 1 hour to complete because the thumb drive I added has some quite slow write performance.

Step 4

Only do this after making sure that re-silvering (step 3) completed :-).

Remove (unplug) the old (small capacity) USB boot device. Verify that the pool is healthy after unplugging: zpool status freenas-boot must show pool state: ONLINE with one device being online. In my case: da2p2 ONLINE. Remember the device name; it is needed in step 6.

Step 5

Set the autoexpand property on the boot pool:

zpool set autoexpand=on freenas-boot

In my case, this command took quite a while to return.

Note: after this command returns zfs list still shows the old (small) capacity for freenas-boot (1.69 G available, in my case).

Step 6

Trigger the automatic expansion:

zpool online -e freenas-boot da2p2

What is this doing? Why does this work? Quote from the ZFS documentation:

You can expand the pool size automatically when a smaller disk is replaced by a larger disk by using the zpool online -e command even if the disk is already online.

Step 7

Validate. zfs list should now show the newly available capacity (in my case: 5.39 G available).

Step 8

The ZFS pool from which the system boots now has more capacity. If you came here because your system update failed with “insufficient space to install update” then you can now retry updating FreeNAS.

If you are using a USB thumb drive as slow as mine then the update procedure can easily take 2-3 hours. During that time I used iostat -x da2 1 (executed in a remote shell, on the FreeNAS system) to confirm that data is actually being copied around (as opposed to the update procedure hanging indefinitely as of an error).

Final words

You can also use this technique to simply mirror your boot device; to operate the freenas-boot pool from two USB sticks. For enhanced fault tolerance of your FreeNAS setup. That is advisable and indeed what I did after switching to a larger pool size.

Command line: extract all zip files into individual directories

I have been professionally using Linux desktop environments for the last 10 years. They all have a number of deficiencies that get in the way of productivity. The lack of a decent GUI archive extraction helper, integrated with the graphical file manager, is just one tiny aspect.

On a fresh Windows system one of the first applications I typically install is 7zip. It adds convenient entries to the context menu of archive files. For example, it allows for selecting multiple archive files at once, offering a 7-Zip - Extract To "*\" in the context menu (found an example screenshot here). That will extract each selected archive file into an individual sub-directory (with the sub-directory’s name being the base name of the archive file w/o file extension). Quite useful!

I have looked a couple of times over the years, but I never found a similar thing for a modern Gnome desktop environment. Let me know if you know of a reliable solution that is well-integrated with one of the popular graphical file managers such as Thunar.

The same can of course be achieved from the terminal. What follows is the one-liner I have been using for the past couple of years. I usually look it up from my shell command history:

find -name '*.zip' -exec sh -c 'unzip -d "${1%.*}" "$1"' _ {} \;

This extracts all zip files in the current directory into individual sub-directories.

If you do not want to extract all zip files in the current directory but only a selection thereof then adjust the command (well, this is where a GUI-based solution would actually be quite handy, no?).