Monthly Archives: October 2019

Building Pandas from source: conflicting types for ‘_xgetbv’ (while building BLOSC)

Quick note: I tried building pandas from source (Fedora 30).

During

$ python -m pip install -r requirements-dev.txt

I ran into

blosc/shuffle.c:177:1: error: conflicting types for ‘_xgetbv’

A little more context:

$ python -m pip install -r requirements-dev.txt
 
[...]
 
Installing collected packages: blosc, bottleneck, numexpr, soupsieve, beautifulsoup4, llvmlite, numba, thrift, fastparquet, html5lib, lxml, jdcal, et-xmlfile, openpyxl, pyarrow, PyQt5-sip, pyqt5, tables, python-snappy, s3fs, sqlalchemy, xarray, xlrd, xlsxwriter, xlwt, odfpy, pyreadstat, pandas-sphinx-theme
  Running setup.py install for blosc ... error
    Complete output from command /home/jp/.pyenv/versions/374-pandas-dev/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-fydbjk8m/blosc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-x9bwxqkv/install-record.txt --single-version-externally-managed --compile --install-headers /home/jp/.pyenv/versions/374-pandas-dev/include/site/python3.7/blosc:
    SSE2 detected
    AVX2 detected
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/blosc
 
[...]
 
    c-blosc/blosc/shuffle.c:177:1: error: conflicting types for ‘_xgetbv’
      177 | _xgetbv(uint32_t xcr) {
          | ^~~~~~~
    In file included from /usr/lib/gcc/x86_64-redhat-linux/9/include/immintrin.h:43,
                     from c-blosc/blosc/blosc-common.h:71,
                     from c-blosc/blosc/shuffle.h:18,
                     from c-blosc/blosc/shuffle.c:10:
    /usr/lib/gcc/x86_64-redhat-linux/9/include/xsaveintrin.h:60:1: note: previous definition of ‘_xgetbv’ was here
       60 | _xgetbv (unsigned int __A)
          | ^~~~~~~
    In file included from c-blosc/blosc/shuffle.c:11:
    c-blosc/blosc/shuffle-generic.h:61:13: warning: ‘unshuffle_generic_inline’ defined but not used [-Wunused-function]
       61 | static void unshuffle_generic_inline(const size_t type_size,
          |             ^~~~~~~~~~~~~~~~~~~~~~~~
    c-blosc/blosc/shuffle-generic.h:32:13: warning: ‘shuffle_generic_inline’ defined but not used [-Wunused-function]
       32 | static void shuffle_generic_inline(const size_t type_size,
          |             ^~~~~~~~~~~~~~~~~~~~~~
    error: command 'gcc' failed with exit status 1

The python-blosc documentation says :

Compiler specific optimisations are automatically enabled by inspecting the CPU flags building Blosc. They can be manually disabled by setting the following environmental variables: DISABLE_BLOSC_SSE2 and DISABLE_BLOSC_AVX2.

I ignorantly retried building with AVX2 instructions disabled:

$ DISABLE_BLOSC_AVX2=true pip install blosc
Collecting blosc
  Using cached https://files.pythonhosted.org/packages/6d/3b/2b707cd330a205ba5c69b5e8bfa9c05691442e45ce9ce882c4c8d343e61a/blosc-1.8.1.tar.gz
Installing collected packages: blosc
  Running setup.py install for blosc ... done
Successfully installed blosc-1.8.1
$ python -m pip install -r requirements-dev.txt
...
$ python setup.py build_ext --inplace -j 4
...

That worked. In my development setup I do not care about BLOSC performance, which is why I am OK with that workaround.

By the way, I needed to set up the following dependencies on Fedora 30:

sudo dnf install @development-tools
sudo dnf install libzstd-devel gcc-c++ snappy-devel

FreeNAS: insufficient space to install update (how to replace the USB boot device with a larger one)

I have been running a FreeNAS system at home over the last six years on a self-built machine. In 2013 I started with FreeNAS 9.1.0. I updated conservatively over the years without running into problems (this is rare — kudos to the engineering team behind this!). Recently I tried to install one of the last 11.2 patch releases and ran into the following error:

insufficient space to install update

My 4 GB USB thumb drive (which I have been using as the boot device during all these years) got too small.

Here is how to replace the FreeNAS boot device with a device of larger capacity; without downtime. 

Step 1

Plug in an additional USB thumb drive (8 GB capacity in my case).

Step 2

In the web interface, under System -> Boot Environments choose attach and select the device representing the newly plugged-in USB stick (da2, in my case). Select use all disk space. The message attaching device pops up and disappears shortly thereafter. Behind the scenes this adds the new storage device to the existing ZFS pool freenas-boot.

Step 3

Wait for the “re-silvering” to complete: under the hood, ZFS mirrors (copies) all data from the original USB thumb drive to the new device (which is now also part of the ZFS pool freenas-boot). Quote from the ZFS documentation:

The process of moving data from one device to another device is known as resilvering and can be monitored by using the zpool status command.

I monitored the progress with a shell, periodically invoking the command

zpool status freenas-boot

The output of that command is unambiguous, it either says that re-silvering is currently in progress, or not. In my case, the re-silvering took more than 1 hour to complete because the thumb drive I added has some quite slow write performance.

Step 4

Only do this after making sure that re-silvering (step 3) completed :-).

Remove (unplug) the old (small capacity) USB boot device. Verify that the pool is healthy after unplugging: zpool status freenas-boot must show pool state: ONLINE with one device being online. In my case: da2p2 ONLINE. Remember the device name; it is needed in step 6.

Step 5

Set the autoexpand property on the boot pool:

zpool set autoexpand=on freenas-boot

In my case, this command took quite a while to return.

Note: after this command returns zfs list still shows the old (small) capacity for freenas-boot (1.69 G available, in my case).

Step 6

Trigger the automatic expansion:

zpool online -e freenas-boot da2p2

What is this doing? Why does this work? Quote from the ZFS documentation:

You can expand the pool size automatically when a smaller disk is replaced by a larger disk by using the zpool online -e command even if the disk is already online.

Step 7

Validate. zfs list should now show the newly available capacity (in my case: 5.39 G available).

Step 8

The ZFS pool from which the system boots now has more capacity. If you came here because your system update failed with “insufficient space to install update” then you can now retry updating FreeNAS.

If you are using a USB thumb drive as slow as mine then the update procedure can easily take 2-3 hours. During that time I used iostat -x da2 1 (executed in a remote shell, on the FreeNAS system) to confirm that data is actually being copied around (as opposed to the update procedure hanging indefinitely as of an error).

Final words

You can also use this technique to simply mirror your boot device; to operate the freenas-boot pool from two USB sticks. For enhanced fault tolerance of your FreeNAS setup. That is advisable and indeed what I did after switching to a larger pool size.

Command line: extract all zip files into individual directories

I have been using Linux desktop environments professionally for the last 10 years. There is a lot to not like about them. For example, the lack of a really good graphical archive extraction helper, integrated with the graphical file manager.

On a fresh Windows system one of the first applications I typically install is 7zip. It adds convenient entries to the context menu of archive files. For example, it allows selecting multiple archive files at once, offering a 7-Zip - Extract To "*\" in the context menu (found an example screenshot here). That will extract each selected archive file into an individual subdirectory (with the subdirectory’s name being the base name of the archive file w/o file extension). That can be very useful. In a number of attempts, I could not find something similar for a modern Gnome desktop environment (let me know if you know of a reliable solution that is well-integrated with one of the popular graphical file managers).

Not such a big deal, of course, the same can be achieved from the terminal. This is the one-liner I have been using for that for a couple of years. I usually look it up from my shell command history:

find -name '*.zip' -exec sh -c 'unzip -d "${1%.*}" "$1"' _ {} \;

This extracts all zip files in the current directory into individual sub-directories.

If you do not want to extract all zip files in the current directory but only a selection thereof then adjust the command (well, see, this is where the GUI-based solution I referred to above is actually quite useful).