I have been running a FreeNAS system at home over the last six years on a self-built machine. In 2013 I started with FreeNAS 9.1.0. I updated conservatively over the years without running into problems (this is rare — kudos to the engineering team behind this!). Recently I tried to install one of the last 11.2 patch releases and ran into the following error:

insufficient space to install update

My 4 GB USB thumb drive (which I have been using as the boot device during all these years) got too small.

Here is how to replace the FreeNAS boot device with a device of larger capacity; without downtime. 

Step 1

Plug in an additional USB thumb drive (8 GB capacity in my case).

Step 2

In the web interface, under System -> Boot Environments choose attach and select the device representing the newly plugged-in USB stick (da2, in my case). Select use all disk space. The message attaching device pops up and disappears shortly thereafter. Behind the scenes this adds the new storage device to the existing ZFS pool freenas-boot.

Step 3

Wait for the “re-silvering” to complete: under the hood, ZFS mirrors (copies) all data from the original USB thumb drive to the new device (which is now also part of the ZFS pool freenas-boot). Quote from the ZFS documentation:

The process of moving data from one device to another device is known as resilvering and can be monitored by using the zpool status command.

I monitored the progress with a shell, periodically invoking the command

zpool status freenas-boot

The output of that command is unambiguous, it either says that re-silvering is currently in progress, or not. In my case, the re-silvering took more than 1 hour to complete because the thumb drive I added has some quite slow write performance.

Step 4

Only do this after making sure that re-silvering (step 3) completed :-).

Remove (unplug) the old (small capacity) USB boot device. Verify that the pool is healthy after unplugging: zpool status freenas-boot must show pool state: ONLINE with one device being online. In my case: da2p2 ONLINE. Remember the device name; it is needed in step 6.

Step 5

Set the autoexpand property on the boot pool:

zpool set autoexpand=on freenas-boot

In my case, this command took quite a while to return.

Note: after this command returns zfs list still shows the old (small) capacity for freenas-boot (1.69 G available, in my case).

Step 6

Trigger the automatic expansion:

zpool online -e freenas-boot da2p2

What is this doing? Why does this work? Quote from the ZFS documentation:

You can expand the pool size automatically when a smaller disk is replaced by a larger disk by using the zpool online -e command even if the disk is already online.

Step 7

Validate. zfs list should now show the newly available capacity (in my case: 5.39 G available).

Step 8

The ZFS pool from which the system boots now has more capacity. If you came here because your system update failed with “insufficient space to install update” then you can now retry updating FreeNAS.

If you are using a USB thumb drive as slow as mine then the update procedure can easily take 2-3 hours. During that time I used iostat -x da2 1 (executed in a remote shell, on the FreeNAS system) to confirm that data is actually being copied around (as opposed to the update procedure hanging indefinitely as of an error).

Final words

You can also use this technique to simply mirror your boot device; to operate the freenas-boot pool from two USB sticks. For enhanced fault tolerance of your FreeNAS setup. That is advisable and indeed what I did after switching to a larger pool size.