Linux - LXD: Move containers between ZFS pools

The situation: I have a shiny new (to me) server set up, and though I’m short a couple of caddies to drop my ZFS pool into it, I want to start setting up the segregated LXD containers for each of my services. So I set the containers up on a temporary pool on a spare 1TB hard disk, and I’ll want to move those containers over to another pool later.

How do we move those containers over? It turns out, as of at least LXD 3.6 (might work on earlier versions), LXD supports multiple storage pools, and you can migrate containers between them. The downside is you can’t keep the same container name, so you move it in two parts - move it to the new storage pool with a temporary name, then move it back to it’s original name.

In my situation I have a ZFS pool called “zeefus”, with a storage pool for LXD on it called “lxd”. I just made a seperate storage pool called “lxd2”, but this could be on an entirely different ZFS pool and it’d work fine:

root@lxdhost:/zeefus/storage/homes# zfs create zeefus/lxd2
root@lxdhost:/zeefus/storage/homes# lxc storage create lxd2 zfs source=zeefus/lxd2
root@lxdhost:/zeefus/storage/homes# lxc storage list
+------+-------------+--------+------------------+---------+
| NAME | DESCRIPTION | DRIVER |      SOURCE      | USED BY |
+------+-------------+--------+------------------+---------+
| lxd  |             | zfs    | zeefus/lxd       | 7       |
+------+-------------+--------+------------------+---------+
| lxd2 |             | zfs    | zeefus/lxd2      | 0       |
+------+-------------+--------+------------------+---------+
root@lxdhost:/zeefus/storage/homes# lxc move test-one test-one -s lxd2
Error: Add container info to the database: This container already exists
root@lxdhost:/zeefus/storage/homes# lxc move test-one test-one-new -s lxd2
root@lxdhost:/zeefus/storage/homes# lxc storage list
+------+-------------+--------+------------------+---------+
| NAME | DESCRIPTION | DRIVER |      SOURCE      | USED BY |
+------+-------------+--------+------------------+---------+
| lxd  |             | zfs    | zeefus/lxd       | 6       |
+------+-------------+--------+------------------+---------+
| lxd2 |             | zfs    | zeefus/lxd2      | 1       |
+------+-------------+--------+------------------+---------+
root@lxdhost:/zeefus/storage/homes# lxc move test-one-new test-one
root@lxdhost:/zeefus/storage/homes# zfs list | grep lxd2
zeefus/lxd2                                                                          618M   889G    24K  none
zeefus/lxd2/containers                                                               618M   889G    24K  none
zeefus/lxd2/containers/test-one                                                      618M   889G   618M  /var/snap/lxd/common/lxd/storage-pools/lxd2/containers/test-one
zeefus/lxd2/custom                                                                    24K   889G    24K  none
zeefus/lxd2/custom-snapshots                                                          24K   889G    24K  none
zeefus/lxd2/deleted                                                                   24K   889G    24K  none
zeefus/lxd2/images                                                                    24K   889G    24K  none
zeefus/lxd2/snapshots                                                                 24K   889G    24K  none

Removing the storage pool after all containers were destroyed required a reboot.

Other than that, it worked fine, though of course it’s fairly slow as it must rsync the container data across. I can’t imagine it won’t work when I go to move between ZFS pools, and then hopefully I can destroy the temporary pool completely and remove the extra disk without any data loss.