MacOS: Samba as a Time Machine server
After redoing the entire process and verifying that it does indeed work, here’s the absolute minimum configuration I’ve found to work. I originally tried Feld’s method but couldn’t get it to work, I’m suspecting it was AFP/Avahi changes, though it’s possible I screwed something up. This way is significantly easier, as long as you only have Mac users who are not afraid of the command line.
You need Samba 4.8 or higher in order to get the “fruit” extensions, and I believe (but have not confirmed) that you need High Sierra or later or it seems to expect AFP. I believe Spotlight isn’t required, as I installed Samba from FreeBSD’s packages with no special configuration.
Create the filesystem
100GiB is plenty for my uses as my Macbook is mostly empty, YMMV though.
zfs create tank/timecapsule
zfs set quota=100G tank/timecapsule
mkdir /tank/timecapsule/fwaggle-mbp
chown fwaggle:fwaggle /tank/timecapsule/fwaggle-mbp
Samba Configuration
I ruthlessly nicked Adam Demasi’s configuration and modified it to suit:
[timemachine]
comment = Time Machine
path = /tank/timecapsule
browseable = no
writeable = yes
create mask = 0600
directory mask = 0700
spotlight = yes
vfs objects = catia fruit streams_xattr
fruit:aapl = yes
fruit:time machine = yes
Obviously you’d set user or host restrictions as appropriate for your network.
Configure Time Machine
Simply issue tmutil setdestination 'smb://user:pass@server/share/machine-name'
(replacing the URL with the appropriate information) on the Mac terminal, then check Time Machine’s configuration to make sure it’s set up correctly. It won’t show the free space until after the first run, and the first backup takes forever, but afterwards it’s super quick and everything shows correctly (though Apple measure things in GB instead of GiB - or vice versa, I get them confused - so my 100GiB partition shows up as ~107GB).