FreeNAS 11.1 iSCSI datastore in ESXi 6.7

In the beginning, there was a bare-metal FreeNAS install hosting a datastore over iSCSI to a separate ESXi host. This worked well, but required two hosts running 24x7. However, ESXi supports hardware RAID cards just fine. I tried going this route, but my PERC 6i is absolute garbage and, aside from providing super slow transfers, was actually timing out on disc writes. So, I decided to attempt to go "hyperconverged." 



The concept is simple, install ESXi on an SSD along side a FreeNAS VM that coexists on a datastore on the same SSD. A separate vSwitch and portgroup (with no uplink) is created for iSCSI traffic and configured with an MTU of 9000. The only real challenge is ensuring that ESXi rescans the datastore after FreeNAS boots so that the remaining VMs can boot. To that end, I created a shell script that can run at boot.

I won't go in to details on how to setup ESXi nor will I detail configure FreeNAS to use iSCSI.  There are numerous tutorials available online that can help. I'm just detailing the magic sauce I used to ensure FreeNAS boots up and that ESXi is sees the datastore before proceeding.


In short, I edited /etc/rc.local.d/local.sh so that at boot time, it will power on FreeNAS, wait 2 minutes to allow FreeNAS to boot, start rescanning the iSCSI storage device until the device is ok or degraded* and then allow booting to continue. Since we needed our script to rescan the device after FreeNAS is loaded, it was necessary to script the the loading of the VM instead of setting it to autostart in the GUI.  Well, technically I do have ESXi configured to automatically start FreeNAS first, but that is only so that automated shutdown will shutdown FreeNAS last. Once ESXi sees the VM is running, and it tries to power it on, it will actually error as it is already started. Other than throwing an arbitrary error at boot time, the script seems to do the job.  Without further ado, here is my local.sh script that loads my FreeNAS VM:

#!/bin/sh

# local configuration options

# Note: modify at your own risk!  If you do/use anything in this

# script that is not part of a stable API (relying on files to be in

# specific places, specific tools, specific output, etc) there is a

# possibility you will end up with a broken system after patching or

# upgrading.  Changes are not supported unless under direction of

# VMware support.

# Note: This script will not be run when UEFI secure boot is enabled.

# Find Vmid of the FreeNAS vm

export FreeNASvm="$(vim-cmd vmsvc/getallvms | grep 'FreeNAS.vmx' | cut -b 1-2)"

# Power on the FreeNAS second

vim-cmd vmsvc/power.on $FreeNASvm

# Booting takes at least 2 minutes, wait instead of checking every ~15 seconds

sleep 120

# Wait for the iSCSI to become available

export status=0

until [ $status -eq 1 ]

do

   sleep 5

   esxcli storage core adapter rescan -A="vmhba64"

   echo Waiting on FreeNAS to boot...

   sleep 10

   esxcli storage core device list -d="naa.6589aba000000134e8ba11c4ba3dd2bc" | grep '  Status: ' | grep 'degraded\|on' > /dev/null && status=$((1))

done

# Yipee, continue booting...

echo FreeNAS has finished loading!

exit 0

NOTE: 
FreeNAS.vmx - the vmx associated with your VM
vmhba64 - the name of your iSCSI adapter
naa.6589aba000000134e8ba11c4ba3dd2bc - your iSCSI device

*- I didn't bother putting two virtual interfaces on FreeNAS and ESXi for iSCSI. Since there isn't two paths between FreeNAS and iSCSI, ESXi flags the device as 'degraded.' If there were a physical connection, I'd recommend the redundancy. As the connection is virtual, I'm willing to assume the risk associated with having a single path.