Sometime bad things happens
One of my FreeBSD servers failed to boot after number of power outages on the
remote site. However IPMI was still working, so I been able to see remote state.
Problem was with damaged ZFS root pool, so single user mode was also not able to boot.
At this location i also had Mikrotik router connected to the same switch, so i
decided to try to fix this server using network (PXE) boot.
It was a challenge for me, because i never tried to boot from the
RouterOS device and was not sure if it will work at all.
Configuring Mikrotik DHCP server
To get network boot possible there are 2 minimal pre-requirements:
- DHCP server to provide DHCP responses with a special parameters
(file to boot).
- TFTP server to host files required for network boot.
Normally FreeBSD also wants NFS for the remote booting, but in this case we will avoid it.
To configure DHCP on the Mikrotik i created new DHCP server and dedicated DHCP network for it, which includes
/ip dhcp-server add interface=MYLAN name=tftpsever /ip dhcp-server network add address=10.0.31.0/24 boot-file-name=gpxelinux.0 dns-server=10.0.31.1 \ gateway=10.0.31.1 netmask=24
Also static lease for the server ethernet card MAC address been added:
/ip dhcp-server lease add address=10.0.31.30 always-broadcast=yes mac-address=\ E4:1F:13:6C:12:34 server=tftpsever
Also i would recommend to enable DHCP logging to monitor status of the requests.
Configuring Mikrotik TFTP server
When DHCP configuration is completed, it is time to enable and configure TFTP on
the router. In the RouterOS you have to specify parameters for every hosted file.
Below is my configuration:
/ip tftp add real-filename=gpxelinux.0 req-filename=gpxelinux.0 add real-filename=chain.c32 req-filename=chain.c32 add real-filename=pxelinux.cfg/default req-filename=\ pxelinux.cfg/default add real-filename=memdisk req-filename=memdisk add real-filename=menu.c32 req-filename=menu.c32 add real-filename=reboot.c32 req-filename=reboot.c32 add allow-rollover=yes real-filename=\ mfsbsd-11.0-RELEASE-amd64.img req-filename=mfsbsd-11.0-RELEASE-amd64.img
Creating bootable mfsbsd installation on the TFTP server
To recover FreeBSD i decided to use mfsbsd project. This is FreeBSD live CD/USB
which loads to the memory (so no NFS needed) and provides a lot of useful tools
out of the box. Also due to it small size it was possible to fit it on the router flash memory, so no additional downloads or mounts were needed. And, of course, mfsbsd contains everything needed to repair broken ZFS pool. I been using USB image based on 11.0 RELEASE.
But mfsbsd alone is not enough – to boot from network we also need files from the syslinux project. I been using [outdated] version 4.04, just because i had some projects using it in the past and it seems that some modules were renamed in the recent version.
List of the files to upload:
gpxelinux.0 (syslinux-4.04/gpxe/gpxelinux.0) chain.c32 (syslinux-4.04/com32/modules/chain.c32) memdisk (syslinux-4.04/memdisk/memdisk) menu.c32 (syslinux-4.04/com32/menu/menu.c32) reboot.c32 (syslinux-4.04/com32/modules/reboot.c32) mfsbsd-11.0-RELEASE-amd64.img (USB image from the mfsbsd web site) pxelinux.cfg/default (configuration, see below)
To work properly with IPMI and boot mfsbsd image we need to create configuration. Line
serial 1 115200 is to define serial port output, and menu ui is mostly to ensure that something useful is going on.
Content of the
serial 1 115200 console 1 ui menu.c32 menu title Utilities label mfsbsd menu label mfsBSD kernel memdisk initrd mfsbsd-11.0-RELEASE-amd64.img harddisk raw
Files needs to be copied to the router flash, i used
sftp tool to upload them.
Make sure that you have enabled PXE boot in the BIOS (i been able to do this
using IPMI). Also, at least on my server, i had to enable not only Legacy PXE
(which is in use), but also UEFI IPMI or network boot was never starting.
If everything is done correctly you should see DHCP request followed by TFTP.
In my case it was 2 times:
- UEFI PXE, which failed because i am using legacy boot.
- Legacy PXE which succeed. After loading all required files boot menu was
shown and i been able to select mfsbsd item.
- After selecting this item FreeBSD started to load, but no output been shown
on IPMI console. This is because by default mfsbsd is not showing anything on the
serial console. It should be easy to fix, but in my case it was not needed.
In a few minutes i found another DHCP request in the Mikrotik logs, this time
from the mfsbsd, and been able to reach the server using SSH (credentials are root/mfsbsd). It is possible to use RouterOS ssh client (
/system ssh user=root 10.0.31.30).
Finally fixing ZFS pool
My attempt to import failed
zroot pool from the mfsbsd failed as well. Moreover –
zpool import -F zroot also been not working. Before giving up i been able to find non-documented
-X switch, which together with
-F been able to completely restore pool in ~30 minutes! After reboot server been able to boot normally.
- Check what needs to be done to create UEFI compatible FreeBSD network boot.
- Create additional menu items for RescueCD.
- Create mfsbsd which sends output to the serial port 2, to be compatible with IPMI.
- Utilize latest syslinux instead of very outdated one.