Tag Archives: ipmi

Network FreeBSD boot using Mikrotik router TFTP

Sometime bad things happens

One of my FreeBSD servers failed to boot after number of power outages on the
remote site. However IPMI was still working, so I been able to see remote state.

Problem was with damaged ZFS root pool, so single user mode was also not able to boot.

At this location i also had Mikrotik router connected to the same switch, so i
decided to try to fix this server using network (PXE) boot.
It was a challenge for me, because i never tried to boot from the
RouterOS device and was not sure if it will work at all.

Configuring Mikrotik DHCP server

To get network boot possible there are 2 minimal pre-requirements:

  • DHCP server to provide DHCP responses with a special parameters
    (file to boot).
  • TFTP server to host files required for network boot.

Normally FreeBSD also wants NFS for the remote booting, but in this case we will avoid it.

To configure DHCP on the Mikrotik i created new DHCP server and dedicated DHCP network for it, which includes boot-file-name property:

/ip dhcp-server
add interface=MYLAN name=tftpsever

/ip dhcp-server network
add address=10.0.31.0/24 boot-file-name=gpxelinux.0 dns-server=10.0.31.1 \
    gateway=10.0.31.1 netmask=24

Also static lease for the server ethernet card MAC address been added:

/ip dhcp-server lease
add address=10.0.31.30 always-broadcast=yes mac-address=\
      E4:1F:13:6C:12:34 server=tftpsever

Also i would recommend to enable DHCP logging to monitor status of the requests.

Configuring Mikrotik TFTP server

When DHCP configuration is completed, it is time to enable and configure TFTP on
the router. In the RouterOS you have to specify parameters for every hosted file.

Below is my configuration:

/ip tftp
add real-filename=gpxelinux.0 req-filename=gpxelinux.0
add real-filename=chain.c32 req-filename=chain.c32
add real-filename=pxelinux.cfg/default req-filename=\
    pxelinux.cfg/default
add real-filename=memdisk req-filename=memdisk
add real-filename=menu.c32 req-filename=menu.c32
add real-filename=reboot.c32 req-filename=reboot.c32
add allow-rollover=yes real-filename=\
    mfsbsd-11.0-RELEASE-amd64.img req-filename=mfsbsd-11.0-RELEASE-amd64.img

Creating bootable mfsbsd installation on the TFTP server

To recover FreeBSD i decided to use mfsbsd project. This is FreeBSD live CD/USB
which loads to the memory (so no NFS needed) and provides a lot of useful tools
out of the box. Also due to it small size it was possible to fit it on the router flash memory, so no additional downloads or mounts were needed. And, of course, mfsbsd contains everything needed to repair broken ZFS pool. I been using USB image based on 11.0 RELEASE.

But mfsbsd alone is not enough – to boot from network we also need files from the syslinux project. I been using [outdated] version 4.04, just because i had some projects using it in the past and it seems that some modules were renamed in the recent version.

List of the files to upload:

gpxelinux.0 (syslinux-4.04/gpxe/gpxelinux.0)
chain.c32 (syslinux-4.04/com32/modules/chain.c32)
memdisk (syslinux-4.04/memdisk/memdisk)
menu.c32 (syslinux-4.04/com32/menu/menu.c32)
reboot.c32 (syslinux-4.04/com32/modules/reboot.c32)
mfsbsd-11.0-RELEASE-amd64.img (USB image from the mfsbsd web site)
pxelinux.cfg/default (configuration, see below)

To work properly with IPMI and boot mfsbsd image we need to create configuration. Line serial 1 115200 is to define serial port output, and menu ui is mostly to ensure that something useful is going on.

Content of the pxelinux.cfg/default file:

serial 1 115200
console 1
ui menu.c32
menu title Utilities

label mfsbsd
  menu label mfsBSD
  kernel memdisk
  initrd mfsbsd-11.0-RELEASE-amd64.img harddisk raw

Files needs to be copied to the router flash, i used sftp tool to upload them.

Configuring server

Make sure that you have enabled PXE boot in the BIOS (i been able to do this
using IPMI). Also, at least on my server, i had to enable not only Legacy PXE
(which is in use), but also UEFI IPMI or network boot was never starting.
If everything is done correctly you should see DHCP request followed by TFTP.
In my case it was 2 times:

  1. UEFI PXE, which failed because i am using legacy boot.
  2. Legacy PXE which succeed. After loading all required files boot menu was
    shown and i been able to select mfsbsd item.
  3. After selecting this item FreeBSD started to load, but no output been shown
    on IPMI console. This is because by default mfsbsd is not showing anything on the
    serial console. It should be easy to fix, but in my case it was not needed.

In a few minutes i found another DHCP request in the Mikrotik logs, this time
from the mfsbsd, and been able to reach the server using SSH (credentials are root/mfsbsd). It is possible to use RouterOS ssh client (/system ssh user=root 10.0.31.30).

Finally fixing ZFS pool

My attempt to import failed zroot pool from the mfsbsd failed as well. Moreover –
zpool import -F zroot also been not working. Before giving up i been able to find non-documented -X switch, which together with -F been able to completely restore pool in ~30 minutes! After reboot server been able to boot normally.

Todo

  • Check what needs to be done to create UEFI compatible FreeBSD network boot.
  • Create additional menu items for RescueCD.
  • Create mfsbsd which sends output to the serial port 2, to be compatible with IPMI.
  • Utilize latest syslinux instead of very outdated one.
Advertisements
Tagged , , ,

How to access Integrated Management Module on IBM System x3650 M3 server under FreeBSD

IBM System x3650 M3 server provides nice looking Integrated Management Module (IMM) GUI/CLI which can be accessed remotely (using dedicated network interface) or directly from host. In this short article I will describe how to do this from FreeBSD host machine.All tests were done with FreeBSD 10.1-RELEASE-p6 using GENERIC kernel.

  1. We will need to find virtual network card provided by IMM (RNDISCDC ETHER IBM):
    root@host /root]# usbconfig
    ugen0.1: <UHCI root HUB Intel> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen2.1: <EHCI root HUB Intel> at usbus2, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
    ugen1.1: <UHCI root HUB Intel> at usbus1, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen4.1: <UHCI root HUB Intel> at usbus4, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen3.1: <UHCI root HUB Intel> at usbus3, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen6.1: <EHCI root HUB Intel> at usbus6, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
    ugen5.1: <UHCI root HUB Intel> at usbus5, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen3.2: <RNDISCDC ETHER IBM> at usbus3, cfg=1 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
    In our case it is ugen3.2.
  2. This USB device supports 2 USB configuration – default (and active on boot) – RNDIS or alternate – CDC. FreeBSD works fine with CDC, so we need to switch this USB device to it:
    root@host /root]# usbconfig -d ugen3.2 set_config 1
    After this device should be detected by FreeBSD and dmesg should contain something like this:
    umodem0: at uhub4, port 2, addr 2 (disconnected)
    cdce0: on usbus3
    ue0: on cdce0
    ue0: Ethernet address: e6:1f:13:5e:ab:cd
  3. Now only thing left is to run dhclient on the new network interface:
    [root@host /root]# dhclient ue0
    DHCPREQUEST on ue0 to 255.255.255.255 port 67
    DHCPACK from 169.254.95.118
    bound to 169.254.95.120 -- renewal in 300 seconds.

    Here we can see that address of the IMM is 169.254.95.118. We can use it to connect with telnet or https to get IMM interface.
  4. Username should be admin, and password could be changed using “ipmitool” utility:
    [root@host /root]# ipmitool user set password 2

Thats it 🙂 Using IMM you can manage your hardware, monitor server and do many other interesting things.

Tagged , , , ,