Tag Archives: freebsd

C++ cross-compilation on Linux for the FreeBSD target using GCC or CLANG


Sometime it could be convenient to generate FreeBSD binaries on the Linux host. Just a few examples:

  • CI/CD deployment with a compilation check on the Linux Containers. It is especially the case for the SaaS environments where you are not able to select another OS (or it is not supported by the vendor)
  • Ability to automatically generate binaries for the all supported platforms with a minimal cost/efforts.
  • Using single docker or other linux-specific technology to cover all platforms.

I am using osxcross without any problem to generate OSX binaries, so decided to do the same with FreeBSD. This article shows how to do this and will provide some automated examples.

Using GCC

Most of the articles in the net i found are recommending to use GCC, which is default compiler on Linux distributions. GCC supports cross-compilation natively, but to achieve this you need to build entire kit from scratch.

Short overview of the process:

  1. Compile install latest binutils with --target=x86_64-pc-freebsd10 configure argument.
  2. Get FreeBSD libs/includes and install them to the buildroot location
  3. Compile and install GMP with --host=x86_64-pc-freebsd10 configure argument.
  4. Compile and install MPC with --host=x86_64-pc-freebsd10 configure argument.
  5. Finally compile GCC, you will have to specify sysroot path and all the libs location. Also you will have to manually define #define HAVE_ALIGNED_ALLOC 1 due to bug in the GCC autoconf scripts.

I created Dockerfile with all steps mentioned. GCC natively supporting only libstdc++ which is not available on the FreeBSD. Below is an example:

root@96de1e64fff1:/# /opt/cross-freebsd/bin/x86_64-pc-freebsd10-c++ -o helo helo.cpp
root@96de1e64fff1:/# file helo
helo: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.3, not stripped
root@96de1e64fff1:/#  readelf -d helo|grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.7]

You can override this by overriding this using compiler/linker flags:

root@96de1e64fff1:/# /opt/cross-freebsd/bin/x86_64-pc-freebsd10-c++ -std=c++11 -I/opt/cross-freebsd/x86_64-pc-freebsd10/usr/include/c++/v1 -nostdinc++ -nodefaultlibs -lc++ -lm -lc -lgcc_s -lgcc -o helo helo.cpp

root@96de1e64fff1:/# readelf -d helo|grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libc++.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcxxrt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.7]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]

Some comments on the commandline:

  • -std=c++11 -nostdinc++ needed to avoid errors on libc++ headers
  • -nodefaultlibs -lc++ -lm -lc -lgcc_s -lgcc will tell gcc to not use with a predefined libraries but to use libc++ instead.

Problems with GCC cross-build:

  • Entire build chain need to be built from scratch. Not a big issue if you are using containers, but still takes some time to maintain it.
  • Natively only libstdc++ is supported which is now not part of the FreeBSD installation. To workaround this you can provide it with your code or to build static. It is possible to use libc++ instead but this required a lot of custom flags and may not work reliable.
  • Natively FreeBSD is not using gcc (with only exception of some ports), so behavior of the resulted code may be different due to some compiler differences.

Using Clang

After all i decided to take another approach. Clang is now (starting from FreeBSD 10) default compiler, as well as libc++ is used instead of libstdc++. Great news is that clang already supports cross-compilation without need to completely recompile entire toolset. I been able to get working FreeBSD binaries on default Ubuntu LTS buildhost.

Below is a part of the Dockerfile to fetch required headers and libraries, install them to /opt/cross-freebsd-10 and fix broken links:

# Get FreeBSD libs/headers, extract and fix broken links
RUN cd /tmp && wget http://ftp.plusline.de/FreeBSD/releases/amd64/10.4-RELEASE/base.txz \
&& mkdir -p /opt/cross-freebsd-10 \
&& cd /opt/cross-freebsd-10 \
&& tar -xf /tmp/base.txz ./lib/ ./usr/lib/ ./usr/include/ \
&& cd /opt/cross-freebsd-10/usr/lib \
&& find . -xtype l|xargs ls -l|grep ' /lib/' \
| awk '{print "ln -sf /opt/cross-freebsd-10"$11 " " $9}' \
| /bin/sh && \
rm -f /tmp/base.txz

Now it is possible to create FreeBSD binaries using additional compiler flags: -target x86_64-unknown-freebsd10.0 --sysroot=/opt/cross-freebsd-10/


root@4f9b820ae1c9:/# clang++ -target x86_64-unknown-freebsd10.0 --sysroot=/opt/cross-freebsd-10/ -o helo helo.cpp
root@4f9b820ae1c9:/# file helo
helo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.4, not stripped

Resulted binary will run on FreeBSD host only.

Lets check dependencies:

root@4f9b820ae1c9:/# readelf -d helo|grep NEEDED
0x0000000000000001 (NEEDED) Shared library: [libc++.so.1]
0x0000000000000001 (NEEDED) Shared library: [libcxxrt.so.1]
0x0000000000000001 (NEEDED) Shared library: [libm.so.5]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.7]

As we could see dependencies are fine and matching normal FreeBSD 10.4 installation. Of course you may have many buildroots to simplify multiply targets.


It was found that both GCC and CLANG are able to cross-compile to FreeBSD using Linux host. However, CLANG require less efforts for this, mostly due to implemented -target switch and natively supported libc++ used by the FreeBSD project. I think it is better to use it for this task, with only exception if your source code has some strict GCC dependencies.

I implemented cross compilation to the FreeBSD as part of the CircleCI tests for the smartomontools project. I am planning to play with other targets (ARM, PPC) as well.

Tagged ,

Network FreeBSD boot using Mikrotik router TFTP

Sometime bad things happens

One of my FreeBSD servers failed to boot after number of power outages on the
remote site. However IPMI was still working, so I been able to see remote state.

Problem was with damaged ZFS root pool, so single user mode was also not able to boot.

At this location i also had Mikrotik router connected to the same switch, so i
decided to try to fix this server using network (PXE) boot.
It was a challenge for me, because i never tried to boot from the
RouterOS device and was not sure if it will work at all.

Configuring Mikrotik DHCP server

To get network boot possible there are 2 minimal pre-requirements:

  • DHCP server to provide DHCP responses with a special parameters
    (file to boot).
  • TFTP server to host files required for network boot.

Normally FreeBSD also wants NFS for the remote booting, but in this case we will avoid it.

To configure DHCP on the Mikrotik i created new DHCP server and dedicated DHCP network for it, which includes boot-file-name property:

/ip dhcp-server
add interface=MYLAN name=tftpsever

/ip dhcp-server network
add address= boot-file-name=gpxelinux.0 dns-server= \
    gateway= netmask=24

Also static lease for the server ethernet card MAC address been added:

/ip dhcp-server lease
add address= always-broadcast=yes mac-address=\
      E4:1F:13:6C:12:34 server=tftpsever

Also i would recommend to enable DHCP logging to monitor status of the requests.

Configuring Mikrotik TFTP server

When DHCP configuration is completed, it is time to enable and configure TFTP on
the router. In the RouterOS you have to specify parameters for every hosted file.

Below is my configuration:

/ip tftp
add real-filename=gpxelinux.0 req-filename=gpxelinux.0
add real-filename=chain.c32 req-filename=chain.c32
add real-filename=pxelinux.cfg/default req-filename=\
add real-filename=memdisk req-filename=memdisk
add real-filename=menu.c32 req-filename=menu.c32
add real-filename=reboot.c32 req-filename=reboot.c32
add allow-rollover=yes real-filename=\
    mfsbsd-11.0-RELEASE-amd64.img req-filename=mfsbsd-11.0-RELEASE-amd64.img

Creating bootable mfsbsd installation on the TFTP server

To recover FreeBSD i decided to use mfsbsd project. This is FreeBSD live CD/USB
which loads to the memory (so no NFS needed) and provides a lot of useful tools
out of the box. Also due to it small size it was possible to fit it on the router flash memory, so no additional downloads or mounts were needed. And, of course, mfsbsd contains everything needed to repair broken ZFS pool. I been using USB image based on 11.0 RELEASE.

But mfsbsd alone is not enough – to boot from network we also need files from the syslinux project. I been using [outdated] version 4.04, just because i had some projects using it in the past and it seems that some modules were renamed in the recent version.

List of the files to upload:

gpxelinux.0 (syslinux-4.04/gpxe/gpxelinux.0)
chain.c32 (syslinux-4.04/com32/modules/chain.c32)
memdisk (syslinux-4.04/memdisk/memdisk)
menu.c32 (syslinux-4.04/com32/menu/menu.c32)
reboot.c32 (syslinux-4.04/com32/modules/reboot.c32)
mfsbsd-11.0-RELEASE-amd64.img (USB image from the mfsbsd web site)
pxelinux.cfg/default (configuration, see below)

To work properly with IPMI and boot mfsbsd image we need to create configuration. Line serial 1 115200 is to define serial port output, and menu ui is mostly to ensure that something useful is going on.

Content of the pxelinux.cfg/default file:

serial 1 115200
console 1
ui menu.c32
menu title Utilities

label mfsbsd
  menu label mfsBSD
  kernel memdisk
  initrd mfsbsd-11.0-RELEASE-amd64.img harddisk raw

Files needs to be copied to the router flash, i used sftp tool to upload them.

Configuring server

Make sure that you have enabled PXE boot in the BIOS (i been able to do this
using IPMI). Also, at least on my server, i had to enable not only Legacy PXE
(which is in use), but also UEFI IPMI or network boot was never starting.
If everything is done correctly you should see DHCP request followed by TFTP.
In my case it was 2 times:

  1. UEFI PXE, which failed because i am using legacy boot.
  2. Legacy PXE which succeed. After loading all required files boot menu was
    shown and i been able to select mfsbsd item.
  3. After selecting this item FreeBSD started to load, but no output been shown
    on IPMI console. This is because by default mfsbsd is not showing anything on the
    serial console. It should be easy to fix, but in my case it was not needed.

In a few minutes i found another DHCP request in the Mikrotik logs, this time
from the mfsbsd, and been able to reach the server using SSH (credentials are root/mfsbsd). It is possible to use RouterOS ssh client (/system ssh user=root

Finally fixing ZFS pool

My attempt to import failed zroot pool from the mfsbsd failed as well. Moreover –
zpool import -F zroot also been not working. Before giving up i been able to find non-documented -X switch, which together with -F been able to completely restore pool in ~30 minutes! After reboot server been able to boot normally.


  • Check what needs to be done to create UEFI compatible FreeBSD network boot.
  • Create additional menu items for RescueCD.
  • Create mfsbsd which sends output to the serial port 2, to be compatible with IPMI.
  • Utilize latest syslinux instead of very outdated one.
Tagged , , ,

Acmetool utility port for the FreeBSD

I am actively using Lets Encrypt certificates for my private and business projects. Initially i been using official Python client to obtain them, and it was all kind of possible problems. At some point i migrated to the Acmetool client which works perfectly and allows me to maintain hundreds of the certificates with a minimal efforts. This is feature list from the web site:

✅ Zero-downtime autorenewal
✅ Supports any webserver
✅ Fully automatable
✅ Single-file dependency-free binary
✅ Idempotent
✅ Fast setup

Only problem for me was lack of the FreeBSD port – yes, you can grab FreeBSD binaries from the author web site, install them somewhere to the /opt/, but its not a Jedi Path. So i had to create FreeBSD port, which after few months of aging was finally accepted to the tree. There are small changes in the port compared to the author build – paths are FreeBSD-style and builds should be repeatable (i hope, at least). As usual – feel free to send me PR-s, bugreports and suggestions.

Lets Encrypt!

Tagged , , ,

Reading DHT11 temperature sensor on Raspberry Pi under FreeBSD

Connecting sensor to the RPi

DHT-11 is a very cheap temperature/humidity sensor which is commonly used in the IoT devices. It is not very accurate, so for the accurate measurement i would recommend to use DHT21 instead. Anyway, i had DHT-11 in my tool box, so decided to start with it. DHT-11 using very simple 1 wire protocol – host is turning on chip by sending 18ms low signal to the data output and then reading 40 bytes of data. Details about the protocol could be found in the specification. To read data from the chip it should be connected to the power (5v) and gpio pin. I used pin 2 as VCC, 6 as GND and 11 as GPIO (it is GPIO17, see pinout):

FreeBSD support

There is no support for this device out of the box. I found some sample code on the github, see lex/freebsd-gpio-dht11 repository. This code was a good starting point, but soon i found 2 issues with it:

  1. Results are very unreliable, probably due to gpio decoding algorithm.
  2. Checksum is not validated, so sometime values are bogus.

Initially i was thinking to fix this myself, but later found kernel module for this purpose, 1 wire over gpio. This module contains DHT11 kernel driver (gpio_sw) which implements DHT-11 protocol in the kernel space and exporting /dev/sw0 for the userland. Driver compiles on FreeBSD11/ARM without any changes. Use make install to install the driver.

Putting all the things together

  • To specify GPIO pin with a sensor put hint.gpio_sw.0.pin=17 into /boot/loader.conf.
  • I found that gpio_sw.ko needs to be loaded after kernel initialization, or device is not created. So i have added /sbin/kldload gpio_sw.ko to the /etc/rc.local file.
  • To make /dev/sw0 available for the non-root users you should add devfs_system_ruleset="localrules" to the /etc/rc.conf and add into /etc/devfs.rules this section:
add path 'sw0' mode 0644
  • Sample program (test.c) shows non human-readable data, use something like printf("h:%d.%d %%, t:%d.%d C\n",Buf[0],Buf[1],Buf[2],Buf[3]); if you want to get humidity and temperature, e.g.
root@rpi-b:/home/freebsd/gpio_sw # ./test
h:22.0 %, t:12.0 C

That is it, after reboot you should have working /dev/sw0 device.

Solving problems with LUA

Final goal was to add this sensor to the domoticz software. It is using LUA scripting to extend it functionality, e.g. to obtain data from non-supported or non standard devices. So, i decided to read /dev/sw0 from the LUA. I wrote simple test script:

file = io.open ("/dev/sw0", "rb")
out = file:read (5)

However script was always returning nil and i had to use truss tool to understand the problem. This is part of the trace:

open("/dev/sw0",O_RDONLY,0666)             = 3 (0x3)
fstat(3,{ mode=crw-r--r-- ,inode=96,size=0,blksize=4096 }) = 0 (0x0)
ioctl(3,TIOCGETA,0xbfbfe28c)             = 0 (0x0)
read(3,0x2065b000,4096)              ERR#22 'Invalid argument'

As you could see – LUA trying to read 4096 bytes, despite the fact that we specified 4. And driver checks this and returns the error. To fix this i did a small patch to avoid error and make LUA happy:

-- gpio_sw/gpio_sw.c    2014-05-12 11:26:51.000000000 +0000
+++ gpio_sw.mod/gpio_sw.c   2017-01-14 14:10:33.736813000 +0000
@@ -273,9 +273,10 @@
   duprintf("read - start, uio_resid=%i\n", uio->uio_resid);
   struct gpio_sw_softc *sc = cdev->si_drv1 ;
-  if ( uio->uio_resid >= sc->BufSize) return EINVAL ;
+  if ( uio->uio_resid >= sc->BufSize) sc->Len=sc->BufSize-1 ; // to work with buffered IO
+  else sc->Len=uio->uio_resid;
   if ( ! sc->GpioStatus) return ENXIO ;
-  sc->Len = uio->uio_resid ;
   int i = 0 ;
   while ( i < sc->Len)

After modification script works as it should:

[root@rpi-b /home/freebsd]# lua52 test.lua

Now chip is connected to the domoticz and reporting actual temperature/humidity.

Tagged , , ,

Domoticz on RPi 1 using FreeBSD/ARM

FreeBSD, Rasberry and Domoticz

I decided to create a Home Automation controller from my old Raspberry 1 device. In the past i used Linux on it but now it is supported by FreeBSD/ARM, so i decided to give it a try.

Installing FreeBSD

This is an easiest part. RPi 1 is fully supported, so i just downloaded FreeBSD11 image and put it on the SD card using dd tool. That`s it, FreeBSD boots normally (no installation process required) and is ready to use. I only added ntpd/sshd to the autostart process.

Installing domoticz

Domoticz is an open source home automation system, written on C++. There is a FreeBSD port for it, so first thing i tried to do was to install binary package using pkg tool. Unfortunately there is no binary package, so i had to install ports. To save some time before starting domoticz compilation i would recommend to install all build requirements using pkg manager, compiling them on RPi 1 would take a lot of time. Some tips on getting domoticz compiled on RPi 1:

  • Clang/C++ is using a lot of memory, so swap is required. I added swap space on the connected USB storage, with 512Mb swap file.
  • Compilation process will take a lot of hours. Probably cross-compilation would be much faster, but then it wont be a normal port build. Anyway, this process is a good crash test for the hardware.
  • I found that build failed on a 2 files: hardware/I2C.cpp and hardware/PiFace.cpp. This is the reason why there is no binary package. To fix the build – replace __arm__ with __linuxarm__. This will fix the build process, issue is reported to the upstream and FreeBSD maintainer.

Using domoticz

I did not found anything platform-specific for the domoticz on the FreeBSD. After installation i only imported database from the OSX installation and copied my LUA scripts for the custom sensors + installed required LUA modules from the ports. Despite the ugly build performance domoticz works very fast and using just about 25Mb of RAM. Currently I2C support in the hardware/I2C.cpp is implemented only for Linux/ARM, i am planning to fix that in a few weeks (i do not have any supported i2c devices now). FreeBSD/ARM itself works very stable, i did not had any problems with it.

Tagged , ,

How to access Integrated Management Module on IBM System x3650 M3 server under FreeBSD

IBM System x3650 M3 server provides nice looking Integrated Management Module (IMM) GUI/CLI which can be accessed remotely (using dedicated network interface) or directly from host. In this short article I will describe how to do this from FreeBSD host machine.All tests were done with FreeBSD 10.1-RELEASE-p6 using GENERIC kernel.

  1. We will need to find virtual network card provided by IMM (RNDISCDC ETHER IBM):
    root@host /root]# usbconfig
    ugen0.1: <UHCI root HUB Intel> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen2.1: <EHCI root HUB Intel> at usbus2, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
    ugen1.1: <UHCI root HUB Intel> at usbus1, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen4.1: <UHCI root HUB Intel> at usbus4, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen3.1: <UHCI root HUB Intel> at usbus3, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen6.1: <EHCI root HUB Intel> at usbus6, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
    ugen5.1: <UHCI root HUB Intel> at usbus5, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
    ugen3.2: <RNDISCDC ETHER IBM> at usbus3, cfg=1 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
    In our case it is ugen3.2.
  2. This USB device supports 2 USB configuration – default (and active on boot) – RNDIS or alternate – CDC. FreeBSD works fine with CDC, so we need to switch this USB device to it:
    root@host /root]# usbconfig -d ugen3.2 set_config 1
    After this device should be detected by FreeBSD and dmesg should contain something like this:
    umodem0: at uhub4, port 2, addr 2 (disconnected)
    cdce0: on usbus3
    ue0: on cdce0
    ue0: Ethernet address: e6:1f:13:5e:ab:cd
  3. Now only thing left is to run dhclient on the new network interface:
    [root@host /root]# dhclient ue0
    DHCPREQUEST on ue0 to port 67
    DHCPACK from
    bound to -- renewal in 300 seconds.

    Here we can see that address of the IMM is We can use it to connect with telnet or https to get IMM interface.
  4. Username should be admin, and password could be changed using “ipmitool” utility:
    [root@host /root]# ipmitool user set password 2

Thats it 🙂 Using IMM you can manage your hardware, monitor server and do many other interesting things.

Tagged , , , ,

Percona XtraBackup FreeBSD port updated to the version 2.1.3

Version 2.1.3 adds some new functionality (including encryption) and also supports MySQL 5.6. Also, to support decompression of the compressed archives I decided to add qpress archiver to the ports tree. See http://www.freshports.org/databases/xtrabackup/ for the details. Please also make a note, that new versions support only MySQL 5.1 (with InnoDB plugin only), MySQL 5.5 and MySQL 5.6.

Tagged , ,

Android Manager Agent protocol research

My mobile phone comes with Android Manager Agent. It is Android agent + Windows desktop application to manage phone, including managing contacts, SMS, media files, applications, etc. It is possible to use this software via WIFI or USB. Unfortunately desktop client is Windows only and i was not able to run it in Wine.

So i decided to reverse protocol using wire-shark and Java decompiler. It was found that communication is not encrypted, only protection is MD5 of the agent PIN (in WIFI mode). Every packet contains from “magic” header, command type, status field and (optionally) data field. My goal is to write utility to backup/restore contacts book and calendar (yes, i don`t want to store it on google) and, possibly to make a GUI for offline browsing/editing.

Right now client is in very early stage and only able to connect to device (WIFI) and request some information. Patches, comments and suggestions are welcome.

Prototype is hosted on github – https://github.com/samm-git/one_touch_993D_gsm/blob/master/android_manager_client.pl

Tagged , , ,

ATOP 2.0.2 performance monitor ported to FreeBSD

Today i updated my FreeBSD port of the ATOP full-screen performance monitor to the recent version. PR is already sent, it should be in a port tree soon. Atop is one of my favorite tool to monitor performance not only because it shows much more then system TOP(1), but also because it allows to record all activity to binary log to work with it later or generate different reports.

FreeBSD port supports all Linux version functionality except per-process network utilization (netatop). Netatop module porting/replacement is not something trivial and will require kernel patching.

Tagged , , ,