BusyBox on the FreeBSD

As an experiment i decided to play with a BusyBox on a FreeBSD. BusyBox combines tiny versions of many common UNIX utilities into a single small executable.

Some of my goals

1. Create minimal environment suitable for the embedded use. BusyBox on Linux provides a fairly complete environment for any small or embedded system, so was thinking to try the same on the FreeBSD
2. Attempt to reduce Rasberry/BSD boot time. My profiling shows that actually userland boot make take same amount of time as kernel, sometime more. I think reason could be BSD RC init, some of the /sbin/init logic, etc. Not really easy to profile as a lot of this tools are not providing timestamp. To do some “poor man profiler” i patched cu tool to show timestamp on every line printed.
3. Create tiny environment for the FreeBSD Jail. BusyBox could be compiled statically and is commonly used in Docker as a minimal base. Also some of the projects using BusyBox to create custom applets in the embedded world (e.g. RIPE Atlas or Ubiquiti devices)
4. Self education, to better understand how FreeBSD init and friends works, to compare this with Linux one

Initial state

I was surprised to find that BusyBox is already exists in the FreeBSD Ports. However, version in the port is outdated and does not contain many must-have applets. Moreover – i found that it is crashing on arvmv6 arch (tested on Rasberry Pi 1). However, it was a good start! Also i found that there is some very basic and initial support in the busybox source code, so hopefully author would accept non-Linux patches.

So i decided to update port to the latest version and to fix some issues found. Issue with an arm crash is actually clang/arm problem which i was able to workaround. Also i been able to fix few applets and get more tools working

Current state and future improvements:

I submitted PR to update port in the latest version and to include fixes i have done. Currently such applets are compiled in:

addgroup, ar, arch, ash, awk, base64, basename, bc, bunzip2, bzcat, bzip2, cal, cat, chgrp, chmod, chown, chroot, cksum, clear, cmp, comm, cp, cpio, crontab, cttyhack, cut, dc, dd, delgroup, diff, dirname, dnsd, dos2unix, dpkg, dpkg-deb, du, echo, ed, env, expand, expr, factor, fakeidentd, fallocate, false, fatattr, find, flock, fold, fsync, ftpd, ftpget, ftpput, getopt, grep, groups, gunzip, gzip, hd, head, hexdump, hexedit, hostid, hostname, httpd, id, inetd, install, iostat, ipcalc, kill, killall, killall5, less, link, ln, logger, logname, logread, lpq, lpr, ls, lzcat, lzma, lzop, man, md5sum, microcom, mkdir, mkfifo, mknod, mktemp, more, mpstat, mv, nc, nice, nl, nmeter, nohup, nologin, nuke, od, paste, patch, pgrep, pidof, pipe_progress, pkill, pmap, poweroff, printenv, printf, ps, pscan, pwd, pwdx, readlink, readprofile, realpath, reboot, renice, reset, resize, resume, rev, rm, rmdir, rpm, rpm2cpio, run-parts, scriptreplay, sed, seq, setsid, sh, sha1sum, sha256sum, sha3sum, sha512sum, shred, shuf, sleep, smemcap, sort, split, ssl_client, stat, strings, stty, su, sulogin, sum, svok, sync, syslogd, tail, tar, tee, telnet, telnetd, test, tftp, tftpd, timeout, top, touch, tr, traceroute, traceroute6, true, truncate, tty, ttysize, uname, uncompress, unexpand, uniq, unix2dos, unlink, unlzma, unxz, unzip, usleep, uudecode, uuencode, vi, volname, watch, wc, wget, which, whoami, xargs, xxd, xz, xzcat, yes, zcat

I been able to test/fix most of them, they seems to work fine. One of the problems i found is that all tools working with a process are depending on the linux procfs. As for now i am just changing path from /proc to /compat/linux/proc/ and added message about linprocfs requirement to the port. However, for the real use this needs to be patched to use native BSD KVM API. Also most of the network interface related applets will needs to be ported to make network configurable. However, basic functionality works and i been able to create busybox-only jail with working networking tools, shell, etc. Statically contained busybox takes about 4Mb. May be it would be easier to take some of the tools missing from /rescue compared to the the busybox port, will see. Another step to complete is to get init working. After this point it should be possible to get FreeBSD kernel + busybox userland only as a running system. I will try to send all my BSD specific patches to upstream, will see if they would be accepted or not.

Testing in jail:

1. Install busybox with a STATIC option set
2. Create directory for the port. I am using /root/test for the example. Create some sub directories: mkdir -p /root/test/dev /root/test/bin. Copy busybox to the jail: cp /usr/local/bin/busybox /root/test/bin/
3. Create /etc/jail.conf:

testjail {
   path = /root/test;
   mount.devfs;
   host.hostname = testhostname;
   ip4.addr = 192.168.101.113;
   interface = ue0;
   exec.start = "/bin/busybox";
}

1. Finally run jail using jail -c testjail. You should see ash prompt! You can create all /bin links using busybox itself:

cd /bin
./busybox --list|./busybox xargs -n1 ./busybox ln -s busybox

To use network tools you will also need to create /etc/resolv.conf file with a dns server to use. E.g.

# echo nameserver 8.8.8.8 > /etc/resolv.conf
# wget google.com
Connecting to google.com (172.217.23.206:80)
Connecting to www.google.com (172.217.23.228:80)
index.html           100% |*************************************************| 11897  0:00:00 ETA
#

Comments and suggestions are welcome.