Fixing USB detection with uboot on RPI

As part of my experiments with RPi/FreeBSD i decided to move FreeBSD completely to the USB token. Some of the pros:

• Easy to mount on normal laptop if i need to fix something, backup, etc.
• I have tonn of them 🙂
• I think that performance of the recent usb3 flash is better compared to typical SD. However need to verify that.
• Possibility to use SSD/HDD via USB-Sata bridge

Cons:

• RPi 1 does not support real usb boot, you still need to have SD Card to initiate boot process

How RPi 1 boots with FreeBSD:

RPi expects FAT partition on SD Card which contains some files to boot. This includes bootcode.bin (removed on RPi4), start*.elf (primary firmware, different versions, reads config.txt file), u-boot (specified as kernel=u-boot.bin in the config.txt) which loads ubldr (arm port of the loader(8)). UBLDR scanning for the UFS partition to load kernel and pass control to it. Diagram of the process provided below:

As ubldr does not have hardware specific code it is relying on u-boot API to access the disks. And here problems starts – 2 of my 3 USB drives were showing

starting USB...
USB0:   Core Release: 2.80a
scanning bus 0 for devices... usb_new_device: Cannot read configuration, skipping device XXXX:XXXX

on boot. I decided to debug why it happens.

Debugging u-boot

First thing i decided to do is to build a latest u-boot for the RPI platform. To do this i been using official u-boot git – git://git.denx.de/u-boot.git. I was using docker container with Ubuntu 18.04 to avoid any build issues. You will have to install cpp-arm-linux-gnueabi, gcc-arm-linux-gnueabi and binutils-arm-linux-gnueabi packages. Next is to set cross-compiler using export CROSS_COMPILE=arm-linux-gnueabi- command.

In the u-boot folder we will need to configure u-boot for our board and usecase:

1. Run make rpi_defconfig. This will select our board.
2. Using make menuconfig enable CONFIG_API configurable (or just add CONFIG_API=y to .config file)
3. Run make command. This should create u-boot.bin file which you could copy to the SD Card. I am recommending to use different name, e.g. uboot-new.bin, this would allow to revert to the previous loader. Change kernel line in the config.txt to point to the new u-boot.

At this point i got 2 news: good one that my u-boot is working and bad that it has 100% same USB issue. So i decided to debug it. You can enable USB debugging by adding #define DEBUG in the beginning of the common/usb.c file and re-compile u-boot. This would print a lot of debug lines on initial boot and we have a typical heisenbug after this stage – USB detection is fixed 🙂 Okay, at least we know that this is a timing issue and probably debug output prevents it to happens.

After number of experiments i was able to get it working without debug. Here is a diff:

diff --git a/common/usb.c b/common/usb.c
index b70f614d24..121df520bc 100644
--- a/common/usb.c
+++ b/common/usb.c
@@ -1086,10 +1086,11 @@ int usb_select_config(struct usb_device *dev)
         * requests in the first microframe, the stick crashes. Wait about
         * one microframe duration here (1mS for USB 1.x , 125uS for USB 2.0).
         */
-       mdelay(1);
+       mdelay(300);

        /* only support for one config for now */
        err = usb_get_configuration_len(dev, 0);
+       mdelay(100);
        if (err >= 0) {
                tmpbuf = (unsigned char *)malloc_cache_aligned(err);
                if (!tmpbuf)
@@ -1107,6 +1108,7 @@ int usb_select_config(struct usb_device *dev)
        usb_parse_config(dev, tmpbuf, 0);
        free(tmpbuf);
        usb_set_maxpacket(dev);
+       mdelay(100);
        /*
         * we set the default configuration here
         * This seems premature. If the driver wants a different configuration

After all it detects USB device and now it is visible (and bootable!) in ubldr.

Bus usb@7e980000: scanning bus usb@7e980000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found

Ubldr will scan all devices and will try to boot from the first bootable ufs parition found. This way i was able to move all the files to the USB flash with only few files left on the SD.

BusyBox on the FreeBSD

As an experiment i decided to play with a BusyBox on a FreeBSD. BusyBox combines tiny versions of many common UNIX utilities into a single small executable.

Some of my goals

1. Create minimal environment suitable for the embedded use. BusyBox on Linux provides a fairly complete environment for any small or embedded system, so was thinking to try the same on the FreeBSD
2. Attempt to reduce Rasberry/BSD boot time. My profiling shows that actually userland boot make take same amount of time as kernel, sometime more. I think reason could be BSD RC init, some of the /sbin/init logic, etc. Not really easy to profile as a lot of this tools are not providing timestamp. To do some “poor man profiler” i patched cu tool to show timestamp on every line printed.
3. Create tiny environment for the FreeBSD Jail. BusyBox could be compiled statically and is commonly used in Docker as a minimal base. Also some of the projects using BusyBox to create custom applets in the embedded world (e.g. RIPE Atlas or Ubiquiti devices)
4. Self education, to better understand how FreeBSD init and friends works, to compare this with Linux one

Initial state

I was surprised to find that BusyBox is already exists in the FreeBSD Ports. However, version in the port is outdated and does not contain many must-have applets. Moreover – i found that it is crashing on arvmv6 arch (tested on Rasberry Pi 1). However, it was a good start! Also i found that there is some very basic and initial support in the busybox source code, so hopefully author would accept non-Linux patches.

So i decided to update port to the latest version and to fix some issues found. Issue with an arm crash is actually clang/arm problem which i was able to workaround. Also i been able to fix few applets and get more tools working

Current state and future improvements:

I submitted PR to update port in the latest version and to include fixes i have done. Currently such applets are compiled in:

addgroup, ar, arch, ash, awk, base64, basename, bc, bunzip2, bzcat, bzip2, cal, cat, chgrp, chmod, chown, chroot, cksum, clear, cmp, comm, cp, cpio, crontab, cttyhack, cut, dc, dd, delgroup, diff, dirname, dnsd, dos2unix, dpkg, dpkg-deb, du, echo, ed, env, expand, expr, factor, fakeidentd, fallocate, false, fatattr, find, flock, fold, fsync, ftpd, ftpget, ftpput, getopt, grep, groups, gunzip, gzip, hd, head, hexdump, hexedit, hostid, hostname, httpd, id, inetd, install, iostat, ipcalc, kill, killall, killall5, less, link, ln, logger, logname, logread, lpq, lpr, ls, lzcat, lzma, lzop, man, md5sum, microcom, mkdir, mkfifo, mknod, mktemp, more, mpstat, mv, nc, nice, nl, nmeter, nohup, nologin, nuke, od, paste, patch, pgrep, pidof, pipe_progress, pkill, pmap, poweroff, printenv, printf, ps, pscan, pwd, pwdx, readlink, readprofile, realpath, reboot, renice, reset, resize, resume, rev, rm, rmdir, rpm, rpm2cpio, run-parts, scriptreplay, sed, seq, setsid, sh, sha1sum, sha256sum, sha3sum, sha512sum, shred, shuf, sleep, smemcap, sort, split, ssl_client, stat, strings, stty, su, sulogin, sum, svok, sync, syslogd, tail, tar, tee, telnet, telnetd, test, tftp, tftpd, timeout, top, touch, tr, traceroute, traceroute6, true, truncate, tty, ttysize, uname, uncompress, unexpand, uniq, unix2dos, unlink, unlzma, unxz, unzip, usleep, uudecode, uuencode, vi, volname, watch, wc, wget, which, whoami, xargs, xxd, xz, xzcat, yes, zcat

I been able to test/fix most of them, they seems to work fine. One of the problems i found is that all tools working with a process are depending on the linux procfs. As for now i am just changing path from /proc to /compat/linux/proc/ and added message about linprocfs requirement to the port. However, for the real use this needs to be patched to use native BSD KVM API. Also most of the network interface related applets will needs to be ported to make network configurable. However, basic functionality works and i been able to create busybox-only jail with working networking tools, shell, etc. Statically contained busybox takes about 4Mb. May be it would be easier to take some of the tools missing from /rescue compared to the the busybox port, will see. Another step to complete is to get init working. After this point it should be possible to get FreeBSD kernel + busybox userland only as a running system. I will try to send all my BSD specific patches to upstream, will see if they would be accepted or not.

Testing in jail:

1. Install busybox with a STATIC option set
2. Create directory for the port. I am using /root/test for the example. Create some sub directories: mkdir -p /root/test/dev /root/test/bin. Copy busybox to the jail: cp /usr/local/bin/busybox /root/test/bin/
3. Create /etc/jail.conf:

testjail {
   path = /root/test;
   mount.devfs;
   host.hostname = testhostname;
   ip4.addr = 192.168.101.113;
   interface = ue0;
   exec.start = "/bin/busybox";
}

1. Finally run jail using jail -c testjail. You should see ash prompt! You can create all /bin links using busybox itself:

cd /bin
./busybox --list|./busybox xargs -n1 ./busybox ln -s busybox

To use network tools you will also need to create /etc/resolv.conf file with a dns server to use. E.g.

# echo nameserver 8.8.8.8 > /etc/resolv.conf
# wget google.com
Connecting to google.com (172.217.23.206:80)
Connecting to www.google.com (172.217.23.228:80)
index.html           100% |*************************************************| 11897  0:00:00 ETA
#

Comments and suggestions are welcome.

Finding “orphaned” packages on the FreeBSD

On some of my old FreeBSD systems i found that there are packages installed locally but already removed from the ports. As result – such packages can not be upgraded automatically and may cause security risk and problems on OS upgrades later.

I did not found any pkg command which allows to find such orphanes but found that its very easy to do in a simple one-liner:

pkg info --origin -a | \
awk '{print "ls /usr/ports/"$2 " > /dev/null 2>/dev/null || echo Origin not found: "$1}'| \
sh

Ports collection needs to be installed and updated before running it. Sample output provided below:

$ pkg info --origin -a|awk '{print "ls /usr/ports/"$2 " > /dev/null 2>/dev/null || echo Origin not found: "$1}'|sh
Origin not found: GeoIP-1.6.11
Origin not found: bind99-9.9.11P1_1
Origin not found: cdiff-1.0.3,1
Origin not found: libcheck-0.10.0
Origin not found: p5-Geo-IP-1.51
Origin not found: pecl-intl-3.0.0_11
Origin not found: php56-5.6.32_1
Origin not found: php56-bz2-5.6.32_1
Origin not found: php56-ctype-5.6.32_1
Origin not found: php56-curl-5.6.32_1
Origin not found: php56-dom-5.6.32_1
Origin not found: php56-exif-5.6.32_1
Origin not found: php56-fileinfo-5.6.32_1
Origin not found: php56-filter-5.6.32_1
Origin not found: php56-gd-5.6.32_1
Origin not found: php56-hash-5.6.32_1
Origin not found: php56-iconv-5.6.32_1
Origin not found: php56-json-5.6.32_1
Origin not found: php56-ldap-5.6.32_1
Origin not found: php56-mbstring-5.6.32_1
Origin not found: php56-mcrypt-5.6.32_1
Origin not found: php56-mysql-5.6.32_1
Origin not found: php56-mysqli-5.6.32_1
Origin not found: php56-openssl-5.6.32_1
Origin not found: php56-pdo-5.6.32_1
Origin not found: php56-pdo_mysql-5.6.32_1
Origin not found: php56-posix-5.6.32_1
Origin not found: php56-session-5.6.32_1
Origin not found: php56-simplexml-5.6.32_1
Origin not found: php56-wddx-5.6.32_1
Origin not found: php56-xml-5.6.32_1
Origin not found: php56-xmlreader-5.6.32_1
Origin not found: php56-xmlwriter-5.6.32_1
Origin not found: php56-xsl-5.6.32_1
Origin not found: php56-zip-5.6.32_1
Origin not found: php56-zlib-5.6.32_1
Origin not found: swig13-1.3.40_1

In example above it is clear that php56 needs to be replaces with a recent one, as well as few other packages.

How to support old OSX version with a recent xcode.

About

In this post i decided to share my experience about supporting build for old MacOSX versions using recent xcode/clang. I found that this topic is very tricky and documentation is not covering many parts of it.

How things should work

I am currently using OSX 10.14.1 as a build host with a latest Command Line Tools installed (by xcode-select --install). SDK 10.14.1 should support compilation for any OSX starting from 10.9 to the 10.14. We can specify version to use using -mmacosx-version-min= switch which is used by compiler/linker to generate correct binaries from the target system. Sounds brilliant! So lets test this and see why it is not always as good as it could be.

Testing with plain C code

Lets start with a simple code. I will use clock_gettime as an example. This POSIX function was added in OSX 10.12 and frequently used in opensource software. Here is a sample code i will use:

/*
 simple gettime test
 */
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <time.h>
#define BILLION  1000000000L;

int main( int argc, char **argv )
  {
    struct timespec start, stop;
    double accum;

    if( clock_gettime( CLOCK_REALTIME, &start) == -1 ) {
      perror( "clock gettime" );
      exit( EXIT_FAILURE );
    }
    sleep(1);

    if( clock_gettime( CLOCK_REALTIME, &stop) == -1 ) {
      perror( "clock gettime" );
      exit( EXIT_FAILURE );
    }

    accum = ( stop.tv_sec - start.tv_sec )
          + ( stop.tv_nsec - start.tv_nsec )
            / BILLION;
    printf( "%lf\n", accum );
    return( EXIT_SUCCESS );
}

So, lets start testing.

1. clang -Wall -o gettime gettime.c gives no errors and producing working binary. We can test minimal supported version with otool -l gettime|grep -A 4 'LC_BUILD_VERSION' – in our case it is 10.14.
2. Lets try to restrict version with 10.9. Command clang -mmacosx-version-min=10.9 -Wall -o gettime gettime.c silently producing output and gettime binary! Minimal osx version is now marked as 10.9. That is weird, but nm -a gettime shows reference to _clock_gettime symbol which should not exist on 10.9 at all! As expected, this binary fails if running on OSX 10.11:

       ./gettime
       dyld: lazy symbol binding failed: Symbol not found: _clock_gettime
         Referenced from: /Users/vagrant/./gettime
         Expected in: /usr/lib/libSystem.B.dylib
       dyld: Symbol not found: _clock_gettime
         Referenced from: /Users/vagrant/./gettime
         Expected in: /usr/lib/libSystem.B.dylib
       Trace/BPT trap: 5
   

   So despite the switch clang created broken binary!

3. It is possible to disable weak imports by using -no_weak_imports linker flag. Lets try it:

   clang -Wl,-no_weak_imports -mmacosx-version-min=10.9 -Wall -o gettime gettime.c
   

   will fail to link with an error:

   ld: weak import of symbol '_clock_gettime' not supported because of option: -no_weak_imports for architecture x86_64.
   

Sounds like a solution! Moreover – we can add -Werror=partial-availability key and if it is used – compilation will fail early, with gettime.c:17:9: error: 'clock_gettime' is only available on macOS 10.12 or newer [-Werror,-Wunguarded-availability] error which is explaining where the problem is. So, looks like we can just add -Werror=partial-availability to C(XX)FLAGS and -Wl,-no_weak_imports to the link flags to solve all the issues?
Not so easy 😦

Testing with autoconf or cmake

In the real world software is typically built using some build system, which trying to detect what is available on the host to set defines accordingly. In the opensource world autoconf and cmake are very popular choices. Now lets try to see if we can build some projects with a compatibility flags set to 10.9. As a real-world example i will use XZ Utils which uses autoconf as a build system.

1. Lets try to build XZ Utils 5.2.4 with default CFLAGS/LDFLAGS. Typical ./configure && make command should provide binary src/xz/.libs/xz with minimal supported OS set to 10.14. But out goal is to compile it for 10.9+, so lets do make clean
2. Now lets try to set LD/CFLAGS to enforce 10.9 compatibility:

   export CFLAGS="-mmacosx-version-min=10.9 -Werror=partial-availability"
   export LDFLAGS="-Wl,-no_weak_imports"
   

   and re-run ./configure && make. This time build will fail with ../../src/common/mythread.h:250:19: error: '_CLOCK_REALTIME' is only available on macOS 10.12 or newer [-Werror,-Wunguarded-availability] error message.

Looking in the source it is clear that HAVE_CLOCK_GETTIME is set despite the fact that we used correct compiler and linker flags. Why it happens? Due to autoconf detection algorithm. To detect if function is available its trying to compile and link such simple code:

#ifdef __cplusplus
extern "C"
#endif
char clock_gettime ();
int
main ()
{
return clock_gettime ();
  ;
  return 0;
}

As you could see – autoconf includes own function prototype (because it only wants to check that function exists in the system) and in this case all linker/preprocessor version selection logic in clang/osx will just never run because it depends on definitions from the SDK headers.

Lets check this in the command line:

clang -mmacosx-version-min=10.9 -Werror=partial-availability -Wl,-no_weak_imports -o test test.c

will return no errors, but resulted binary will link to the _clock_gettime symbol which will be not available on target OS. I found that cmake checks are behaving exactly the same, resulting with a broken binary or compilation/link error if appropriate flags are set.

Workarounds

If it is your own code which does not have million dependencies – probably easiest way is to just patch autoconf/cmake checks to not prototype this functions but use system headers instead. However, if you have a tonn of dependencies – it would take huge amount of time to patch every failed one. So i found another workaround.

If recent (10.14) SDK is not easy to use in out scenario with autoconf/cmake lets use one from 10.9 🙂 I downloaded and unpacked 10.9 SDK to the /Library/Developer/CommandLineTools/SDKs/MacOSX10.9.sdk directory. I found that it is enough to set SDKROOT environment variable to the MacOSX10.9.sdk directory to fix the problem. In this case SDK will not contain any definitions of the non-compatible symbols and autoconf/cmake checks will work as expected, so you will get 10.9+ compatible Mach-O binary in the end of the game.

Summary

It is possible, but not trivial to keep compatibility with an older systems with a recent clang/OSX.

If you know better solution or workaround or just found this post useful – please let me know in the comments.

MacBook Pro USB-C 10G interconnection

I just found that it is possible to connect 2 2016 Macbook Pro laptops with an USB-C cable. Just in case – one from power adapter will not work, but another one from the Lenovo monitor works just fine. Connection is named as Thunderbolt Bridge in the network preferences.

There is no connection speed in the ifconfig output (just “active” mark), so decided to measure it with iperf3 – it shows about 4.20 Gbits/sec with default MTU on TCP test. And when i set MTU to 9000 i finally got 10.4 Gbits/sec, what is amazing 🙂 So now i know fastest and very easy method to transfer data between new macbooks just by using usb-c data cable. Rare moments when i really like Apple engineering 🙂

Smartmontools SVN builds

Smartmontools SVN builds (Linux, Windows, FreeBSD and OSX) are now available at https://builds.smartmontools.org/. Builds are done using docker and CircleCI and should be 100% repeatable. Keep in mind that these builds are not releases – some functionality could be broken or not yet completed.

GIMP 2.10.6/OSX port released

You can download latest GIMP 2.10.6/OSX build from the github project page. Feel free to report any bugs. It will become an official download soon.

First release of the Gimp 2.10.2/OSX port

There is no [yet] official dmg package of the Gimp Open Source Image Editor for OSX, so i created a new one. Work is mostly done, there are few non-critical issues i am planning to fix meantime. Also my plans are to merge my patches with an upstream, so hopefully this would become official port at some point.

Testers are welcome. OSX build could be found on project releases page.

How my Mac Book Pro 2017 destroyed 2 USB-C devices

Bad things happens

At morning my new Macbook was not working well – when i been disconnecting charger it was not detecting this properly, so i decided to reset SMC, and this resolved the problem. In the office i connected my passive USB-C hub with HDMI/USB output and it starts to smell very bad. I was stupid enough to connect my USB-C ubikey to the same socket and it was destroyed immediately, with the same smell and some lights. Both devices were destroyed. MacBook was working like nothing happens.

I realized that i already saw post about it on Reddit and been able to find original post Macbook Pro frying USB peripherals. It was looking exactly as my case. So i decided to validate it.

Measuring USB-C power voltage

I do have usb-c->microusb adapter from some charger + microusb power to power adapter plug. I connected to cables to it and been able to measure the voltage, thats how it looks:

Now i connected it to the voltmeter and plugged in to the affected port with a usb-c charger plugged in a different port:

20V!!! instead of maximum 5 allowed. All other sockets were provided ~1v, probably that is idle current when no data lines are connected:

I tried different combinations, but only top right socket was failing. So it seems to be another “known issue” in this new model, in addition to very problematic keyboard.

Summary

I been using only original charger with original cable but got usb-c devices fried. According to the post i provided below – i am not alone with such issue. Tomorrow i will send this Mac to repair and hopefully replacement will not be affected. I think that Apple engineers should consider this issue as very serious one, but giving the fact how they handle keyboard-related issues it seems to be unlikely.

Fixing Google Music Manager on OSX

Sometime i am purchasing music on Google Music. Once i decided to download entire connection
and found that its not as easy as i exptected.

1. With a browser you can download only album by album + it warns that download
   counter is limited to 2 times only (WTF?!?)
2. Recommended way to download is “Google Music Manager” which should do everything automatically.

Okay google, lets try to install it. It installs fine, but download failing completely. On the web i found
that it was that way > 1 year already! (e.g. see this thread.

Also i tried to use Google support and it was as useless as possible.

So there is nothing to do but to try to fix it myself.

Starting debugging

Music Manager process name is MusicManagerHelper. This is QT4 (!) application.
Output of the lsof command shows that there is a log in the ~/Library/Logs/MusicManager/MusicManagerHelper.log.
Output shows that there is an error with certificate validation on every download attempt:

Failed to Verify Certificate: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.googleusercontent.com (unable to get local issuer certificate:0x00000000) [/Volumes/Android/buildbot/src/googleplex-android/jumper-stable/jumper/source/Shared/HttpClients/CurlHttpClientImpl.cpp:44 ::CurlSslVerifyCertificate()]

Ok, its better. Something i tried without any luck:

• Adding google certs to the system keychain with a trust settings
• Trying to find where its storing root certs (its compiled in)
• Trying to rebuild libQtNetwork 4 on OSX (seems to be completely broken with a recent compiler/sdk)

Finally, solving issue using LLDB

Using debugger i found that libQtNetwork contains statically compiled OpenSSL.
After all i been able to identify function responsible for the certificate validation – it is ssl_verify_cert_chain.
Lets try to use lldb to override it return value!

1. Starting lldb with lldb -p
2. Adding breakpoint with breakpoint set -n ssl_verify_cert_chain --shlib libQtNetwork.4.dylib command.
3. Attaching python script to the breakpoint:
   breakpoint command add –script-type python

   print "starting"
   thread = frame.GetThread()
   return_value =  lldb.frame.GetValueForVariablePath("*self")
   thread.ReturnFromFrame(frame, return_value)
   print "ending"
   lldb.debugger.GetSelectedTarget().process.Continue()
   DONE
   

4. Trying to download again. LLDB should print “starting” and “ending” on
   any download and it finally works!
   

Afterwords

• Google is an Evil. I spent a lot of time to just download content i paid for! And support was completely useless.
• Google Music Manager, at least on OSX, is insecure, broken and outdated application.
• Putting root certificates to the bundle is a very poor coding style.
  Not sure if it comes from QT or from Google itself, but doing that way is embedding timebomb to your app.
• Dtrace and LLDB are cool 🙂