I am using wireless devices from the Ubiquiti Networks. Usually everything works fine, but in rare cases of software/hardware bug it would be great to automatically restart device when needed. AirOS provides this functionality, it is called “ping watchdog” and is located in the web interface, “services” tab. However there is no a lot of documentation about how it works, so i decided to research this. Screenshot of the watchdog interface with default values provided below: .
Under the hood
Ubnt AirOS is OpenWRT based OS with ssh enabled, so we can ssh to the device to find how this watchdog works. If ping watchdog is enabled in the web interface you should see something like this in the process list:
/bin/pwdog -d 300 -p 300 -c 3 -m 300 -e /bin/support /tmp/emerg /etc/persistent/emerg.supp emerg 0; reboot -f 192.168.1.1
This “pwdog” service is a custom busybox applet which is based on busybox ping implementation with modifications to implement watchdog functionality. I been able to find it source code on the github.
So there is detailed description of the
pwdog service logic:
- On system start it waits
-dseconds (300 by default), to allow initialization of the hardware and software. I would not recommend to reduce this value, or you will have a chance that device will never start. In the web interface it is “Startup Delay:” value.
- After initial delay it will send ICMP ping to the specified host (last parameter) and will wait
-pseconds (300 by default, “Ping Interval:” in the web interface). After this step 2 will be repeated.
- If there is no reply
-ctimes (by default – 3) pwdog will run command specified in the
/bin/support /tmp/emerg /etc/persistent/emerg.supp emerg 0; reboot) or just reboot if it is not specified. In this example watchdog also saves support information. In the web interface you can modify this value using “Failure Count To Reboot.:” parameter.
- There is also
-mparameter which defines low memory threshold. It is enabled by default and is not configurable via web interface.
Below i tested how it works in the command line, with modified parameters:
XM.v5.6.6# /bin/pwdog -d 1 -p 3 -c 3 -m 300 -e /usr/bin/echo -v 192.168.1.1 pwdog: pwdog: do_now=0, initial_sleep=1, timeout=3, retry_count=3, low_mem=300 exec=`/usr/bin/echo` pwdog: PING Watchdog is checking 192.168.1.1 (192.168.1.1). pwdog: Missed 1 ping replies in a row. pwdog: Missed 2 ping replies in a row. pwdog: Missed 3 ping replies in a row. pwdog: 4 ping replies missed. Executing `/usr/bin/echo`.
ICMP watchdog in AirOS is not a very smart service and default configuration does not look optimal for me – in fact its enough to miss only 3 ICMP packets to start reboot process. Also it will fire only after 15 (300*3) minutes of the link failure. So i would probably recommend to increase number of counts and decrease ping interval. Also i am thinking about porting apinger to this device, because it provides much more advanced icmp check functionality.