Reading DHT11 temperature sensor on Raspberry Pi under FreeBSD

Connecting sensor to the RPi

DHT-11 is a very cheap temperature/humidity sensor which is commonly used in the IoT devices. It is not very accurate, so for the accurate measurement i would recommend to use DHT21 instead. Anyway, i had DHT-11 in my tool box, so decided to start with it. DHT-11 using very simple 1 wire protocol – host is turning on chip by sending 18ms low signal to the data output and then reading 40 bytes of data. Details about the protocol could be found in the specification. To read data from the chip it should be connected to the power (5v) and gpio pin. I used pin 2 as VCC, 6 as GND and 11 as GPIO (it is GPIO17, see pinout):
rpi_pinout

FreeBSD support

There is no support for this device out of the box. I found some sample code on the github, see lex/freebsd-gpio-dht11 repository. This code was a good starting point, but soon i found 2 issues with it:

  1. Results are very unreliable, probably due to gpio decoding algorithm.
  2. Checksum is not validated, so sometime values are bogus.

Initially i was thinking to fix this myself, but later found kernel module for this purpose, 1 wire over gpio. This module contains DHT11 kernel driver (gpio_sw) which implements DHT-11 protocol in the kernel space and exporting /dev/sw0 for the userland. Driver compiles on FreeBSD11/ARM without any changes. Use make install to install the driver.

Putting all the things together

  • To specify GPIO pin with a sensor put hint.gpio_sw.0.pin=17 into /boot/loader.conf.
  • I found that gpio_sw.ko needs to be loaded after kernel initialization, or device is not created. So i have added /sbin/kldload gpio_sw.ko to the /etc/rc.local file.
  • To make /dev/sw0 available for the non-root users you should add devfs_system_ruleset="localrules" to the /etc/rc.conf and add into /etc/devfs.rules this section:
[localrules=10]
add path 'sw0' mode 0644
  • Sample program (test.c) shows non human-readable data, use something like printf("h:%d.%d %%, t:%d.%d C\n",Buf[0],Buf[1],Buf[2],Buf[3]); if you want to get humidity and temperature, e.g.
root@rpi-b:/home/freebsd/gpio_sw # ./test
h:22.0 %, t:12.0 C
Hum=5632
Tem=3072

That is it, after reboot you should have working /dev/sw0 device.

Solving problems with LUA

Final goal was to add this sensor to the domoticz software. It is using LUA scripting to extend it functionality, e.g. to obtain data from non-supported or non standard devices. So, i decided to read /dev/sw0 from the LUA. I wrote simple test script:

file = io.open ("/dev/sw0", "rb")
out = file:read (5)
print(string.byte(out).."."..string.byte(out,2).."%;"..string.byte(out,3).."."..string.byte(out,4).."C")
file:close()

However script was always returning nil and i had to use truss tool to understand the problem. This is part of the trace:

open("/dev/sw0",O_RDONLY,0666)             = 3 (0x3)
fstat(3,{ mode=crw-r--r-- ,inode=96,size=0,blksize=4096 }) = 0 (0x0)
ioctl(3,TIOCGETA,0xbfbfe28c)             = 0 (0x0)
read(3,0x2065b000,4096)              ERR#22 'Invalid argument'

As you could see – LUA trying to read 4096 bytes, despite the fact that we specified 4. And driver checks this and returns the error. To fix this i did a small patch to avoid error and make LUA happy:

-- gpio_sw/gpio_sw.c    2014-05-12 11:26:51.000000000 +0000
+++ gpio_sw.mod/gpio_sw.c   2017-01-14 14:10:33.736813000 +0000
@@ -273,9 +273,10 @@
 {
   duprintf("read - start, uio_resid=%i\n", uio->uio_resid);
   struct gpio_sw_softc *sc = cdev->si_drv1 ;
-  if ( uio->uio_resid >= sc->BufSize) return EINVAL ;
+  if ( uio->uio_resid >= sc->BufSize) sc->Len=sc->BufSize-1 ; // to work with buffered IO
+  else sc->Len=uio->uio_resid;
+
   if ( ! sc->GpioStatus) return ENXIO ;
-  sc->Len = uio->uio_resid ;
   int i = 0 ;
   while ( i < sc->Len)
   {

After modification script works as it should:

[root@rpi-b /home/freebsd]# lua52 test.lua
22.0%;12.0C

Now chip is connected to the domoticz and reporting actual temperature/humidity.

Tagged , , ,

Domoticz on RPi 1 using FreeBSD/ARM

FreeBSD, Rasberry and Domoticz

I decided to create a Home Automation controller from my old Raspberry 1 device. In the past i used Linux on it but now it is supported by FreeBSD/ARM, so i decided to give it a try.

Installing FreeBSD

This is an easiest part. RPi 1 is fully supported, so i just downloaded FreeBSD11 image and put it on the SD card using dd tool. That`s it, FreeBSD boots normally (no installation process required) and is ready to use. I only added ntpd/sshd to the autostart process.

Installing domoticz

Domoticz is an open source home automation system, written on C++. There is a FreeBSD port for it, so first thing i tried to do was to install binary package using pkg tool. Unfortunately there is no binary package, so i had to install ports. To save some time before starting domoticz compilation i would recommend to install all build requirements using pkg manager, compiling them on RPi 1 would take a lot of time. Some tips on getting domoticz compiled on RPi 1:

  • Clang/C++ is using a lot of memory, so swap is required. I added swap space on the connected USB storage, with 512Mb swap file.
  • Compilation process will take a lot of hours. Probably cross-compilation would be much faster, but then it wont be a normal port build. Anyway, this process is a good crash test for the hardware.
  • I found that build failed on a 2 files: hardware/I2C.cpp and hardware/PiFace.cpp. This is the reason why there is no binary package. To fix the build – replace __arm__ with __linuxarm__. This will fix the build process, issue is reported to the upstream and FreeBSD maintainer.

Using domoticz

I did not found anything platform-specific for the domoticz on the FreeBSD. After installation i only imported database from the OSX installation and copied my LUA scripts for the custom sensors + installed required LUA modules from the ports. Despite the ugly build performance domoticz works very fast and using just about 25Mb of RAM. Currently I2C support in the hardware/I2C.cpp is implemented only for Linux/ARM, i am planning to fix that in a few weeks (i do not have any supported i2c devices now). FreeBSD/ARM itself works very stable, i did not had any problems with it.

Tagged , ,

PPTP on OSX Sierra

After upgrading to OSX Sierra (10.12.2) my PPTP connections just disappeared. Also it is not possible to create new PPTP connections anymore. Yes, i know that PPTP protocol is old, could be insecure, but i still need it to connect to some legacy systems. And, after all, i don`t like when someone telling me what should i do 🙂

After quick research it was found that actually only GUI part is removed, but PPTP is still in the Darwin internals. You can use it using pppd tool from the command line (using sudo). Configuration should be placed into /etc/ppp/peers/ directory, e.g. /etc/ppp/peers/pptpvpn. Here is a sample config, which works for me:

        plugin PPTP.ppp
        noauth
#        logfile /tmp/ppp.log
        remoteaddress gwvpn.example.com
        redialcount 1
        redialtimer 5
        idle 1800
        mru 1368
        mtu 1368
        receive-all
        novj 0:0
        ipcp-accept-local
        ipcp-accept-remote
        #noauth
        refuse-pap
        refuse-chap-md5
        user userid
        hide-password
        mppe-stateless
        mppe-128
        looplocal
        password userpw
        nodetach
        ms-dns 8.8.8.8
        # used in ip-up script
        ipparam gwvpn 

To connect to the VPN use pppd call pptpvpn command. Use man pppd for the possible options in the config. Hopefully it will still be there some time 🙂

Tagged , , ,

Check how your web site looks on IPv6 only connection

Today i found great resource for the IPv6 adopters. NAT64 Check website checks and compares website look and feel in NAT64, IPv4-only and IPv6-only modes. This way i been able to find that on my of the my web sites IPv6 was not configured correctly (browser was automatically falling back to v4, so i did not found that myself). Another interesting issue is some external scripts hosted on public CDN-s. Some of them do not have IPv6, e.g. very popular maxcdn.bootstrapcdn.com.

Service works by downloading all resources in 3 different network environments. Also it is trying to compare rendered image of the web site to find, for example, broken geo based lookup or other issues, like incorrect virtual host configuration which will cause different rendering result.

Thats how it looks if everything is fine:

screen-shot-2016-11-28-at-21-45-30

So, thank you Jan Žorž, and lets make more websites IPv6 ready 🙂

Tagged ,

‘MySQL server gone away’ error, keepalive and PHP

One of the favorites errors for the many MySQL users is “mysql server gone aways”. It provides minimal data to debug the issue, is very annoying and could be caused by number of reasons. Most popular are:

  • Internal MySQL timeouts (configurable, could be found using “get variable …’
  • MySQL max_allowed_packet limit
  • Networking layer
  • Probably some other mysterious reasons.

In this post i would cover networking layer, because i think it is hard to diagnose and very common. Of course, before jumping to it you should check mysql logs and timeouts to make sure that they are not a root cause. In my case all MySQL server timeouts were good, but application was still failing after running very large tasks. Using strace and tcpdump i found the reason.

  1. App was opening connection to MySQL.
  2. After this some slow code was running, during this time MySQL connection was not in use.
  3. In 1 hr (code was still running) remote side was closing connection.
  4. When app was finally done and trying to reuse connection – it was failing because connection was already closed.

Of course app is also guilty – slow code, no good connection handling, but my task was to find some quick workaround. First question was why connection was closed in 1Hr. Reason was Amazon ELB (load balancer) which was in use. It has idle timeout, which could be configured up to 3600 seconds.Such situation is not uncommon – many routers or balancers do have some idle connection timeouts. So i decided to find why there were no keepalive packets sent. And found that it was a bug of the PHP, which is fixed in the recent version (5.6). After updating php to the recent version i been able to find that keepalive is working. You can do this using netstat -to tool or using tcpdump. Thats how keepalive packet looks like:

04:46:55.034560 IP 10.0.21.105.39853 &gt; 10.0.52.203.mysql: Flags [.], ack 7907432, win 3086, options [nop,nop,TS val 3156463616 ecr 488286410], length 0
04:46:55.035543 IP 10.0.52.203.mysql &gt; 10.0.21.105.39853: Flags [.], ack 9089476, win 136, options [nop,nop,TS val 488361560 ecr 3156163057], length 0

So now Linux will send keepalive packets regularly and this not allow ELB to close idle mysql connection.

Tagged

Creating list of the folders on big NFS mount

Today i found that on one of my NFS shared (140+ Gb of small files) some directory permissions are set in a wrong way. I decided to fix all of them using find /nfsmount -type d|xargs chmod 755. However after running find command i found that it painfully slow and would take enormous amount of time. After using strace reason was found: it is using newfstatat call for the every file in the directory. On NFS it would take a lot of time. Below is an example from the strace:

newfstatat(7, &quot;s00594.jpg&quot;, {st_mode=S_IFREG|0660, st_size=152035, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;68394s4.jpg&quot;, {st_mode=S_IFREG|0660, st_size=221090, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;fg120lgb.jpg&quot;, {st_mode=S_IFREG|0660, st_size=12910, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;86240_1_low.jpg&quot;, {st_mode=S_IFREG|0660, st_size=14544, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;am1051-30cm.jpg&quot;, {st_mode=S_IFREG|0660, st_size=10091, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;46939-9.jpg&quot;, {st_mode=S_IFREG|0660, st_size=121149, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;22293(11).jpg&quot;, {st_mode=S_IFREG|0660, st_size=139348, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;51643(1).jpg&quot;, {st_mode=S_IFREG|0660, st_size=163897, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;53868t(2).jpg&quot;, {st_mode=S_IFREG|0660, st_size=17221, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;40419_t.jpg&quot;, {st_mode=S_IFREG|0660, st_size=4247, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;36116t.jpg&quot;, {st_mode=S_IFREG|0660, st_size=17580, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;89239_1_high(6).jpg&quot;, {st_mode=S_IFREG|0660, st_size=88781, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;61392-1(1).jpg&quot;, {st_mode=S_IFREG|0660, st_size=67362, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;98872_1_low(1).jpg&quot;, {st_mode=S_IFREG|0660, st_size=26867, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;31269.jpg&quot;, {st_mode=S_IFREG|0660, st_size=60999, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;s00437t.jpg&quot;, {st_mode=S_IFREG|0660, st_size=7505, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;30813-sub7(2)(4).jpg&quot;, {st_mode=S_IFREG|0660, st_size=155762, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, &quot;118b-090_t.jpg&quot;, {st_mode=S_IFREG|0660, st_size=4019, ...}, AT_SYMLINK_NOFOLLOW) = 0

Interesting thing that i have no idea why find is doing that. There is a getdents call which provides all required information for the entire directory without need to access every file. I found that this syscall works well on NFS, so i created my own tool to list this mount quickly. After it was done i got directory listing in less than 5 minutes! My tool is provided on the github, however it is interesting why GNU find tool itself is so non-optimal.

Tagged ,

ICMP Watchdog in the Ubiquiti Networks devices

About watchdog

I am using wireless devices from the Ubiquiti Networks. Usually everything works fine, but in rare cases of software/hardware bug it would be great to automatically restart device when needed. AirOS provides this functionality, it is called “ping watchdog” and is located in the web interface, “services” tab. However there is no a lot of documentation about how it works, so i decided to research this. Screenshot of the watchdog interface with default values provided below: Screen Shot 2016-07-18 at 08.38.52.

Under the hood

Ubnt AirOS is OpenWRT based OS with ssh enabled, so we can ssh to the device to find how this watchdog works. If ping watchdog is enabled in the web interface you should see something like this in the process list:

/bin/pwdog -d 300 -p 300 -c 3 -m 300 -e /bin/support /tmp/emerg /etc/persistent/emerg.supp emerg 0; reboot -f 192.168.1.1

This “pwdog” service is a custom busybox applet which is based on busybox ping implementation with modifications to implement watchdog functionality. I been able to find it source code on the github.

So there is detailed description of the pwdog service logic:

  1. On system start it waits -d seconds (300 by default), to allow initialization of the hardware and software. I would not recommend to reduce this value, or you will have a chance that device will never start. In the web interface it is “Startup Delay:” value.
  2. After initial delay it will send ICMP ping to the specified host (last parameter) and will wait -p seconds (300 by default, “Ping Interval:” in the web interface). After this step 2 will be repeated.
  3. If there is no reply -c times (by default – 3) pwdog will run command specified in the -e argument (/bin/support /tmp/emerg /etc/persistent/emerg.supp emerg 0; reboot) or just reboot if it is not specified. In this example watchdog also saves support information. In the web interface you can modify this value using “Failure Count To Reboot.:” parameter.
  4. There is also -m parameter which defines low memory threshold. It is enabled by default and is not configurable via web interface.

Below i tested how it works in the command line, with modified parameters:

XM.v5.6.6# /bin/pwdog -d 1 -p 3 -c 3 -m 300 -e /usr/bin/echo -v 192.168.1.1
pwdog[993]: pwdog: do_now=0, initial_sleep=1, timeout=3, retry_count=3, low_mem=300 exec=`/usr/bin/echo`
pwdog[993]: PING Watchdog is checking 192.168.1.1 (192.168.1.1).
pwdog[993]: Missed 1 ping replies in a row.
pwdog[993]: Missed 2 ping replies in a row.
pwdog[993]: Missed 3 ping replies in a row.
pwdog[993]: 4 ping replies missed. Executing `/usr/bin/echo`.

Conclusion

ICMP watchdog in AirOS is not a very smart service and default configuration does not look optimal for me – in fact its enough to miss only 3 ICMP packets to start reboot process. Also it will fire only after 15 (300*3) minutes of the link failure. So i would probably recommend to increase number of counts and decrease ping interval. Also i am thinking about porting apinger to this device, because it provides much more advanced icmp check functionality.

Tagged , , ,

Q&D perl script to check balance on the mujkaktus.cz GSM prepaid

I found that one of the best options for the GSM trackers in CZ is mujkaktus card. It allows per-traffic tarification with a very reasonable rates. Also it is anonymous and no contract required. To automatically check balance on the card i created a small perl script which i decided to share. I am planning to integrate it with Nagios to alert if balance is low.

#!/usr/bin/perl

use strict;
require LWP::UserAgent;

# username/password
my $username = 'example@example.com';
my $password = 'example';

binmode STDOUT, ":utf8"; # suppress UTF-8 warnings
$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0; # disable ssl checking

# Create a request
my $req = HTTP::Request->new(POST => 'https://www.mujkaktus.cz/.gang/login');
$req->content_type('application/x-www-form-urlencoded');
$req->header('Cookie' => 'COOKIE_SUPPORT=true'); # required
$req->content('username='.$username.'&password='.$password.'&submit=P%C5%99ihl%C3%A1sit');
my $ua = LWP::UserAgent->new;
$ua->cookie_jar( {} ); # 
my $response = $ua->request($req); 
# status line should be '302 Found' if password is correct
die "Unexpected http response, check login/password\n" if ($response->status_line ne "302 Found");

# we will not follow redirect, but fetch "moje-sluzby" page
$req = HTTP::Request->new(GET => 'https://www.mujkaktus.cz/moje-sluzby');
my $response = $ua->request($req);
if ($response->decoded_content =~ m/stav kreditu<\/h3><div class=\"box-format\"><div id=\"[^"]+\"><p><span class="text-1">([0-9,]+)/i) {
	print "Balance: $1 CZK\n";
}
else {
	die "Unable to fetch balance data\n";
}

P.S. i found that there are in fact 2 web interfaces – old, legacy one, which i am using and a new one, AJAX based. To get old web interface you should disable Javascript in your browser. Hopefully they will keep this legacy interface for some time, AJAX based is much more complicated and will require more efforts to get data.

Tagged , ,

Using TK103A GPS tracker with traccar server

TK-103A tracker

Some time ago i decided to install on my car GPS tracker to get information about my routes, car location, etc. After quick research i found “Mini TK103A” tracker on the eBay, which is costs about 30$.

s-l1600

Device looks solid and can be configured by sms commands. Most important are “begin123456” (initialization), “admin123456 (adds numbers to the trusted list) and adminip (gprs settings). Full command list is provided in the documentation.

“USB” port

Tracker do have micro-USB socket, however it is not real USB, it is just serial port soldered on microusb plug. I been able to get information from it using USB-Serial TTL converter. It sends a lot of debug information on 115200/8N1 speed. Debugging information is useful when you configuring and testing the tracker.

04-26 15-63-40  EINT PWR CONNECT
04-26 15-63-40  motion_close
01-15 00-00-00  SENDDATA:0
01-15 00-00-00  NO SERVICE
01-15 00-00-00  T-card not ready!
01-15 00-00-00  FILE2222:
                         01-15 00-00-00:

01-15 00-00-00  password1=:123456
01-15 001-15 00-00-00  CENTER NUMER1:+420123123123
01-15 00-00-00  CENTER NUMER2:
01-15 00-00-00  CENTER NUMER3:
01-15 00-00-00  CENTER NUMER4:
01-15 00-00-00  CENTER NUMER5:
01-15 00-00-00  heartbeat time:3
01-15 00-00-00  SENDDATA:0
01-15 00-00-00  send Packet time:15
01-15 00-00-00  sms_gprs=1
01-15 00-00-00  time_zone:2,8,0
01-15 00-00-00  voice_temp:1
01-15 00-00-00  shave alarm:0,35
01-15 00-00-00  ACC:0
01-15 00-00-00  speed alarm:0,120
01-15 00-00-00  speed alarm time:5
01-15 00-00-00  s alarm time:5
01-15 00-00-00  move alarm=0
01-15 00-00-00  JT=0
01-15 00-00-00  JT TIME=3
01-15 00-00-00  TRACE :2
01-15 00-00-00  lang=1
01-15 00-00-00  APN=1
01-15 00-00-00  ���ϴ�ʱ��:1
01-15 00-00-00  powr=1
01-15 00-00-00  weilan:0
01-15 00-00-00  num:255
01-15 00-00-00  loud_spe=1
01-15 00-00-04  NO SERVICE
01-15 00-00-04  NO SERVICE
01-15 00-00-04  NO SERVICE
01-15 00-00-04  NO SERVICE
01-15 00-00-04  NO SERVICE
01-15 00-00-05  NO SERVICE
01-15 00-00-07  NETWORK NORMAL
01-15 00-00-07  NETWORK NORMAL
01-15 00-00-10  T-card not ready!
01-15 00-00-10  FILE2222:
                         01-15 00-00-10:

01-15 00-00-12   IMEI��:352887072123123
01-15 00-00-12   IP/PORT:1.2.3.4/9000
01-15 00-00-12   VER:MAUI.10A.W11.08.MP.V25 2015/09/11 12:38
01-15 00-00-12   ---------------------------------------------------
01-15 00-00-12   SIM CARD------------------OK!
01-15 00-00-12   GSM Signal----------------OK!
01-15 00-00-12   SOCKET----------------NG
01-15 00-00-12   G-Senser------------------OK!
01-15 00-00-12   GPS Location----------NG
01-15 00-00-12   PWR EINT--------------NG
01-15 00-00-12   ACC EINT--------------NG
01-15 00-00-12   SOS EINT--------------NG
01-15 00-00-12   BATTER/Vin-----------4.11/11.97
01-15 00-00-12   ---------------------------------------------------
01-15 00-00-12   GPS Location:86,Satellite:2-----------
01-15 00-00-12  num:255

I also found some references that this port can be used to reflash the tracker, however i never tried that.

Sending data to the server

After GPRS host/port configuration you can enable GPRS mode where all data will be sent to the remote server. I found that OpenSource TracCar software supports such devices and provides web+android interfaces. It was found that this specific tracker using GT06 binary protocol. Traccar supports it out of the box, you just have to choose correct port on the server/client. Traccar also supports data logging to the external database (MySQL, Pg, etc.), so it should be easy to integrate it with anything you need.

Some security considerations

All data from tracker to the monitoring system is sent unencrypted and can be easily decoded on transit if traffic is captured. This device also allows to add some “security alarm” features, including ignition and oil pump control. I personally feel that it is very dangerous and should not be used at all. I think such features are good example of the InternetOfShit coming 🙂

Tagged , , , ,

Let`s Encrypt is now public beta :)

Starting from today Let’s Encrypt service is a public beta 🙂 This mean you can create certificates trusted by most of the browsers right now. So it is a perfect time to create SSL certificates for all your TLS services and get rid of self-signed certificates. Also it will help to switch to the new HTTP/2 protocol.

Only few things i am missing – non-python client without tonn of half-implmented features and some similar s-mime solution. First thing should be easy to implement, its just a matter of time, protocol is open and easy to understand.