Tag Archives: linux

Creating list of the folders on big NFS mount

Today i found that on one of my NFS shared (140+ Gb of small files) some directory permissions are set in a wrong way. I decided to fix all of them using find /nfsmount -type d|xargs chmod 755. However after running find command i found that it painfully slow and would take enormous amount of time. After using strace reason was found: it is using newfstatat call for the every file in the directory. On NFS it would take a lot of time. Below is an example from the strace:

newfstatat(7, "s00594.jpg", {st_mode=S_IFREG|0660, st_size=152035, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "68394s4.jpg", {st_mode=S_IFREG|0660, st_size=221090, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "fg120lgb.jpg", {st_mode=S_IFREG|0660, st_size=12910, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "86240_1_low.jpg", {st_mode=S_IFREG|0660, st_size=14544, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "am1051-30cm.jpg", {st_mode=S_IFREG|0660, st_size=10091, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "46939-9.jpg", {st_mode=S_IFREG|0660, st_size=121149, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "22293(11).jpg", {st_mode=S_IFREG|0660, st_size=139348, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "51643(1).jpg", {st_mode=S_IFREG|0660, st_size=163897, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "53868t(2).jpg", {st_mode=S_IFREG|0660, st_size=17221, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "40419_t.jpg", {st_mode=S_IFREG|0660, st_size=4247, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "36116t.jpg", {st_mode=S_IFREG|0660, st_size=17580, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "89239_1_high(6).jpg", {st_mode=S_IFREG|0660, st_size=88781, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "61392-1(1).jpg", {st_mode=S_IFREG|0660, st_size=67362, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "98872_1_low(1).jpg", {st_mode=S_IFREG|0660, st_size=26867, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "31269.jpg", {st_mode=S_IFREG|0660, st_size=60999, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "s00437t.jpg", {st_mode=S_IFREG|0660, st_size=7505, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "30813-sub7(2)(4).jpg", {st_mode=S_IFREG|0660, st_size=155762, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(7, "118b-090_t.jpg", {st_mode=S_IFREG|0660, st_size=4019, ...}, AT_SYMLINK_NOFOLLOW) = 0

Interesting thing that i have no idea why find is doing that. There is a getdents call which provides all required information for the entire directory without need to access every file. I found that this syscall works well on NFS, so i created my own tool to list this mount quickly. After it was done i got directory listing in less than 5 minutes! My tool is provided on the github, however it is interesting why GNU find tool itself is so non-optimal.

Advertisements
Tagged ,

ICMP Watchdog in the Ubiquiti Networks devices

About watchdog

I am using wireless devices from the Ubiquiti Networks. Usually everything works fine, but in rare cases of software/hardware bug it would be great to automatically restart device when needed. AirOS provides this functionality, it is called “ping watchdog” and is located in the web interface, “services” tab. However there is no a lot of documentation about how it works, so i decided to research this. Screenshot of the watchdog interface with default values provided below: Screen Shot 2016-07-18 at 08.38.52.

Under the hood

Ubnt AirOS is OpenWRT based OS with ssh enabled, so we can ssh to the device to find how this watchdog works. If ping watchdog is enabled in the web interface you should see something like this in the process list:

/bin/pwdog -d 300 -p 300 -c 3 -m 300 -e /bin/support /tmp/emerg /etc/persistent/emerg.supp emerg 0; reboot -f 192.168.1.1

This “pwdog” service is a custom busybox applet which is based on busybox ping implementation with modifications to implement watchdog functionality. I been able to find it source code on the github.

So there is detailed description of the pwdog service logic:

  1. On system start it waits -d seconds (300 by default), to allow initialization of the hardware and software. I would not recommend to reduce this value, or you will have a chance that device will never start. In the web interface it is “Startup Delay:” value.
  2. After initial delay it will send ICMP ping to the specified host (last parameter) and will wait -p seconds (300 by default, “Ping Interval:” in the web interface). After this step 2 will be repeated.
  3. If there is no reply -c times (by default – 3) pwdog will run command specified in the -e argument (/bin/support /tmp/emerg /etc/persistent/emerg.supp emerg 0; reboot) or just reboot if it is not specified. In this example watchdog also saves support information. In the web interface you can modify this value using “Failure Count To Reboot.:” parameter.
  4. There is also -m parameter which defines low memory threshold. It is enabled by default and is not configurable via web interface.

Below i tested how it works in the command line, with modified parameters:

XM.v5.6.6# /bin/pwdog -d 1 -p 3 -c 3 -m 300 -e /usr/bin/echo -v 192.168.1.1
pwdog[993]: pwdog: do_now=0, initial_sleep=1, timeout=3, retry_count=3, low_mem=300 exec=`/usr/bin/echo`
pwdog[993]: PING Watchdog is checking 192.168.1.1 (192.168.1.1).
pwdog[993]: Missed 1 ping replies in a row.
pwdog[993]: Missed 2 ping replies in a row.
pwdog[993]: Missed 3 ping replies in a row.
pwdog[993]: 4 ping replies missed. Executing `/usr/bin/echo`.

Conclusion

ICMP watchdog in AirOS is not a very smart service and default configuration does not look optimal for me – in fact its enough to miss only 3 ICMP packets to start reboot process. Also it will fire only after 15 (300*3) minutes of the link failure. So i would probably recommend to increase number of counts and decrease ping interval. Also i am thinking about porting apinger to this device, because it provides much more advanced icmp check functionality.

Tagged , , ,

Lets encrypt!

Some time ago i subscribed to the Lets Encrypt beta participation program. Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit.
Yesterday i finally get an email, telling that my requested domains (smartmontools.org/www.smartmontools.org) are white-listed. So i decided to give it a try.

Installation

To use Lets Encrypt! service you will have to install ACME protocol client. ACME (Automatic Certificate Management Environment) is a protocol to automate all operations with PKI certificates. Current implementation is written on Python language and available on the github repository. You could find a lot of information about it usage in the online manual. I was trying to run it on the CentOS 6.7 and installation failed because of old (2.6) Python version. However, after some research, i been able to find a pull request with a patch for the 2.6 support. Hopefully it will go into mainline at some point, because py26 is still widely used. After this i been able to complete installation with letsencrypt-auto.

Usage

Lets Encrypt require you to verify that you own requested domain, as most other CA do. However with ACME this cold be done 100% automatically. There are different options on how to do this, initially i tried --standalone option. With it letsencrypt client creates standalone webserver for the authentication. However if you already have web server on port 80 you will have to stop it when client is running. It was working for me, but it requires short downtime, so i decided to look on other options. After all i found webroot authenticator, which allows to just create some files in the web root and later automatically removes them. To automate the process i created configuration file /etc/letsencrypt/cli.ini:

# This is an example of the kind of things you can do in a configuration file.
# All flags used by the client can be configured here. Run Let's Encrypt with
# "--help" to learn more about the available options.

# Use a 2048 or 4096 bit RSA key
rsa-key-size = 2048

# Use production server
server =  https://acme-v01.api.letsencrypt.org/directory

# Uncomment and update to register with the specified e-mail address
email = nospam@example.com

# Uncomment to use a text interface instead of ncurses
text = True

# Uncomment to use the standalone authenticator on port 443
# authenticator = standalone
# standalone-supported-challenges = dvsni

# Uncomment to use the webroot authenticator. Replace webroot-path with the
# path to the public_html / webroot folder being served by your web server.
authenticator = webroot
webroot-path = /var/www/html/smartmontools/static

# automatically agree with license
agree-dev-preview = True

# renew certificate if it is already exists
renew-by-default = True

I also had to make sure that nginx can provide required files to the remote, so i added such lines in my nginx site configuration:

    location /.well-known/acme-challenge/ {
        alias /var/www/html/smartmontools/static/.well-known/acme-challenge/;
    }

To use certificates in nginx i added path to the new certificates and key to the configuration:

    ssl_certificate /etc/letsencrypt/live/smartmontools.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/smartmontools.org/privkey.pem;

Now to re-new my certificates i just need to run

./letsencrypt-auto -d smartmontools.org -d www.smartmontools.org certonly

command and it will do the job. Also dont forget to reload nginx service if certificates are already configured.

Notes

LetsEncrypt certificates will expire in a 90 days, so it is recommended to renew them every 60 days. Also it is very recommended to setup nagios check to send an alert if expiration time is less then one week. In the feature i would also try to use ACME client on the OpenWRT box, but hopefully there will be some more suitable alternative for the embedded hardware. Finally i would recommend to test your web server SSL configuration with SSL Server Test from SSL Labs.

Tagged , , ,

Smartmontools daily builds

Sometime i need to audit some servers and often smartmontools is very old, not installed at all (and repositories are broken) or not working for some reasons. Thats one of the reasons why http://builds.smartmontools.org was created. You can download latest SVN builds for the following systems:

  • Darwin (OSX) package, Mach-O universal binary with 2 architectures: i386+x86_64
  • Win32 installer (32 and 64 bit)
  • Linux: i686,x86_64,static and dynamic
  • Source code

Service is now in “experimental” status, please report any issues with it here or on https://smartmontools.org.

Tagged , , , ,

LXC on OpenWRT/Turris presentation

Slides from my presentation @ Turris Evening by cz.nic about LXC in OpenWRT/Turris. Video will follow soon, if you are interested.

 

Update: video from the presentation:

Tagged , , ,

Monitoring WAN status on OpenWRT using Alarm Pinger

Idea

I am connected to the Internet using wireless link which is sometime not very stable. I decided to monitor status of the link to make sure that I am aware of the problem. Initially i tried to monitor link with Monit or Nagios + fping, but results were not very good, this software is not designed for continues monitoring with very small interval. So I decided to find some alternatives.

About Alarm Pinger

I was using Alarm Pinger (apinger) with pfSense distribution — it was used to monitor WAN links to switch between them if needed.

Alarm Pinger (apinger) is a little tool which monitors various IP devices by simple ICMP echo requests. There are various other tools, that can do this, but most of them are shell or perl scripts, spawning many processes, thus much CPU-expensive, especially when one wants continuous monitoring and fast response on target failure. Alarm Pinger is a single program written in C, so it doesn’t need much CPU power even when monitoring many targets with frequent probes. Alarm Pinger supports both IPv4 and IPv6.

This tool supports multiply monitoring targets, external scripts, email notification, daemon mode. Only problem was that tool was not available as OpenWRT package. So i decided to port it.

OpenWRT port

After few tests I found, that code can be compiled with only few minor patches (autoconf related). You can grab Makefile for package from this pull request. Hopefully it will be integrated in the official packages feed soon. Update: port merged.
Port provides init.d script and sample configuration. In the feature I am also planning to make Luci integration to show link status from the web interface.

To buid package on Turris I would recommend to use my turris buildroot docker image.

## Service configuration

I am using very simple configuration to monitor status of the Wireless link using pings to the ISP gateway:

# we need to use root because "rainbow" tool fails to work from other uid. 
user "root"
group "root"

# status file with link quality information
status {
    file "/tmp/apinger.status"
    interval 1s
}
# command to run, with alarm type and reason
# if used with multiply targets %t needs to be added
alarm default {
    command on "/root/gateway.sh %A %r"
    command off "/root/gateway.sh %A %r"
}
# This alarm will be fired when target doesn't respond for 30 seconds.
alarm down "down" {
    time 30s
}
# This alarm will be fired when responses are delayed more than 80ms
# it will be canceled, when the delay drops below 50ms
alarm delay "delay" {
    delay_low 50ms
    delay_high 80ms
}
# This alarm will be fired when packet loss goes over 5%
# it will be canceled, when the loss drops below 3%
alarm loss "loss" {
    percent_low 3
    percent_high 5
}
target default {
    interval 1s
    avg_delay_samples 10
    avg_loss_samples 50
    avg_loss_delay_samples 20
    alarms "down","delay","loss"
}
# ISP Gateway host to monitor. You can define many targets in case of MultiWAN. 
target "1.2.3.4" {
    description "ISP Gateway"
}

Also I am using simple script to change WAN LED color in case of problems:

#!/bin/sh

DEF_COLOR=33FF33 # see https://gitlab.labs.nic.cz/turris/rainbow/blob/master/turris.c
WARNING_COLOR=FFFF00 # yellow
DOWN_COLOR=red
RAINBOW=/usr/bin/rainbow

logger "event: $@"
# read data from status file
STATUS=`grep  "Active alarms:" /tmp/apinger.status`

case "$@" in
"delay ALARM")
  touch /tmp/apinger.delay.flag
  ;;
"delay alarm canceled")
  rm -f /tmp/apinger.delay.flag
  ;;
"down ALARM")
  touch /tmp/apinger.down.flag
  ;;
"down alarm canceled")
  rm -f /tmp/apinger.down.flag
  ;;
"loss ALARM")
  touch /tmp/apinger.loss.flag
  ;;
"loss alarm canceled")
  rm -f /tmp/apinger.loss.flag
  ;;
esac
# link is down
if [ -e /tmp/apinger.down.flag ]; then
  ${RAINBOW} wan ${DOWN_COLOR}
  exit
fi
# loss or delay
if [ -e /tmp/apinger.loss.flag -o -e /tmp/apinger.delay.flag ]; then
  ${RAINBOW} wan ${WARNING_COLOR}
  exit
fi
# no active alarms found
${RAINBOW} wan ${DEF_COLOR}

This works pretty good – if line is down – WAN color is red, if it is unstable or congested – yellow. We can also monitor link status manually:

root@turris:~# cat /tmp/apinger.status
Fri Apr 10 12:39:24 2015

Target: 1.2.3.4
Description: ISP Gateway
Last reply received: #2876 Fri Apr 10 12:39:23 2015
Average delay: 3.247ms
Average packet loss: 0.0%
Active alarms: None
Received packets buffer: ################################################## ###################.

Todo

I am planning to extend functionality of the script with some cool features:

  • Integrate with Luci to show status in the web interface.
  • Add support for the failover switch to the LTE channel if link is down (and LTE dongle connected).
  • Enable rrdtools support provided by apinger.
Tagged , , , ,

Using g.729 codec with Asterisk on Raspberry Pi (or other ARM device)

I decided to build home PBX based on Asterisk VoIP server running on my Raspberry Pi device. One of the reasons for this was ability to build cheap GSM gate for home use using chan_dongle. But one of the problems i have found was lack of g.729 codec for the Asterisk on ARM.

On Intel platform it is possible to use codecs from asterisk.hosting.lv or to buy commercial codec from Digium. Unfortunately its not the case for the ARM. So i decided to see if it possible to port some existing g.729 codec.

Project asterisk-g72x is only Open Source g.729 for Asterisk implementation i am aware of. Internally it is using 2 libraries:

  • IPP from Intel
  • Or ITU based reference code.

It is not possible to compile recent IPP versions on ARM and a lot of ASM code making porting of it problematic. ITU g.729 code is on a plain C, but is painfully slow. It compiles on ARM, but performance is terrible. Asterisk eating 100% of CPU on recoding and drops frames. So it was not an option. So i decided to find alternative codec. In the net i found 2 Open Source projects with g.729 implementation suitable for ARM

  1. Experimental version of G.729 codec for ARM device. It seems to be ITU source code with ARM assembler code for some operations.This code also using many global variables so is not ready for multi-thread software like asterisk without additional changes. Also there are issues open from 2011 without any reaction from developers.
  2. Bcg729. It is a software G729A encoder and decoder library written in C, developed by Belledonne Communications, the company supporting the Linphone project. It was written from scratch and is NOT a derivative work of ITU reference source code in any kind.

BCG 729 also supports concurrent channel encoding/decoding for multi-call applications such as Asterisk. I was able to adopt bcg729 for use with asterisk-g72x project, instead of slow ITU code. If you want to test it – use my BitBucker fork. I will post benchmarks later, but now i see about 15-20% CPU load on g.729 encoding with Asterisk, so it should be able to support 4-5 concurrent channels in time.

Tagged , , , , , ,

JVPN 0.7.0 released

JVPN is a Perl script to connect to the Juniper VPN with Host Checker enabled.
New version (0.7.0) adds host checker support, as well as some other features and bug fixes:

+ Added experimental code to run host checker from jvpn
+ Include LWP traffic dump in the debug output
+ Added sample script to restore original DNS settings after connect
+ Added experimental grid cards support
# fixed processing of last line w/o \n character in configuration file

Download location could be found in JVPN post.

Tagged , , ,

JVPN 0.6.1 released

JVPN is a Perl script to connect to the Juniper VPN with Host Checker enabled.
New version (0.6.1) adds ability to store password/token in configuration or to use external scripts to provide it. Also it adds ability to define custom URL and addressing issues with scripting support added in 0.6.0.
You can download it from JVPN post.

Tagged , , , , ,

Using RSA soft token from Linux

I already discussed RSA hard tokens and ability to OCR them. This time i would like to discuss RSA Soft Tokens support in Linux. Despite the fact that there is no official Linux support i found that it works perfectly. There are at least 3 ways to use them:

I am currently using options 3 and 2 and keeping one as backup.

  1. Using official Windows software in Wine. It installs and run just fine
  2. Using Android/Iphone application on mobile phone, you will need to covert the key using RSA token converter.
  3. Using non-official open-source Linux client – stoken. It has CLI and GUI (GTK2) versions, supports batch integration and works perfectly for me.

All tokens should show equal numbers. Below is a screenshot of native Windows client and stoken/stoken-gui.
tokens
I found that all options works perfectly for me, currently i am mostly using stoken CLI because it was easy to integrate it with JVPN tool. I will update jvpn nearest days to support external authenticators.

Tagged , ,