Using Debian Linux on IOMEGA StorCenter Pro IX-12

Monitoring and fine tuning

In the previous post I covered process of the replacing buggy closed-source firmware with the Debian Linux on the  IOMEGA StorCenter Pro IX-12 NAS device. But for effective and reliable usage installation process is not enough – you also need to re-implement hardware and software monitoring. StorCenter Pro IX-12 is pretty standard intel-based box, but it has some vendor-specific customization which we will discuss now. Below is a [possibly incomplete] list:

  • Temperature sensors. In the BIOS i found that devices shows temperature for the:
    • CPU temperature
    • Ambient Temperature
    • NIC Temperature
    • Midplane 1 and Midplane 2 temperature
    • PSU1 and PSU2 temperature
  • FAN rotation speed:
    • FAN1 – FAN3 sensors (no idea where they are)
    • PSU1  fan and PSU2 fan sensors
  • LEDs and buttons:
    • Power LED (seems to support 2 colours + blinking)
    • HDD LEDs (every caddy has Status and Activity LEDs, Status could be green, red or blinking)
    • Power Button (controlled by ACPI, works without modifications)
    • Reset (or “Factory Defaults”?) button near network cable socket.

Initially I expected that this functionality is managed using IPMI interface, like in most modern server boards. But I found no IPMI management, so  I decided to look on the original firmware to find how to manage this devices. Old firmware is still existing on the 1GB Flash drive connected to the motherboard, so I mounted it and found that software is living in squashfs file-system in a file (probably to save disk space) and found that it using strange “/proc/wix/*” interface to access hardware-specific features. Also I found that module which providing this interface is compiled into the kernel, so according to GNU GPL sources must be provided.

Reimplementing vendor-specific functionality

Then I found that there is archive with GPL sources on the IOMEGA site, so I downloaded and extracted it. It was a nice finding – archive contains not only source of user-land, but also patches to the kernel. I found that all hardware-specific kernel code is open source (hooray!) and is available in the ix12wix_100305v2.patch file. This patch contains kernel configuration for the 2.6.32 kernel, as well as many self-written kernel modules with hardware-specific code. Initially I was thinking to port them to 3.2 kernel, but after looking on the code I decided not to do this. From my personal opinion code is quick & dirty, without attempts to use standard Linux interfaces and subsystems. Also I don`t think that we really need this /proc/wix/* interface (why “/proc”, it should be “/sys”) .  But sources itself are very important to understand how to deal with this non-standard devices. Some findings:

  • Most of devices are on the I2C bus and kernel modules are working with them directly
  • PSU units are also connected via I2C
  • LEDs functionality is implemented in a very tricky way . HDD “Status” and “Fault” LEDs (1-12) are connected to the 3 different GPIO modules: PCA9552, PCA9539 and PCA9557 (on I2C bus). “Power” and “Sys ID” LEDs are connected to the Intel ICH9 GPIO (on PCI bus).
  • There is ICH9 watchdog device, utilized by standard iTCO_wdt driver. Good finding, we can use it in Linux.
  • Temperature sensors are also on different devices: on every PSU, 2 TMP175 sensors on midplane, and 3 (CPU, NIC and Ambient) on sch5027e controller. All of them are connected via I2C bus.
  • There is also another watchdog on baseaddr = 0x0500 (referenced as SIOWDTimer) which is used to poweroff the server in case of temperature events.

I2C bus is detected and working with standard i2c_i801 module. I am also installed loaded i2c_dev kernel module and i2c-tools package to play with I2C bus.

This is I2C bus detection results and some my comments

0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- 08 -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- 18 -- -- -- -- -- -- --
20: -- -- -- -- -- -- 26 -- -- -- -- -- -- -- 2e --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- 44 -- -- -- -- -- 4a 4b -- -- -- --
50: -- 51 -- 53 54 55 56 57 58 59 -- -- -- -- -- --
60: 60 -- -- -- -- -- -- -- -- 69 -- -- -- -- 6e --
70: -- -- -- -- 74 -- -- --

Comments:
0x08 – Unknown
0x18 – pca9557 chip, Fault LEDs, HDD 5-12
0x26 – pca9535 chip, Hardware alarms (?) for different events
0x2e – M/B FAN and Temperature monitoring chip
0x44 – Unknown
0x4a, 0x4b – tmp75 temperature sensors on midplane
0x51, 0x53 – DIMM DDR2 SDRAM 1024MB eeprom
0x54, 0x55 – DELTA DPS-700LB Power Unit eeprom (PSU1, PSU2)
0x56 – M24C16/08/04/02/01 I2C EEPROM (???)
0x57 – AT24C02 I2C EEPROM
0x58, 0x59 – Sensors for the PSU1 and PSU2
0x60 – pca9552 chip, HDD Status LEDs, HDD1-HDD12
0x69, 0x6e – Unknown
0x74 – pca9539 chip, HDD fault leds, HDD1-HDD4

I used i2cdetetect tool to scan I2C bus and wix kernel module sources to find corresponding devices.

System Temperature and FAN sensors

The most important is to monitor temperature and FAN status. Temperature may go high because of fan problems or because of external conditions. We have a lot of information sources for the temperature monitoring:

  • 3FAN and 3 temperature sensors on sch5027e
  • 2 LM175 temperature monitoring chips installed on midplane.
  • CPU internal temperature sensors
  • Hard disk internal temperature sensors
  • Power Supply Unit fan RPM and temperature sensors

Lets see how we can access this data.

I decided not to do one more non-standard interface, but to use lm_sensors software to get the information. sensors-detect tool was able to detect LM85 monitoring chip and suggested to load lm85 module. Also CPU sensors were detected and “coretemp” kernel module was recommended to load. I found that LM85 module is, in fact, working with sch5027e chip, so probably they have compatible interfaces. On this stage I was able to get information from CPU and motherboard sensors.

Then I found that LM175 should be supported by “lm75” kernel module, but nothing was detected when I loaded it with modprobe. After looking into module source code I found that there is no detection procedure for the LM175, so we need to manually specify sensors addresses. I tried to do this using this 2 commands:


echo tmp75 0x4b > /sys/bus/i2c/devices/i2c-0/new_device
echo tmp75 0x4a > /sys/bus/i2c/devices/i2c-0/new_device

And right after this I got 2 new sensors, available with “sensors” command or via sysfs interface. I added this commands to /etc/rc.local to detect them on boot. It is possible to view sensors directly using sysfs or (more convenient) with “sensors” tool from lm_sensors. I modified default configuration (/etc/sensors.conf) to better reflect ix12 hardware:


chip "lm85-*" "lm85b-*" "lm85c-*" "adm1027-*" "adt7463-*" "adt7468-*" "emc6d100-*" "emc6d102-*"
label in1 "Vcore"
label in2 "+3.3V"
label in3 "+5V"
label in4 "+12V"
set in2_min 3.3 * 0.90
set in2_max 3.3 * 1.10
set in3_min 5.0 * 0.90
set in3_max 5.0 * 1.10
# Depending on how your chip is hardwired, you may or may not have
# +12V readings.
set in4_min 12.0 * 0.90
set in4_max 12.0 * 1.10
label temp1 "CPU Temp"
label temp2 "Ambient Temp"
label temp3 "NIC Temp"
ignore fan4


chip "tmp75-i2c-*-4a"
label temp1 "Midplane Temp 1"

chip "tmp75-i2c-*-4b"
label temp1 "Midplane Temp 2"

With this settings after running “sensors” I have:

Adapter: ISA adapter
Core 0:      +45.0°C  (high = +78.0°C, crit = +100.0°C)
Core 1:      +45.0°C  (high = +78.0°C, crit = +100.0°C)

lm85-i2c-0-2e
Adapter: SMBus I801 adapter at 1100
in0:          +2.51 V  (min =  +0.00 V, max =  +3.32 V)
Vcore:        +1.08 V  (min =  +0.00 V, max =  +2.99 V)
+3.3V:        +3.27 V  (min =  +2.97 V, max =  +3.63 V)
+5V:          +5.16 V  (min =  +4.50 V, max =  +5.50 V)
+12V:        +12.12 V  (min = +10.81 V, max = +13.19 V)
fan1:        3985 RPM  (min =  100 RPM)
fan2:        3913 RPM  (min =  100 RPM)
fan3:        4038 RPM  (min =  100 RPM)
CPU Temp:     +35.0°C  (low  = -127.0°C, high = +127.0°C)
Ambient Temp: +46.0°C  (low  = -127.0°C, high = +127.0°C)
NIC Temp:     +45.0°C  (low  = -127.0°C, high = +127.0°C)
cpu0_vid:    +2.050 V

tmp75-i2c-0-4b
Adapter: SMBus I801 adapter at 1100
Midplane Temp 2: +37.0°C  (high = +66.0°C, hyst = +65.0°C)

tmp75-i2c-0-4a
Adapter: SMBus I801 adapter at 1100
Midplane Temp 1: +35.0°C  (high = +66.0°C, hyst = +65.0°C)

Power Supply Unit sensors

As I already notices this device has 2 power supply units (PSU) to provide redundancy. To make it work properly we need to monitor both PSU. Devices itself are DELTA DPS-700LB (you can read name and serial number using I2C). I found no Linux driver for this device in the kernel, but protocol itself is very simple and available in the wix iomega sources, so I decided to use this device from userland.

To query PSU device from Linux i used i2cget command fro the i2c-tools package. Also “i2c-dev” kernel module needs to be loaded. Using kernel module functionality i re-implemented PSU monitoring in Perl. Now it is possible to read voltage, fan speed and PSU temperature using psu_read.pl. If running without arguments it will display PSU sensors information in “sensors” like mode. Running with “-n” will make psu_sensors.pl act as Nagios plugin.  Below is an example of the script output

psu700-i2c-0-59
PSU Temp: 47°C
+12V: +12.0 V
fan1: 6960 RPM

psu700-i2c-0-58
PSU Temp: 47°C
+12V: +12.0 V
fan1: 7680 RPM

Smartmontools

With smartmontools it is possible to monitor drives physical state and temperature. Smartd also provides monitoring and reporting of the SMART attribute changes. Also i am using smartmontols to setup ERC value for the all drives installed in the NAS with a command:
lsscsi |grep ATA|awk '{print $6}'|xargs -n 1 smartctl -l scterc,100,100

I added  this command to the /etc/rc.local file.

Other server-specific hardware

It should be possible to use ix12 hdd LEDs from Linux, but this NAS is installed in the remote data center, so i had no chance to play with it. It should be possible to use it with kernel led and gpio subsystems.

Conclusion

It was possible to get all information from the non-standard hardware to monitor it state. This is a good example why it is so important to have an access to the source code. Feel free to comment this article.

Advertisements
Tagged , , , , , ,

3 thoughts on “Using Debian Linux on IOMEGA StorCenter Pro IX-12

  1. […] Using Debian Linux on IOMEGA StorCenter Pro IX-12 Share this:TwitterFacebookLike this:Like2 bloggers like this post. Tagged debian, iomega, ipv6, […]

  2. Yves says:

    Hi. Can you public “psu_read.pl”? I have dps-600mb, connect them to dvi i2c pinouts and can read model and SN through i2cdump.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: