Building OpenSCADA and firmware for ARM-controllers of ICP DAS company (LP-5141)

23.09.2011 09:26 Age: 13 yrs

By: Roman Savochenko

Name: LP-5xxx

Start: august 2011

Finish: september 2011

Version: 1.0.0

Performers: Roman Savochenko

Description: The project is dedicated to building the OpenSCADA and firmware for ARM controllers of the LP-5xxx and LP-8x{3|4}x series of the ICP DAS company.

Materials: ftp.oscada.org/OpenSCADA/PLC/LP5xx1

The ICP_DAS company produces the controllers on ARM architecture processors for quite a long time with OS Linux as preinstalled environment, which makes interesting the creation or OpenSCADA build or the new firmware with OpenSCADA.

Until the last moment there were a number of reasons that prevent the creation of an OpenSCADA build for ARM-controllers of the ICP DAS:

absence of the ARM-controller;
unavailability of OpenSCADA to work on ARM-architecture;
extremely outdated Linux environment on the ARM-controllers of "ICP DAS".

At the moment, all of these obstacles have been eliminated. So, thanks to Peter Litkovets we got the LP-5141 controller, and by this time OpenSCADA was largely already adapted to work on ARM-architecture, through processing and stabilization on the handheld computer of Nokia company - N800. Later during the same work with the LP-5141 controller it has been eliminated the last obstacle by updating the original software environment while maintaining the original functionality.

In general, the goal of this project is to develop the mechanisms and the creation of OpenSCADA building for the ARM family of the controllers of "ICP DAS" company, as well as the creation of a firmware for the "ICP DAS" controllers, which will fall into the hands of OpenSCADA developers.

1. PLC LP-5141

PLC (Fig. 1) is designed in the monoblock form-factor that does not include built-in computer-process interface, except for the possibility of installing a special expansion board IO, and the connection of the external one is made through the built in RS-485, for example, in the form of modules I-7000 of the the "ICP DAS" company.

Fig. PLC of the LP-5xxx series.

CPU of the controller has the following specifications:

CPU	PXA270 or compatible (32-bit and 520MHz)
SDRAM	128 MB
Flash	64 MB
EEPROM	16 KB Data storage: 40 years; 1,000,000 delete/write cycles.
Extended Flash memory	microSD socket with a 2GB microSD card (can support 16 GB microSDHC card)
64-bit serial number of the equipment	Present
Dual Watchdog Timer	Present
VGA	800 × 600
Ethernet ports	RJ-45 x 2, 10/100 Base-TX Ethernet controller (Auto-negotiation, auto MDI/MDI-X, LED indicator)
USB 1.1 (хост)	1
COM1 (/dev/ttySA0)	RS-232 (RxD, TxD and GND); not isolated
COM2 (/dev/ttySA1)	RS-485 (D2+,D2-); 2500 VDC; isolated
COM3 (/dev/ttySA3)	RS-232 (RxD, TxD and GND); not isolated
Operating temperature	-25 ~ +75 °C

2. Software environment based on OpenSCADA.

Software environment, based on the Linux operating system for ARM controllers of ICP DAS usually includes: the Linux kernel 2.6.19, GLibC 2.2.5 and GCC 2.95. To build a third-party software the ICP DAS provides an SDK with the cross-compiler, set of libraries and headers for them (Linux toolchain). As you can see, the version of GCC and are very old, namely it is the 2001 year. OpenSCADA building in this environment is actually impossible. If the using of the GLibC of 2.2.5 version is still possible, the GCC of version 2.95 has the C++ compiler, which crashes during the building of the OpenSCADA code, and the standard C++ library or STL is extremely limited and it requires the significant and senseless adaptation. For this reason, it is necessary to update the original software environment to version for at least 3 of GCC compiler and C++ library.

Because internal flash memory has a relatively small capacity (64 MB), and a full rebuild of the original software environment of the controller - a very laborious process, it was decided to ensure the compatibility of the basic libraries of old and new environments. Namely, the ability to use libraries like fontconfig, i8k from the source environment and the work of the source environment programs with the new libraries.

In the process of selecting a new software environment for the PXA-270 processor of the "ICP DAS" controller several options have been tried because some of them had one or other disadvantages:

crosstool-0.43 — interesting but outdated project (latest version is 2006 year) to build a Linux cross environment for the necessary equipment. It was postponed after several unsuccessful attempts to build a new environment to work with "Software FP" and "VFP". As it turned out subsequently for the original environment object files the information about the presence there "Software FP" and "VFP" is incorrectly displayed, when in fact the operations with real numbers there are made through "FPA". Subsequently the problem solve, described in section "Remarks", by aid the tool have been built ToolChain grounded on profile "arm-xscale.dat" and "gcc-4.0.2-glibc-2.3.6-tls.dat", which at last have been used on base of new firmware.
PXA-Linux Project — a project to create an independent set of Linux tools for building for the PXA processor. It has the binary building 02-25-2005/bin/arm-linux-toolchain-bin-12-15-04-driscoll.tar.gz. It turned out that this building was built with the "Software FP" and "VFP", and also has a problem in the form of disabling of set of expansions for the gcc compiler, for example: __attribute__((packed)).
Voipac — binary build of the cross compilation tools (arm-linux-gcc-3.4.1.tar.gz) from the firm "Voipac" for their own equipment on the PXA-270 processor. This ToolChain built by help early seen crosstool.

The resulting environment of the controller with OpenSCADA was formed by replacing the original environment core libraries on the libraries from new tools, building of the OpenSCADA and placing the OpenSCADA files to source environment tree.

OpenSCADA was built with disabling of the number of libraries and modules. So, it were build and tested the following modules and functions of OpenSCADA:

OpenSCADA core — built with the LibGD2 library (the work is checked). In the process of checking and adaptation the functions of storage the OpenSCADA configuration in the configuration file are expanded.
DB.SQLite — SQLite DB (the work is checked). The library and module of OpenSCADA to work with SQLite database are built. During the check process the error of GLibC 2.3.2, related to the lack of check on inter-locking of "rw"-resources is found and eliminated.
DB.DBF — DBF DB (the work is checked). Module to work with DBF-files of version 3.
Transport.Sockets — The module the the TCP, UDP and Unix sockets' transport (the work is checked). In the process of TCP-sockets using it was found and corrected the structure's of check the socket's status not initialization.
Transport.SSL — Transport module of the secure socket layer: SSL, TLS (the work is checked).
Transport.Serial — Transport module of serial interfaces (the work is checked). It was found that for some reason COM3 is located on the device /dev/ttySA3.
Protocol.SelfSystem — The module of the own OpenSCADA protocol (the work is checked).
Protocol.HTTP — The module of the HTTP protocol implementation (the work is checked).
Protocol.ModBus — The module of the ModBus protocol implementation (the work is checked). The configuration of the station as a data server through the ModBus protocol.
Protocol.OPC_UA — The module of the OPC UA protocol implementation (the work is checked). The configuration of the station as a data server through the OPC UA protocol . There was a problem of difference of real number representation on x86 and ARM FPA!
Protocol.UserProtocol — The module of free implementation by the user of simple protocols using the programming language of OpenSCADA (the work is checked).
DAQ.System — The module of the operating system data (the work is checked). It is built without the LibSensors library using.
DAQ.JavaLikeCalc — The module of implementation of user Java-like language and calculations based on it (the work is checked).
DAQ.BlockCalc — The module of implementation of block computing (the work is checked).
DAQ.LogicLev — The module of implementation of the logic level data sources (the work is checked).
DAQ.ModBus — The module of working with data sources via the ModBus protocol (the work is checked).
DAQ.DCON — The module of working with data sources via the DCON protocol (the work is checked). For example, with the devices ICP_DAS I-7000. It was found and fixed the crash when receiving a negative response size. The result of the answer is now always returned >=0.
DAQ.ICP_DAS — The module of the ICP_DAS data sources (there is no equipment to check). Serial series: I-87000 and I-7000; Parallel (quick): series I-8000. The module is built with a library libi8k.a for ARM-architecture.
DAQ.DAQGate — The module of implementation of the locking of data sources from other OpenSCADA stations (the work is checked).
DAQ.OPC_UA — The module of working with data sources via the OPC_UA protocol (the work is checked). There was a problem of difference of real number representation on x86 and ARM FPA - the conversion when transmitting is added!
DAQ.Siemens — The module of the working with the Siemens controllers via Industrial Ethernet (ISO_TCP) (the work is checked).
DAQ.SNMP — The data acquisition module of the network equipment via SNMP (the work is checked).
Archive.DBArch — The module of the database archiving (the work is checked).
Archive.FSArch — The module of the file system archiving (the work is checked). There was a problem during the verifying of the archive, giving the following message at startup: "Error archive file structure: <ARCHIVES/VAL/1s/CPULoad_load 2011-09-12 17:47:17.val>. Margin = -8 byte. Will try fix it!". As a result, it was found that the algorithm of fast bits' counting is not aligned to 4 bytes when reading the memory buffer - it is fixed.
UI.WebCfg — The Web configurator module of OpenSCADA (the work is checked).
UI.WebCfgD — The dynamic Web configurator module of OpenSCADA (the work is checked).
UI.VCAEngine — The visualization engine module (the work is checked).
UI.WebVision — The module of the user interfaces visualizer in the UI.VCAEngine by means of the Web-interface (the work is checked).
UI.WebUser — The module of free user's visualization by means of the Web-interface (the work is checked).
Special.FLibComplex1 — The library of compatibility functions with Complex1 ("DIYA" LTD) (the work is checked). It is used for the block programming.
Special.FLibMath —The library of standard mathematical functions (the work is checked).
Special.FLibSYS — The library of system expansion functions of OpenSCADA (the work is checked).

3. Remarks

As it turned out that the operations with real numbers are performed by the FPA co-processor instructions (Float Point Acceleration), which is not present in this processor, and calls of the commands themselves are made through an imitation of an exception in the Linux kernel, then the performance of mathematical calculations is extremely low, even compared to the direct "Software FP", "VFP". For example, the calculation of a single operation sin(pi) is about ~200 ms, compared to ~20 ms for N800 with VFP and ~2 ms at x86, detailed in table by the link. So, this controller shouldn't be considered as the platform for at least any serious calculations! Moreover, the real numbers when working with FPA are stored in a different way - namely, the special big-endian, which requires a transformation in the case of the binary external exchange to the typical little-endianrepresentation, for example for DAQ.OPC_UA.

In the initial configuration the COM1 port (/dev/ttySA0) is used as a console, which is initialized by the kernel command "console=/dev/ttySA0". However, after adjustment this is not necessary and is often desirable to release another one COM-port for other purposes. It is possible to release COM1 from work for a console by reassigning the other device for this role, for example by: $busybox setconsole /dev/tty1. However, this method does not fully release the COM-port and requests are lost, reading by the parallel connection. Probably need to edit the kernel boot options line in the bootloader U-Boot, even enter into it possible only in the position of RS = 2, however for which the initialization parameters are separated from the main mode of RS = 0. It is necessary to achieve the entrance to the U-Boot from the primary mode of RS = 0!

In the process of using the created software environment with OpenSCADA it was found a strange problem: crashing the OpenSCADA during access to an instance of the resource string from different threads. The problem is reproduced when polling the controller via ModBus/RTU with 100 ms frequency, the polling an absent module by DCON on the same RS485 interface as the ModBus/RTU. At the same time the periodic (1 sec) record of two registers to the polled via ModBus/RTU controller is made. The crash does not occur immediately, but within 10 minutes - 5 hours. Analysis of the problem:

(*) In the case of disabling the record to the controller via ModBus/RTU crashes don't occur.
(*) It was found that the crashes begin in case of a achievement by TMdContr::modBusReq() function of DAQ.ModBus module, namely, concurrent access to controller's object property "mPrt" in the context of the "XMLNode req (mPrt);" constructor from different threads of reading and writing.
(*) To clarify the nature of the problem it has been configured the generation of death memory dump, as well as assembled the gdb cross-debugger. During the study of death dump it was found that there is a destruction of the stack, and there is no rational reasons for that in OpenSCADA, so this is a problem of the system environment for specific conditions.
(*) When you access the "mAddr" property, in the first line of the TMdContr::modBusReq() function and in the problem context of "XMLNode req (mPrt);" (with an incomplete record), the crash is not seen.
(*) Checking the call of the second line "XMLNode req (mPrt);" when writing and in general with the "RTU" constant. Replacement with "RTU" in general make the system crash in another place. When you replace only the incomplete record the system doesn't crash. The second and last crashing place was the next line req.setAttr ("id",id()). There is something common among them - the access to the configuration element of string type, using the object of the resource type "ResString". Conclusion: the current software environment does not work correctly with the resource type object "ResString" for read access from two different threads.
(*) The way of storing an "ResString" object in a "TCfg" configuration field element does not matter - namely, the crash is observed when "ResString" is directly stored in the "TCfg", as well as during the direct access to the "string" object inside "ResString" in the case of the "id()".
(*) Check when you disable the resource coverage does not solve the problem.
(*) Building a new environment with the help of "PTXDist" and running it through a "chroot" when you run EABI environment for TionPro270 out of chroot - the "Illegal instruction!" error occurs. The EABI kernel from TionPro270 with similar mistake does not run. The built ToolChain from the profile "arm-xscale_hardfloat-linux-gnu_gcc-4.0.4_glibc-2.3.6_binutils-2.17_kernel-2.6.18.ptxconfig" in the "OSELAS.Toolchain-2011.11.0" crashes with a segmentation fault at the startup of OpenSCADA built under the the original environment.
(*) Building the OpenSCADA including all the necessary modules in the OpenSCADA kernel library does not solve the problem.
(*) Check of the same configuration on the another PXA270 device-(TionPro270(RUS)) for the final clarification of the source of the problem (hardware or software environment) - configuration worked correctly for two days.
(*) To replace in the TCfg object "ResString" to the usual "string" with the removal of access resource to the TConfig object container, reducing the memory consumption by a generalization of the resource for access to the string object - replacement is done, the problem is still not solved.
(*) To build the system with the replacement of "ResString" by the usual "string" in TCfg and to check the current crashing location - the place of the crash is the same.
(*) To get "crosstool" and build more recent toolchain - the toolchain "gcc-4.0.2-glibc-2.3.6-arm-xscale-linux-gnu" is built, the problem is still present, most likely a problem is in working with memory in the current kernel. In the process of studying the precedents with similar problems in the glibc-2.2 - 2.7 which contains the implementation of malloc unsafe for threads (nothread-safe) were found.
(*) Replacing the memory allocator - unable to find the working and thus a transparent implementation of third-party allocator, ptmalloc3 just crashes.
(*) Building of the ToolChain and OpenSCADA with all possible options include thread-safe - in all variants the problem is present, namely the GLibC parameters of the ToolChain: "~ - with-tls - with-__thread" and OpenSCADA is builtwith the following parameters: "-pthread-D_REENTERANT".
(*) Interception of calls for memory allocation and wrapping with a global resource of the "C" functions malloc, free and "C++" functions new, delete - the crash is still present.
(+) Testing the hypothesis of the problematic implementation (atomicity) of the COW (Copy on Write) algorithm in the "string" object - copying from the "string" object store is made by the val.c_str(), which excludes the implementation of COW: confirmed that this problem is connected with the COW algorithm, namely with the atomic access to the strings counter "_Atomic_word _M_refcount;" probably because of an outdated threads system "linuxthread".

4. Conclusion

The result was the building of OpenSCADA for Linux controllers of ARM-architecture of "ICP DAS" company. The building can be loaded directly on any Linux-ARM controller, unpacking it at the root of the original operating system. In this case, however, the duplicates of the old core libraries (/lib/*) will remain, which can be removed after a successful restart.

For family of controllers LP-5x4x the firmware with the building OpenSCADA was created, which can be downloaded in the standard for these controllers way, which is described in the original "ICP DAS" documentation. Firmware for LP-5x4x can possible fit for the LP-5x3x, but it is not checked!

Due to outdated and non-optimal initial building of the Linux environment from "ICP_DAS", which imposes the restriction on freedom of building the optimal environment, after the completion of the project it is planned to apply for support to "ICP_DAS" with the request to correct this situation.

(*) Treatment carried out however no response until the response to the request was not followed!

Links:: Project; Archive of the OpenSCADA building; Firmware with OpenSCADA for LP5x41

<- Back to: Solutions