Decentralized Control of a Material Flow System Enabled by an Embedded Computer Vision System

Decentralized Control of a Material Flow System enabled by an Embedded Computer Vision System Constantin Timm∗ , Frank Weichert† , David Fiedler† , Christian Prasse‡ , Heinrich M¨uller† , Michael ten Hompel§ and Peter Marwedel∗ , ∗ Department

of Computer Science XII, TU Dortmund, University of Technology, Germany Email: [email protected] † Department of Computer Science VII, TU Dortmund, University of Technology ,Germany Email: [email protected] ‡ Fraunhofer Institute for Material Flow and Logistics, Dortmund, Germany Email: [email protected] § Chair for Materials Handling and Warehousing, TU Dortmund, University of Technology, Germany Abstract—In this study, a novel sensor/actuator network approach for scalable automated facility logistics systems is presented. The approach comprises (1) a new sensor combination (cameras and few RFID scanners) for distributed detection, localization and identification of parcels and bins and (2) a novel middleware approach based on a service oriented architecture tailored towards the utilization in sensor/actuator networks. The latter enables a more flexible deploying of automated facility logistics system, while the former presents a novel departure for the detection and tracking of bins and parcels in automated facility logistics systems: light barriers and bar code readers are substituted by low-cost cameras, local conveyor mounted embedded evaluation units and few RFID readers. By combining visionbased systems and RFID systems, this approach can compensate for the drawbacks of each respective system. By utilizing a stateof-the-art middleware for connecting all computer systems of an automated facility logistics system the costs for deployment and reconfiguring the system can be decreased. The paper describes image processing methods specific to the given problem to both track and read visual markers attached to parcels or bins, processing the data on an embedded system and communication/middleware aspects between different computer systems of an automated facility logistics system such as a database holding the loading and routing information of the conveyed objects as a service for the different visual sensor units. In addition, information from the RFID system is used to narrow the decision space for detection and identification. From an economic point of view this approach enables high density of identification while lowering hardware costs compared to state of the art applications and, due to decentralized control, minimizing the effort for (re-)configuration. These innovations will make automated material flow systems more cost-efficient.

I. I NTRODUCTION The increased complexity of facility logistics systems forces the utilization of automated systems. When properly dimensioned and highly utilized, such systems operate very efficiently and effectively, especially when new promising approaches for information and communication technologies are employed such as decentralized systems based on software agents (often associated with the broader vision of the “Internet of Things” [1], [2], [3] and also as a specific vision for logistic networks [4], [5]). This incorporates the development

ID

loc. ID + loc.

loc.

loc.

(a) Central identification

(b) Global identification

Fig. 1. Identification and localization at a (a) central identification point, (b) global identification via RFID for the entire system, respectively for a bigger section and localization by vision at local intersection points.

of the “Radio Frequency Identification” (RFID) technology as a major shift in the way data about logistical objects is managed. All these developments have one common goal: to make automated facility logistics systems easier to change and cost-effective in the long term. In this paper, roller and belt conveyor systems are addressed as typical form of automated transport and distribution of light weight piece goods (parcel, bins, etc.) within a distribution center. Several intersection points redirect the piece goods to the desired destination. To perform this task, the exact position and sequence of all objects must be known. Today the identification of the conveyed object is detected at a so-called Identification Point (I-Point) at the interface between manual and automatic systems (shown in Figure 1(a)). The relatively high cost causes limited installation of further identification equipment beyond this point. Within the automated system, this has the consequence that the sequence of objects must be controlled very carefully [6]. In this paper a sensor/actuator network observing and controlling a conveyor system is presented. This network is based on a web service based middleware which follows the publish/subscribe paradigm, allows the management of

Fig. 2.

Network Structure within the Conveyor System

resources at each sensor node and the communication with(in) different groups. The main sensors in this system are the so-called “Vision System Units” (VSU) for tracking piece goods and controlling intersection points and only one or few RFID scanners monitoring the conveyor system. The main actuators in this network are the switches at the different intersection points which are directly controlled by these VSUs. A VSU realizes the localization and tracking of the piece goods locally at intersection points (cf. Figure 1(b)) by applying a single low-cost camera. This paper will provide the following novel concepts for the considered conveyor system: • Decentralized control of switches at intersection points directly by the VSU • Web service based publish/subscribe middleware for easy (re-)configuration and resource management • Flexible and scalable conveyor system design The paper is structured as follows: After this introduction, related work is presented. The principles of the middleware design for the considered conveyor system are introduced in section III, followed by the presentation of the “Vision System Units” in section IV. Before the conclusion of the paper in section VI, a summary over the scalability of the system and exemplary results are given. II. R ELATED W ORK The localization of active and passive RFID tags in automated facility logistics systems was objective of several Real Time Localization Systems (RTLS), such as LANDMARC [7]. Beside RTLS, the tracking of barcode or ORC for address arrays was subject of systems such as Visicon Singulator [8]. Computer vision systems based on single or multiple standard cameras offer an alternative way for accurate object positioning and tracking at low investment cost. Several standard tracking methods are surveyed in [9]. In contrast to this a combined approach which uses RFID and camera-based location was firstly described in [10]. The use of standardized web services was proposed in many application fields such as wireless sensor networks [11], but to our knowledge this paper is the first one which describes the combination of utilizing

a web service based middleware in a material flow system supported by a low-cost vision-based identification system. III. C OMMUNICATION A SPECTS AND D EPLOYMENT S TRATEGIES The considered conveyor system comprises a large variety of different computer systems. Communication between such systems is typically complex and costly especially when using different protocols and network devices. New and powerful approaches for middleware support in fabrication environments comprise the utilization of web services [12], [13]. A middleware that use these web services and provides additional embedded system related services is called MORE Middleware [14], [15]. This middleware focuses especially on hierarchical network setups [16] and enables the use of web services as a standardized interface for all systems in the network and even for each software component. Furthermore it provides services such as routing of packets in an overlay network and ad-hoc group communication. In the mentioned conveyor system the middleware is used to connect the different VSUs with the RFID database, the in succession located VSUs and the switches with the conveyor system. Following the MORE methodology also the connection of the components inside the VSU is done by the middleware. A. Web service-based Communication in the Conveyor System The considered conveyor system as depicted in Figure 2 is a typical hierarchically structured sensor/actuator network. At the topmost level, large-sized computer systems are used which run e.g. the RFID central database. They are equipped with full featured web services and web service orchestration engines. At this level no resource constraints aside from economical or environment protection reasons exist. At a lower level (Figure 2: white) components of the sensor network such as VSUs are used to aggregate and forward information. They are connected among each other, to the controlled switch and to the central RFID database. These systems are equipped with standardized local-area communication interfaces to allow a communication with the sensor/actuator from outside. The resources of these systems are restricted meaning they are constrained in processing and memory capabilities and they have often energy constraints. In order to cope with these

constraints software running on these systems have to be developed with them in mind, i.e. not all features of topmost level systems such as dynamic service orchestration can be deployed. Systems of the lowest level are highly resource constrained. These systems are the sensor/actuator nodes like the switches and cameras with minimal processing power and only limited communication capabilities.

RFID DB RFID unit

camera embedded evaluation unit

B. MORE Middleware Concepts The design of the MORE Middleware [14], [15] conforms to the paradigm of Service Oriented Architectures by adopting it for external and internal communication to provide a high degree of flexibility. The use of the publish/subscribe paradigm within the MORE Middleware is especially important for sensor/actuators networks and makes the design and deployment of such networks easier. In terms of communication protocols the MORE Middleware satisfies a subset of the DPWS specification [12]. DPWS identifies a minimal set of Web Service specifications tailored towards the needs and capabilities of embedded devices in order to allow for a base level of interoperability between devices and standard Web Services. The major contribution of the MORE middleware are the “Added Value Services” which offer common functionality like service orchestration, group and resource management services (see more details in [15]). These features are motivated not by the need of standardized interfaces but rather by the requirements of using a web service architecture in sensor/actuator networks. One requirement is that the resources of a single sensor node must be organized in a way that their use is optimized, e.g. the sensor node should save as much energy as possible. Another requirement is that services should be re-used and connected efficiently. In a standard web service environment this is done by WS-BPEL engines – allowing service orchestration – which mostly harvest too much resources and are not suited towards the use in embedded systems. A lightweight engine which follows WS-BPEL mechanisms is provided by the MORE middleware [16]. IV. V ISION S YSTEM U NIT In order to realize the detection, identification and localization of bins (cf. Figure 5(a)) within an decentralized material flow system, “Vision System Unit (VSU)” – camera together with “embedded evaluation unit” – for controlling a switch at an intersection point is used (cf. Figure 3). The central hardware unit of every VSU (“gumstix overo” 1 ) is equipped with standard LAN interfaces and usb ports. The gumstix (OMAP 3530 - 720 MHz, 256MB RAM, 256MB Flash) is running a tailored Linux. As input each VSU receives the current camera image and information from a central database such as currently used bin-ID’s within the material flow system or selected bin-ID’s within the VSUs local neighborhood detected by RFID. The output consists of the detected bin-ID and a confidence value. The confidence value describes the 1 Gumstix

Inc.,http://www.gumstix.com

defelcting belt switch bin

Fig. 3. Overview of the conveyor scenario representing a combination of a vision-based system

plausibility of the identification result. If a confidence value is about a certain threshold the VSU can trigger a switch directly. A. MORE Middleware Resource Management for VSUs The controlling and observation of system resources in the “embedded evaluation unit” of the VSUs is the domain of the resource management. The holistic approach of the resource management handles all system resources in a combined fashion in order to take into account the nature of different resources and their trade-off. •

•

Optimize system behavior: This optimization addresses the execution and adaptation of services/applications, e.g. when a bin is far away, not the complete detection power is needed and the frame rate can be reduced. Guarantee proper execution of critical applications/services: The objective is to ensure that critical applications/services perform according to certain requirements. If a bin is detected, the regarding switch must be notificated immediately.

A number of existing optimization techniques are the building blocks for the holistic resource management. These include ”Dynamic Voltage and frequency scaling”, ”Dynamic Power management”, ”Dynamic Application Adaptation” and ”Bandwidth/link availability”. The structure of the MORE middleware resource management is depicted in Figure 4. As one can see the resource management acts as the interface between the middleware and the system resources of the “embedded evaluation unit”. A resource that is crucial for sensor networks is energy. One component besides the network interface which harvests a lot of energy is the main processor. A service oriented management of the dynamic voltage and frequency scaling capabilities which only provides full computational power when needed, reduces power consumption of “embedded evaluation unit” and hence of the complete conveyor system.

MORE Middleware

Resource Management Service

DVFS Unit

MORE Middleware Core

OS/Hardware-Layer

Fig. 4.

DVFS Module (a)

B. Image Pre-Preprocessing For improving the image data acquired by a standard camera within a pre-processing step, image enhancement techniques are applied according to the requirements of the concrete camera. Those enhanced techniques are for example smoothing of noisy images [17] or deinterlacing methods to obtain a full image from half images containing only odd or even pixel lines [18]. In this work an intraframe deinterlacing method proved to generate the best results for marker identification. It calculates missing lines as arithmetic mean of neighboring lines and requires little computational time. C. Detection and Feature Identification With regards to the detection of a single bin, different features can be used. Using colored bins, an efficient approach is the color-based feature detection based on color histograms. A bin is detected within an image region, if the corresponding histogram is similar in sense of the Bhattacharyya distance [19] between the object’s and the template’s histogram. To gain robustness against changing lighting conditions, the template’s histogram can be adapted towards the currently detected object. Integral histograms [20] can be used for efficient histogram calculations. Because of the large number of different bins and a limited number of distinguishable colors, a pure color-based feature is not sufficient for an unique identification, but it is sufficient for a coarse preselection. To solve this problem a marker-based feature can be used. By identifying and reading visual markers attached to the piece goods, the order can be evaluated and a routing decision is possible. Bins of the considered type in this work are shown in Figure 5(a).Internationally standardized Data-Matrix and Quick-Response Marker are used in this work [21], [22]. To identify piece goods the attached two-dimensional marker has to be read. Data Matrix and Quick Response markers contain finder patterns, to determine where markers are present (detection) and how a found marker is oriented in space. Markers are found in edge-images by searching for closed contours describing the outline of a marker’s finder pattern (cf. Fig. 5(b)). Contours are approximated to obtain straight lines, which can be checked for some geometrical constraints. A single part of a Quick Response finder pattern has to consist of three nested squares, with given area ratios. Locations of

(b)

Fig. 5. (a) Intersection Point with Marker-equipped Bin and (b) an applied Sobel Filter (inverted Image) for Edge Detection

found finder patterns are refined by scanning pixel lines and searching for color changes from white to black and vice versa. If the marker is located, its position and size are known, so a grid of sample points can be laid over the found image area. At every grid point a module of the marker can be read. D. Tracking Using a video camera, 10 and more highly coherent images per seconds are available. To benefit from this coherence, a Bayesian filter based tracking algorithm, the so-called condensation algorithm [23], is performed to track multiple image regions containing piece goods. This way it is possible to determine the trajectories of bins over image sequences and to prevent multiple detections of the same object. Consequently not only a single image but multiple image data can be used for the identification of the same marker. Another advantage is the limitation of the search area for the marker detection and identification within the image, which leads to energy saving. V. R ESULTS Working with real-world data, image processing is always a challenging task due to lighting changes, shadows, partial occlusions or noise. Therefore it is important to quantify how confident the VSU’s output information is. The confidence value can depend on several properties like the identification consistency of a tracked piece good within an image sequence or the correct combination of the detected bin color and the attached marker ID, which can be requested via the MORE middleware from the global RFID database. But also partial occlusions of the marker or the amount of noise could have influence on the confidence value. The confidence values are provided to other VSUs via the mentioned publish-/subscribed methods of the MORE middleware, in order to allow other VSU to recheck or to increase the confidence of an identification. The real time image processing on an embedded system such as the used gumstix is a challenging task, e.g. our edge detection filter has a runtime of 50 milliseconds per frame (Resolution 320x320 pixels). The detection of the marker is nevertheless of sufficient quality (cf. 5(b) for an example output) and allows the identification of the marker. Optimization

3,50 Setup 1

R EFERENCES

Setup 2

3,30

3,10

2,90

power [W]

2,70

2,50

2,30

2,10

1,90

1,70

1,50 0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

time [s]

Fig. 6.

DVFS Results for a gumstix

can be achieve, when using an additional marker encoding just a hash-code, which consists of less information, instead of encoding the complete identification number of one marker. Leaving the size of the marker unchanged, a larger module size is possible due to less coded information and consequently the marker can be read from greater distances. This scaling makes another consistency check possible, which can be also reflected in the confidence value: if the piece good moves towards the camera, the hash-code (detected in greater distance) has to match the currently detected complete ID. To avoid detection of two markers, one combined marker with areas of different module size can be used instead. Stepping towards an energy saving system, the confidence value could also be used to control the processing frame rate of the VSU: as long as the confidence of an tracked piece good is below a confidence threshold, the system has to run in full speed mode. Decreasing the frame rate or even stopping identification of a piece good within a tracked image region can save computational time and power. This adaptation is done by the resource management service of each VSU as described in section IV-A. In Figure 6 one can see that in the intervals without identification the CPU frequency is reduced and in the intervals with identification the CPU is in a full speed mode. This reduces power consumption of “embedded evaluation unit” and hence of the complete conveyor system. VI. C ONCLUSIONS This paper described the utilization of a web service based middleware in a material flow system supported by a visionbased identification system. The middleware is tailored towards the use in sensor/actuator networks and supports the principals of a highly scalable and highly (re)-configurable automated facility logistics systems. Due to the use of low-cost components such as a low-cost camera and further due to an efficient and flexible communication approach, the proposed solution presented in this paper makes the deployment, configuration and operation of automated facility logistics system much faster and consumes less energy, thus will reduce the costs of the complete system. From the logistical point of view local controlled conveying systems will be able to become competitive in terms of costs and in conclusion its an important step towards the physical Internet of things in facility logistics.

[1] S. Sarma, D. L. Brock, and K. Ashton, “The networked physical world, proposals for engineering the next generation of computing, commerce & automatic-identification,” MIT Auto-Id Center, Tech. Rep., 2000. [2] E. Fleisch and F. Mattern, Eds., Das Internet der Dinge Ubiquitous Computing und RFID in der Praxis. Springer-Verlag, 2005, in German. [3] C. Floerkemeier, M. Langheinrich, E. Fleisch, F. Mattern, and S. E. Sarma, Eds., The Internet of Things. First International Conference, IOT 2008, Zurich, Switzerland, March 26-28. Springer, 2008. [4] H.-J. Bullinger and M. ten Hompel, Internet der Dinge. SpringerVerlag, 2007, in German. [5] W. G¨unthner and M. ten Hompel, Internet der Dinge in der Intralogistik. Springer-Verlag, 2010, in German. [6] M. ten Hompel and T. Schmidt, “Warehouse management,” SpringerVerlag, Berlin Heidelberg New York, 2008. [7] L. M. Ni, Y. Liu, Y. C. Lau, and A. P. Patil, “Landmarc: indoor location sensing using active rfid,” Wirel. Netw., vol. 10, no. 6, pp. 701–710, 2004. [8] Siemens Technical Press, Industry Sector, “Vereinzelung auf kleinstem Raum mit dem Visicon Singulator von Siemens,” 2009, in German. [9] A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys (CSUR), vol. 38, no. 4, 2006. [10] F. Weichert, D. Fiedler, J. Hegenberg, H. M¨uller, C. Prasse, M. Roidl, and M. ten Hompel, “Marker-based tracking in support of rfid controlled material flow systems,” Journal of Logistics Research, Springer, vol. 2, pp. 13–21, 2010. [11] G. Moritz, E. Zeeb, S. Pr¨uter, F. Golatowski, D. Timmermann, and R. Stoll, “Devices profile for web services in wireless sensor networks: adaptations and enhancements,” in ETFA’09: Proceedings of the 14th IEEE international conference on Emerging technologies & factory automation. Piscataway, NJ, USA: IEEE Press, 2009, pp. 43–50. [12] S. Chan et al., “Device Profile for Web Services,” OASIS, Tech. Rep., February 2006. [Online]. Available: http://schemas.xmlsoap.org/ws/2006/02/devprof/ [13] A. Cannata, M. Gerosa, and M. Taisch, “Socrades: A framework for developing intelligent systems in manufacturing,” dec. 2008, pp. 1904 –1908. [14] J. Schmutzler, U. Bieker, and C. Wietfeld, “Network-centric middleware supporting dynamic web service deployment on heterogeneous embedded systems,” in 14th International Conference on Concurrent Enterprising. Lisboa, Portugal: ICE, June 2008. [15] A. Wolff, S. Michaelis, J. Schmutzler, and C. Wietfeld, “Network-centric middleware for service oriented architectures across heterogeneous embedded systems,” in The Eleventh International IEEE EDOC Conference Workshop, 2007, vol. 1. Annapolis, MD, USA: IEEE, October 2007, pp. 105–108. [16] C. Timm, J. Schmutzler, P. Marwedel, and C. Wietfeld, “Dynamic Web Service Orchestration applied to the Device Profile for Web Services in Hierarchical Networks,” in COMSWARE ’09: Proceedings of the Fourth International ICST Conference on COMmunication System softWAre and middlewaRE, Dublin, Ireland, 06 2009, pp. 1 – 6. [17] R. Gonzalez and R. Woods, Digital Image Processing, 3rd ed. Prentice Hall International, 2008. [18] S. Keller, F. Lauze, and M. Nielsen, “Deinterlacing using variational methods,” IEEE Trans Image Process, vol. 17, no. 11, pp. 2015–2028, 2008. [19] A. Bhattacharyya, “On a measure of divergence between two statistical populations defined by probability distributions,” Bulletin of the Calcutta Mathematical Society, vol. 35, pp. 99–109, 1943. [20] F. Porikli, “Integral histogram: a fast way to extract histograms in cartesian spaces,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR’05, vol. 1, pp. 829–836, 2005. [21] “Norm ISO/IEC 16022 2006. Information technology - Automatic identifcation and data capture techniques - Data Matrix bar code symbology specification,” 2006. [22] “Norm ISO/IEC 18004 2006. Information technology - Automatic identifiation and data capture techniques - QR Code 2005 bar code symbology specification,” 2006. [23] M. Isard and B. A., “CONDENSATION - conditional density propagation for visual tracking,” International Journal of Computer Vision, Springer-Verlag, vol. 29, no. 1, pp. 5–28, 2004.