An FPGA based general purpose data acquisition controller

2092

IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 4, AUGUST 2006

An FPGA- Based General-Purpose Data Acquisition Controller C. C. W. Robson, A. Bousselham, Member, IEEE, and C. Bohm, Member, IEEE

Abstract—System development in advanced FPGAs allows considerable flexibility, both during development and in production use. A mixed firmware/software solution allows the developer to choose what shall be done in firmware or software, and to make that decision late in the process. However, this flexibility comes at the cost of increased complexity. We have designed a modular development framework to help to overcome these issues of increased complexity. This framework comprises a generic controller that can be adapted for different systems by simply changing the software or firmware parts. The controller can use both soft and hard processors, with or without an RTOS, based on the demands of the system to be developed. The resulting system uses the Internet for both control and data acquisition. In our studies we developed the embedded system in a Xilinx Virtex-II Pro FPGA, where we used both PowerPC and MicroBlaze cores, http, Java, and LabView for control and communication, together with the MicroC/OS-II and OSE operating systems. Index Terms—Embedded system, FPGA, networking, object oriented, RTOS, UML.

time. When problems inevitably arise, this dual development environment often makes it difficult to determine whether they are due to hardware or software errors. Network programming is also a major challenge. While a number of small TCP/IP stacks are freely available, they are not trivial to use, and do not always implement the full Berkeley Socket Application Programming Interface (API). To bypass the higher layers for increased execution efficiency and a smaller memory footprint requires a deeper understanding of the subject [1]. Poor documentation is unfortunately very common. Embedded development with an RTOS adds another layer of complexity, requiring the developer to make decisions about such issues as how many tasks should be run and whether all resources should have their own task, stack sizes, priority levels, and scheduling [2]. Real-time systems also require the developer to have expertise in concurrent programming, dealing in topics such as deadlocks, sharing of data, race conditions, and priority inversion [3].

I. INTRODUCTION

III. A FRAMEWORK FOR PROTOTYPING

N

ETWORK-based controllers have become increasingly popular for control and data acquisitions systems as available bandwidth has increased over time. FPGA-based systems are especially interesting because they allow parts of the system to be easily migrated between hardware and software implementations for optimal performance and resource use. These advanced capabilities, however, come at a cost. To develop a network-based controller, especially together with a Real-Time Operating System, RTOS, requires the developer to possess skills in a number of software areas. We have developed a framework for a network-based, controller suitable for FPGA-based systems that can be used on a number of different platforms without any major changes. In most cases, only a simple change of header files is necessary. In this paper, we present a suitable development process for embedded software controllers, and how our framework supports this process. We will show a suitable object-oriented architecture for such a controller. II. CHALLENGES IN EMBEDDED DEVELOPMENT

In embedded development one is often developing the hardware- and the software components of the system at the same Manuscript received June 27, 2005; revised April 25, 2006. The authors are with the Department of Physics, Stockholm University; Stockholm, Sweden (e-mail: [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/TNS.2006.878698

These challenges can become an overwhelming obstacle, especially if the goal is to quickly build a prototype for testing and development. Our solution was to study where potential problems might arise while developing a controller prototype and develop a framework that effectively removes the need for programming skills in these areas. The result is a collection of objects implemented as APIs and modules that encapsulate the most difficult tasks and remove many of the potential pitfalls. We have used an object-oriented approach [4], [5], resulting in an architecture that is easy to adapt, understand, and maintain. By dividing the system into objects we also make it easier in the future to implement some of the parts in hardware without having to change the architectural view. IV. HARDWARE PLATFORMS Our FPGA-based development work so far has been based on hardware platforms using two different embedded CPUs. The Xilinx MicroBlaze is an established soft-core IP. It is based on a 32-bit RISC architecture and is a natural choice in smaller projects where CPU power is not essential. It is also the only choice for Xilinx FPGAs without an embedded hardcore CPU. The MicroBlaze can be run at speeds up to 200 MHz and can be extended with additional coprocessors, instruction sets, and so forth. Fig. 1 shows the architecture we used for MicroBlaze with our controller. The IBM PowerPC 405 is a full 32-bit RISC CPU that is embedded in the Xilinx FPGA Virtex-II Pro and Virtex-4 FPGA families. It can be run at speeds up to 450 MHz and is the best

0018-9499/$20.00 © 2006 IEEE

ROBSON et al.: AN FPGA- BASED GENERAL-PURPOSE DATA ACQUISITION CONTROLLER

2093

Fig. 1. The MicroBlaze based hardware architecture we used for our controller. The MicroBlaze is connected to the Block RAM (BRAM) through the Local Memory Bus (LMB). The rest of the peripherals are connected to the CPU by the On-chip Peripheral Bus (OPB). EMC and MDM are acronyms for External Memory Controller and Microprocessor Debug Module.

Fig. 2. The PowerPC-based hardware architecture we used for our controller. The PowerPC can make use of the Processor Local Bus (PLB), which is faster than the OPB. Both are part of the IBM CoreConnect bus architecture.

Fig. 3. The controller—a standalone static view of the controller as a UML class diagram. Here we can see how the basic parts relate to each other.

choice for CPU-intensive tasks in Xilinx FPGA-based systems. Fig. 2 shows the architecture we used for PowerPC with our controller. When developing embedded software, we are always facing the host-target problem, meaning, we are developing the system on one machine, the host, and running it on another, the target. In order to solve this problem, we decided to also support the 86 architecture from Intel. By doing so, it is possible to develop a large part of the system on an ordinary PC running Windows or any variant of Unix. By simulating IP cores (firmware modules) as software objects, a system can be developed to an advanced state before it needs to be tested on the

actual target. Another benefit of this approach is that it allows concurrent development of multiple projects on a single target. V. SOFTWARE ARCHITECTURE The controller architecture design is based on four modules: the Server, Model, Protocol, and Resource modules, which is shown in Fig. 3. They are designed as classes and interfaces, and we have deliberately designed the Server, Model, and Resource modules to have loose dependencies between them. The Server class is responsible for communication with the client. It can be implemented with different TCP/IP stacks, as

2094

IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 4, AUGUST 2006

Fig. 4. The controller—a standalone dynamic view. In this UML sequence diagram we can see how messages are exchanged in the controller as a result of a request by a client.

long as they implement the Socket interface. We have implemented it as a common API for Winsock1, Unix, Xilnet2, and LWIP3, and to choose between them is a simple matter of selecting the desired header file. The Server behavior is the same for different stacks, except for the XilNet case. XilNet is simple and small, but not easy to work with. It lacks a connect function, so if a client module is required, a connect function must be written. XilNet’s accept function is different from that of Berkeley sockets in that it isn’t blocking, probably because it is designed for simple controllers without interrupts or RTOS. The server can operate in three modes: I—the client connects, the server receives a command, sends data to the client and then closes the connection. II—the client connects, and then the 1Winsock

is a part of Microsoft Windows. Net - A networking library which is a part of the Embedded Development Kit, EDK, from Xilinx. 3LWIP is a light-weight implementation of the TCP/IP protocol suite that was originally written by A. Dunkels of the Swedish Institute of Computer Science. 2LibXil

server repeats the sequence of receiving a command, sending data until the client disconnects the connection, it then closes the connection. III—the client connects and the server receives a command. It then repeats sending data to the client until the client disconnects the connection. It then closes the connection. The client can choose the mode at run time. The Model class is the core logic, the “business logic” of the controller. The model interprets incoming commands from the server with the help of methods defined in the Protocol interface. It determines what resource in the embedded system the client is addressing, and executes the command from the client together with its parameters. The Protocol interface defines a number of methods that all protocol classes in this architecture must implement. In this way it is possible to develop and use many different protocols at the application layer without needing to change anything in the rest of the controller. Currently, we are using http with a web browser as a thin client, and we have developed an application

ROBSON et al.: AN FPGA- BASED GENERAL-PURPOSE DATA ACQUISITION CONTROLLER

2095

Fig. 5. The controller—an operating system static view of the controller as a UML class diagram. In this design we are running the server, model, and the resources as individual tasks. Another approach could be to run the server and the model in the same task.

layer protocol suitable for LabVIEW4 and Java, allowing us to use LabVIEW and Java applications as a client. The Resource class represents objects, functions and IP cores in the system. A resource can, e.g., implement a client, connected to another server, allowing the controller to distribute commands from the original client and exchange information with other network-based controllers, thus making the system highly scalable. A resource can even be another server or a full controller; for example, one could implement one server for distributing control signals and another for exchanging data in the same embedded system. Fig. 4 shows a dynamic view of how the controller handles a simple request, running as standalone. The object-oriented approach simplifies migration from a standalone to an RTOS-based system. The objects represent conceptual parts of the controller with loose dependencies between them. The loose dependencies make it possible to divide the system into separate tasks without any major changes. Fig. 5 presents an example of how this is done. This design has been successfully tested with MicroC/OS-II [6] as an RTOS, but any RTOS could be used. To promote simplicity in development, intertask communication has thus far been implemented by shared variables. We solved many of the inherent problems in this approach by turning off interrupts when addressing these shared variables. This has a large negative impact on system performance, but the benefit of it is a much faster prototyping cycle. VI. IMPLEMENTATION AND RESULTS We have implemented MicroBlaze and PowerPC-based systems in a Xilinx Virtex-II Pro (XC2VP7) on a development board from Memec, FF672 Rev 2. The firmware part of the 4LabVIEW

from National Instruments is a graphical development environment for creating applications.

system was developed and implemented with the Embedded Development Kit, EDK, version 6.3 and 7.1 from Xilinx. The software controller was designed with Objecteering/UML v5.3.05. When implemented in an FPGA, it was possible to limit the size of the controller to below the 64 kB of BRAM. The controller has successfully been tested on several different processors (MicroBlaze, PowerPC, x86, SPARC, and ETRAX) and operating-systems (Windows, Solaris, Linux, MicroC/OS-II, Xilinx Xilkernel, and OSE6). This framework has been successfully used to implement network based controllers on embedded systems based on MicroBlaze [7] and an ETRAX 100LX7 from Axis Communications [8]. We have studied the data throughput in different situations. With a MicroBlaze running at 100 MHz and executing three parallel threads in Xilkernel we have achieved 1.6 MBit/s, when requesting data of small sizes, well within the Maximum Transmission Unit (MTU) of 1500 Bytes for Ethernet. With larger data sizes, this figure will increase due to smaller communication overhead. VII. CONCLUSION We have developed a framework suitable for rapid development of an embedded controller. This framework circumvents many of the difficult issues associated with network-based controllers, making it usable even by less-experienced developers. The most valuable part of this approach, however, is probably that such a framework encourages a coherent, standardized design method. This allows effective re-use of design elements in new projects, and reduced development time. 5A

UML CASE-tool from Objecteering Software is a registered trademark of ENEA Embedded Technology. 7A full system-on-a-chip implementation from Axis Communications 6OSE

2096

IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 4, AUGUST 2006

These advantages, however, come at the cost of decreased performance when it comes to execution speed and code size for the prototype controller. Further optimization will require a more in-depth understanding of system and software issues. We are currently using this framework in our own projects and we plan to develop it further. The distributed potential is of particular interest to us, as it is the key to developing large, scalable, independent data acquisition systems. ACKNOWLEDGMENT The authors would like to thank S. Silverstein for initial help with LabVIEW. REFERENCES [1] W. Richard Steven, “TCP/IP Illustrated, Volume 1: The Protocols Addison-Wesley, 1994 and,” in TCP/IP Illustrated, Vol. 2: The Implementation. Reading, MA: Addison-Wesley, 1995.

[2] M. Barr, Introduction to Rate Monotonic Scheduling. 2002 [Online]. Available: www.embedded.com/story/OEG20020221S0089 [3] Q. Li and C. Yao, Real-Time Concepts for Embedded Systems. Manhasset, NY: CMP Books, 2003. [4] B. P. Douglass, Real Time UML: Advances in the UML for Real-Time Systems, 3rd ed. Reading, MA: Addison-Wesley, 2004. [5] B. P. Douglass, B. P. Douglass, and B. P. Douglass, Designing Real-Time Systems With UML-Part I, Designing Real-Time Systems With UML-Part II, Designing Real-Time Systems With UMLPart III. 1998 [Online]. Available: www.embedded.com/98/9803fe2. htm, www.embedded.com/98/9804fe3.htm, www.embedded.com/98/ 9805fe3.htm [6] J. J. Labrosse, MicroC/OS II: The Real Time Kernel, 2nd ed. Manhasset, NY: CMP Books, 2002. [7] A. Bousselham, C. Robson, P. E. Ojala, and C. Bohm, “A Flexible Data Acquisition Module for a High resolution PET Camera,” in Proc. IEEE-NPSS Real Time Conf., 2005. [8] S. B. Silverstein, J. Rosenqvist, and C. Bohm, “A simple Linux-based platform for rapid prototyping of experimental control systems,” IEEE Trans. Nucl. Sci., vol. 53, no. 3, pp. 927–929, June 2006.