RFnest™: Radio frequency network emulator simulator tool

RFnestTM: Radio Frequency Network Emulator Simulator Tool Justin Yackoski, Babak Azimi-Sadjadi, Ali Namazi, Jason H. Li, Yalin Sagduyu, and Renato Levy Intelligent Automation, Inc., Rockville, MD, USA Email: {jyackoski, babak, anamazi, jli, ysagduyu, rlevy}@i-a-i.com Abstract—RFnestTM is an FPGA based network channel emulator that allows all of the channels for a full network of radio nodes to be emulated in real time, with all communication experiencing a realistic channel impulse response and correct interference. RFnestTM is the first network emulator that allows virtual simulated nodes and real RF nodes to interact and have a shared wireless feeling. In other words, the real RF nodes in RFnestTM receive the signals sent by the virtual simulated nodes on their real radios and vice versa. This capability allows both high fidelity and scalable network simulation and emulation within the same controlled and repeatable environment. This allows RFnestTM to be used for protocol testing, replaying field tests, and model validation. RFnestTM has a modular design with three main components: 1) FPGA based emulation hardware with RF front ends to allow nodes with real radios to send their RF signal over an emulated channel without any modification to the radio, 2) modeling of time-varying channel impulse responses within the emulation hardware, with channel properties based on mobility defined with a scripted or interactive GUI environment, and 3) integration with network emulators and monitoring functionality allowing the user to instantiate, manage, and monitor real and virtual network nodes within the scenario.

I. I NTRODUCTION Despite tremendous research and interest in wireless networks, current capabilities are insufficient to allow efficient test and performance evaluation of wireless protocols and systems. The main reason for this is the different nature of wired and wireless networks rendering the use of wired network techniques inappropriate for the case of wireless networks. In a wireless network the transmission power, modulation, coding, interference, node mobility, and channel conditions all vary and induce significant fluctuations in key quantities such as link capacity. Due to these performance variations and their interdependencies, which are not present in wired networks, the accurate testing and evaluation of wireless systems is a daunting task. Modeling and model-based approaches that consider complex relations, interactions, and interdependencies between wireless connections in a network are needed to help us in development of a systematic and scalable methodology for design, test and evaluation of wireless networks and protocols. The current testing and evaluation frameworks and solutions for wireless networks can be categorized into two groups: Software based simulators/emulators, and Testbeds. The common problems with software based simulators such as nsApproved for Public Release; Distribution Unlimited: 88ABW-2011-2294, 19APR11

2, Opnet and Qualnet are simplistic and unrealistic physical layer and channel models. In one approach, the physical layer transceiver models are developed in Matlab Simulink and they are integrated to the network models developed in Qualnet to estimate the bit error rates. The main issues with this hybrid approach are speed and accuracy. Since the physical layer models are developed in software, they are usually very slow and this approach is not scalable for large networks. Secondly, special attention has to be made to make sure that the Matlab models are realistic and capture some non-linear and unexpected characteristics of the transceiver systems. The experimentation networks (testbeds) are the alternative approach for testing and evaluation of the wireless networks. The main problems with the use of real-size testbeds are cost as well as a lack of flexibility and repeatability of the experiments. The testbeds are built in labs and cannot be easily used to experiment different environmental and terrestrial conditions that can happen in the operational circumstances. For instance, the RoofNet project [1] is conducted over the city of Cambridge and uses 40 nodes. The ORBIT testbed project [2] contains 400 (20x20) 802.11 nodes that form a fixed grid that are installed in an indoor lab platform. The repeatability of the experiments is a main requirement that is hard to manage in realistic testbeds. In the experimental testbeds, since the environment is not completely under user’s control, it is very hard to reproduce the conditions of one experiment completely. CMU’s FPGA-based wireless channel emulator [3] provides a network channel emulation with high fidelity but lacks the scalability that is required in wireless channel evaluation. In this paper we introduce RFnestTM which bridges the high fidelity of a network channel emulator and the scalability of a software based network emulator by allowing a wireless network to be evaluated using a combination of real and virtual radios. When high fidelity is important and many real radios are available, it is desirable to connect possibly several dozen wireless nodes where both the radio and operating system stack are real. In this case, only the transmission medium (i.e. the channel properties) is not completely real. Assuming high-fidelity models are used to set the channel properties, a high confidence in the evaluation results can then be attained. Second, when an evaluation requires a network containing a large number of nodes (e.g. 200), it is desirable to not give up the fidelity possible for a smaller number of nodes but to instead seamlessly augment the set of high-fidelity nodes with additional lower-fidelity nodes with virtual radios.

RFnestTM provides the following capabilities, significantly reducing the time and cost of evaluating wireless networks: • network evaluation in a controlled, repeatable, and realistic mobility model environment with the same radio used in the field • a hybrid software/hardware network emulator to provide scalability as well as high fidelity • replaying of field tests with all their complexity in a lab environment • model validation by creating identical scenarios for real radios and radio models In this paper we describe the components of RFnestTM and how these components are integrated. In Section II we describe the hardware (HW) components which consists of a) daughter boards which act as an interface between the radio under test and the channel emulator, and b) the main FPGA based channel emulator that emulates channel impulse response for all possible channels. In Section III we describe the emulator software and its components that allow for virtual and real nodes to communicate seamlessly. Section IV gives the details of our preliminary tests with RFnestTM HW and its capability to provide wireless hardware in the loop capabilities. We conclude the paper in Section V. II. H ARDWARE I MPLEMENTATION OF RF NEST TM A picture of network channel emulator with 8 RF interface nodes is shown in Figure 1. The implemented network channel emulator consists of 1 main Xilinx vertex 4 FPGA for emulating the wireless channel and 8 (with maximum 12 node capability) RF interface nodes. Each RF interface node includes a Spartan 3 FPGA, an A/D, a D/A, and an RF front end. This HW interfaces with software components including the channel models and the driver/kernel of each real radio. These software components are described in Section III, including the modifications needed to allow interactions between real nodes (with actual RF transceivers) and virtual nodes (with an emulated OS stack and modeled RF communication). The current HW channel emulator design has the following specifications: • RF Band: 2.4 GHz (Wi-Fi) • Tuning Range: 100 MHz • Instantaneous Bandwidth: 24 MHz • Sampling Frequency: 64 MHz • Intermediate Freqency (IF) Center Frequency: 20 MHz • IF Start Frequency: 8 MHz • IF End Frequency: 32 MHz • Sample Resolution: 12 bits DAC, 10 bits ADC • Output Dynamic Range: 50 dB (digital) + 20 dB (analog) • Number of Nodes: 8 half-duplex nodes (up to 12) • Filter Taps: 3 • Primary Delay: 0-16 µs for all nodes, 0-500 ms for one node

Fig. 1. IAI 8-node Network channel emulator hardware and exterior/GUI (inset). To isolate the nodes from each other to minimize the cross talk and unwanted interference each node is shielded properly. The shielding is not shown in the picture.

Virtex4 Main Board Node0

Node Interface

Node Interface

Node4

Node1

Node Interface

Node Interface

Node5

Node2

Node Interface

Node Interface

Node6

Node3

Node Interface

Node Interface

Node7

External Memory (64MB)

External Memory Interface

Ethernet Interface (uBlaze)

Ethernet Link

Fig. 2.

3-Tap Filter Bank

Controller Unit

Main FPGA board block diagram.

Tap Delay: 0-1 µs The emulator consist of two main blocks: 1) Main FPGA Board: where the channel impulse responses, Ethernet controller and configuration management is implemented. 2) RF Nodes: eight equivalent blocks where downconversion to baseband, up-conversion to Wi-Fi, halfduplex control is implemented in each of the 8 RF node controllers. •

A. Main FPGA Board The block diagram of the main FPGA board is shown in Figure 2. In the filter bank there are 28 filters in total, and each filter models the channel between two real nodes. In general, for N nodes there are Nx(N-1) channels between nodes, and assuming the channels are reciprocal, half of the channels need to be modeled inside the FPGA. The block diagram of a 3-tap filter is sketched in Figure 3. The first delay block is implemented in memory, while the other two smaller delays are implemented in FPGA fabric

3-Tap Filter Delay (16uS)

Delay (1uS)

Delay (1uS) Attn

Amplitude/ Doppler

Amplitude/ Doppler

Amplitude/ Doppler

T/R Switch

Settings (Controller)

COM1200 Board

Up Conversion

D/A

Frequency Synthesizer

Configuration Management

Down Conversion

A/D

Gain Control

SPI

SPI Controller

2 bits

Data

Mux/ Demux

12 bits

T/R

Power Detect

1 bit

Data Out (I & Q)

Fig. 4.

RF Node block diagram.

3-Tap Filter.

CORE Real Ghost

Real Ghost

Real Node

Real Node

NEM

Queue Radio

Queue Radio

MAC / PHY

NEM

SVT

Channel

Instrumentation

FPGA-based Propagation Controller

Event Generator

Component added/modified by IAI

Fig. 5.

Our EMANE- and CORE-based architecture.

Management

Transmission patterns

CORE Virtual Node

Real Ghost

Real Ghost

Real Node

Real Node

R to V Packets

V to R Packets

Transport

SVT updates Radio

NEM

Radio

SVR updates NEM

SVT

Queue

Queue

Network Interface

SVR

Queue

Virtual Node

EMANE

MAC / PHY

1) Channel Matrix Updates: This message includes channel filter parameters and these parameters are stored in a memory inside FPGA to be used for channel emulation. 2) Mapping Updates: This message includes virtual node mappings and is used to reconfigure specific channel parameters with updated virtual node settings. 3) Interference Updates: This message includes parameters to model the interference that should be received by the real nodes. The FPGA uses these parameters to generate additive interference at the channel endpoint.

SVR

Transport

Instrumentation

Channel

Multiple software components running on multiple devices (e.g. PCs running the network emulation and channel model, real nodes, etc.) need to continually update the FPGA hardware with new channel matrix settings and node mappings. This communication is implemented via Ethernet. To receive the Ethernet packets, the FPGA is connected to a 10/100 Ethernet PHY which provides 4-bit receive and transmit data, synchronized receive and transmit clock signals and related handshaking signals. The FPGA listens on a pre-defined multicast UDP socket. To reduce processing, we require all communication with the emulation hardware to use a subset of normal networking protocols (e.g., IPv4-only, no fragmented packets, etc.). Note that this is only for internal control traffic within RFnestTM , and does not constrain the traffic sent by the nodes under evaluation. The FPGA processes three types of UDP multicast message as follows:

Network Interface

EMANE

B. Host Interface

Queue

Virtual Node

Radio

Virtual Node

Radio

using look-up tables (LUT). The multipliers are in the complex domain, hence each multiplication process needs four multipliers.

Management

Fig. 3.

Radio

Complex Multipliers running at 4x speed

Radio

Data In (I & Q)

RF Board

Interference patterns Event Generator

Data from virtual to real

FPGA-based Propagation Signal Controller Signal

Data from real to virtual

Channel effects

C. RF Node The block diagram of the RF node is drawn in Figure 4. In this diagram, the A/D resolution is 10 bits and the D/A resolution is 12 bits and they share a 12 bit bus. Since the down-conversion is to baseband, we have I/Q data in the communication between the main board and the RF nodes.

Fig. 6.

Interaction paths between real and virtual nodes.

III. RF NEST TM S OFTWARE C OMPONENTS The RFnestTM software architecture is shown in Figure 5. In a wireless network, there are essentially two aspects of each

node which must be considered. First, the operating system stack, meaning the network layer and above, must exist or be emulated. Second, each node must have either a real wireless radio or a virtual radio via MAC and PHY layer simulation model. Building upon existing methods of creating and connecting these aspects, we have created additional components which allow an integration between real and virtual radios. Below, we describe each such component and their interactions as shown in Figure 6. A. CORE CORE (Common Open Research Emulator) [4] is used to manage the scenario being tested including creating the emulated virtual nodes via Linux network namespaces, configuring EMANE, allowing the user to move nodes via the GUI, and configuring the routing and other network behaviors on both the real and virtual nodes. Since CORE uses EMANE’s event notification API and PHY/MAC models, we have not modified CORE itself. Nonetheless, CORE plays a significant role in the emulation architecture. B. EMANE EMANE (Extendable Mobile Ad-hoc Network Emulator) [5] provides three key features which are used heavily by our emulation architecture. First, EMANE has an event service API which many existing software uses, including CORE, mobility generators, terrain data, as well as various PHY, MAC, and other models provided both by EMANE and third parties. This allows us to easily integrate with existing radio models and other software written for EMANE with little to no changes. For example, we use the EMANE event service to receive node position updates from CORE. Second, EMANE provides a simulation infrastructure for wireless communication that includes a library of various PHY and MAC models. These models can be either used directly or modified to simulate new protocols. Third, EMANE provides virtual network devices which allow emulated nodes (i.e. those created by CORE) to communicate over what appears to be a normal network device. The virtual network device serves as an entry point for providing packets to and from the PHY and MAC models. We have made significant use of this extendable nature to easily develop new or modified EMANE components. 1) Real-Virtual Transport: To allow interaction between real nodes using our emulation hardware and virtual nodes created by CORE/EMANE, we have created an EMANE transport which serves as a bridge between data sent between the EMANE radio simulation models and the Surrogate Virtual Receiver/Surrogate Virtual Transmitter (SVR/SVT). In EMANE, a transport is responsible for transferring packets between each node’s OS stack and the corresponding simulated MAC and PHY layer. By monitoring the MAC addresses specified in packets, our EMANE transport decides whether to keep each packet sent by a virtual node within the “virtual world” (i.e. within EMANE’s models) or to give the packet to the SVT. Similarly, packets received from the SVR which

are addressed to a virtual node are injected into the EMANE simulation. 2) Event Generator: We have developed an Event Generator which is primarily responsible for calculating the current channel properties (e.g., pathloss, channel impulse response, Doppler, delay, etc.) based on both the environment and nodes’ current positions. The full channel properties are sent to the FPGA, while a simplified pathloss-only expression of channel’s properties is sent to the EMANE PHY and MAC models via the EMANE API. Our Event Generator’s role is limited to solely performing channel calculations as nodes move. Handling of individual packets within the virtual world is performed via EMANE’s existing infrastructure which acts upon each packet based on the calculated values we provide. C. Queueing and Channel Switching To facilitate communication between real and virtual nodes, we have created a Linux kernel module and driver modification which allows us to closely monitor and report the behavior of real radios. By tracking each packet as it passes from a network queue to the driver and monitoring the driver’s transmission of each packet, the kernel module is able to know precisely which packet the radio will attempt to transmit next. This information is provided to the emulation hardware via a UDP multicast sent over Ethernet. To minimize delay, the notification packet is written directly by our kernel module into the Ethernet buffer, similar to the way pktgen [6] operates. This notification allows the emulation hardware to modify the channels associated with the SVR and SVT appropriately. The necessary degree of time synchronization is dependent on the protocol and radio hardware used. In our testing using 802.11 DCF, we observe that the notification delay is approximately 100µs and as a result the Ethernet notification is received by the emulation hardware before the wireless signal in question is transmitted by the real radio. D. Surrogate Virtual Transmitter (SVT) The SVT acts as an intermediary for any packets sent from a virtual node to a real node. Packets are directed from the virtual nodes to the SVT using a combination of bridging and filtering rules configured on the EMANE computer, the SVT, and a managed Ethernet switch. Since the SVT notifies the emulation hardware of the virtual sender for each packet before it is transmitted, the channel properties between the SVT and each real node are constantly updated as the SVT’s “virtual identity” changes. A SVT can therefore be multiplexed and queue packets sent from many virtual nodes, limited only by the total bandwidth sent from virtual to real nodes in the evaluation scenario. The real nodes then receive and (possibly) decode these signals, unaware the packet was not sent directly from the virtual node. E. Surrogate Virtual Receiver (SVR) The SVR acts as an intermediary for any packets sent from a real node to a virtual node, forwarding packets received from real nodes to the EMANE models. Since the SVR does not

know which real node is about to transmit to it at a given moment, the senders (i.e. the real nodes) have the responsibility of notifying the emulation hardware to appropriately update the channel between each real node and the SVR. Note that since each real node may be sending to a different virtual node and there are more real nodes than SVRs, the SVR assumes the role of multiple virtual nodes concurrently. While there is a limitation on throughput as with the SVT, more nuanced effects such as the fairness in the real radios’ channel access can be captured. F. Interference We believe it would be extremely difficult to literally synchronize the two (i.e. to have a virtual node sense a carrier at the moment a real node transmits). For 802.11, this likely requires notification within 10µs. Without highly specialized combined radio/notification hardware, such notification is not achievable. For example, using Gigabit Ethernet would result in a notification delay of over 25µs purely due to the switching delay alone [7]. Thus, we have adopted the approach of creating stochastically correct interference effects between the real and virtual worlds. To induce interference on virtual nodes when a real node transmits, the notifications described above to update the SVT and SVR’s identities are also used by EMANE. When a real node transmits, nearby virtual nodes should “feel” the interference of this transmission. Since a virtual node’s world exists solely in the EMANE model, we create ghost copies of each real node within EMANE. When a notification is received by EMANE, a similar packet is injected into the corresponding ghost node’s queue within EMANE and its transmission is then incorporated into the EMANE model’s behavior. Virtual nodes then compete for the channel against ghost nodes which desire the same channel resources as the real nodes, therefore mimicking the impact To induce interference on real nodes when a virtual node transmits, the EMANE models of each ghost node have been instrumented to report the nodes’ interference levels over time. In future work, the resulting interference patterns will be sent to the emulation hardware to allow a corresponding level of interference to be injected into the channels of the real radios. IV. R ESULTS In this section we show examples of RFnestTM in operation. For these evaluations, we have connected six Ubiquity RouterStation Pro devices with Atheros-based SparkLAN WMIA166AGH 802.11b/g wireless cards inside Netgate indoor enclosures and running a modified Linux-based OpenWRT firmware. These devices are used both as real nodes and as the SVT and SVR. We connect each wireless card via RF cables to the RFnestTM emulation hardware. This setup is shown in Figure 7. A. Basic Communication and Isolation To test basic operation, we configured all six devices as real nodes (numbered 0 to 5) and configured the channel properties

Fig. 7. Six 802.11 radios attached to the RFnestTM hardware (using an alternate backplane).

Fig. 8. Example of throughput through hardware across two links (in MBps).

to create three isolated links (i.e., node pairs 0 and 1, 2 and 3, 4 and 5, each has a gain of 1 between them, while all other links have a gain of 0). We then send UDP traffic across the links and measure the throughput. Figure 8 shows that the entire process from receiving an RF signal from one radio to reproducing that signal at the other radio (analog to digital conversion, digital to analog conversion, digital processing, etc.) works correctly. We observe low packet loss and a throughput of approximately 1 MBps. It is important to note that while we only show 2 links as an example, the emulation hardware supports “fully connected” scenarios where all eight radios can talk to (and interfere with) each other, giving a total of 28 bi-directional links. The properties of each link can be independently controlled based on the desired topology and scenario. To test isolation, we have verified that for two links that are configured in the scenario to be isolated, data sent across one link does not affect the throughput or loss on another link. Further, we sent broadcast packets from all eight nodes and observed no receipt of such packets across links with a gain of 0. B. Real and Virtual Interaction To test the real and virtual interaction, we configured the four black nodes shown in Figure 7 as real nodes and use one red node as an SVT and the other as an SVR. We then created

D. Use Cases

Fig. 9. Initial topology where traffic is sent directly between two real nodes.

Fig. 10.

Later topology where OLSR selects virtual nodes to forward.

macaddr: rssi 53 last_rx macaddr: rssi 34 last_rx

0.020000 0.610000

Fig. 11. Example of a real node receiving different signal strengths from each virtual node via a single SVT.

16 virtual nodes completely within EMANE. All nodes then run an unmodified OLSR routing protocol. We then send pings between two of the real nodes. Figure 9 shows a portion of the initial network topology where real nodes 0 and 1 are close to each other and communicate directly. A relevant subset of the 16 virtual nodes are also shown. We use tcpdump and EMANE instrumentation to monitor the path actually taken by each data packet and show the forward and backward paths by a dashed and solid arrow, respectively. When real node 1 moves to the other end of the network as shown in Figure 10, OLSR naturally and correctly discovers a set of appropriate virtual nodes through which to forward traffic. None of the nodes are aware of whether their neighbors are real or virtual, all nodes appear the same to the routing protocol. C. Channel and Identity Synchronization While the routing functions normally, we have taken further steps to verify that each virtual node is actually transmitting/receiving via the SVT/SVR over correct channels. To do this, we examined the output from the madwifi driver, which separately lists the RSSI values (in dB) by MAC address. Figure 11 shows an example of such output from one of the real nodes. The two MAC addresses listed correspond to the simulated wireless interfaces of two nodes created within EMANE. Although the packets transmitted by both virtual nodes are sent by a single SVT, the packets are received with RSSI values which differ by 20 dB. This occurs because as each packet is about to be sent by the SVT, the emulation hardware changes the channel properties to be appropriate for each packet’s virtual sender.

RFnestTM ’s capabilities allow for several interesting use cases that reduce the time and effort needed to evaluation wireless technologies: • Arbitrary network topologies can be easily created in the lab, including scenarios with mobility, directional antennas, and various channel conditions which are otherwise difficult to reproduce. • By adding virtual node nodes a large-scale network can be evaluated before proceeding with a field test. • An identical RF scenario can be given to both a set of real nodes and a set of virtual nodes, allowing detailed model verification and validation. • Channel measurements can be made in the field during a single field test of a desired scenario. These values can then be fed back into RFnestTM to easily replay and reevaluate the scenario in the lab with minimal cost and effort. V. C ONCLUSION We have shown that RFnestTM provides unique capabilities for evaluating wireless networks. By seamlessly integrating an accurate channel environment for a full network of real nodes with real radios with an emulated environment containing software-only nodes, both high fidelity and scalability can be achieved. RFnestTM enables realistic wireless network evaluation in a lab environment, and currently it is being used to test theoretical ideas and engineering designs for various research programs we execute. RFnestTM is an active project, with many ongoing enhancements including support for additional RF bands, increasing the number of real radios supported, and further validation and system integrity work. ACKNOWLEDGMENT The authors would like to thank Air Force for funding this work under contracts FA9550-10-C-0026, FA9550-11-C-0006, and FA8750-10-C-0142. We would also like to thank Captain Craig Baker, Dr. John Matyjas, and Dr. Robert Bonneau for their valuable input during this project. R EFERENCES [1] Daniel Aguayo et al, “Mit roofnet implementation,” Report, August 2003. [Online]. Available: http://pdos.csail.mit.edu/roofnet/design/ [2] “Orbit testbed project,” Report. [Online]. Available: http://www.orbitlab.org/ [3] K. Borries, G. Judd, D. Stancil, and P. Steenkiste, “FPGA-based channel simulator for a wireless network emulator,” in Proc. of 69th IEEE Conference on Vehicular Technology, Barcelona, Spain, April 2009. [4] J. Ahrenholz, C. Danilov, T. Henderson, and J. Kim, “Core: A real-time network emulator,” in Proc. of IEEE MILCOM, San Diego, CA, USA, Nov 2008, pp. 1–7. [5] Naval Research Laboratory, “Extendable mobile ad-hoc network emulator (EMANE).” [Online]. Available: http://cs.itd.nrl.navy.mil/work/emane/ [6] R. Olsen, “pktgen the linux packet generator,” in Proc. of the Linux Symposium, Ottawa, Ontario, Canada, July 2005, pp. 11–24. [7] J. Loeser and H. Haertig, “Low-latency hard real-time communication over switched ethernet,” in Proc. of ECRTS, Catania, Italy, July 2004, pp. 13–22.