There are many real-time operating systems available. This thesis did not evaluate the various flavors of real-time operating systems to determine the most suitable for real-time data acquisition and visualization. The data acquisition system employed aboard SJB had already been developed using QNX. SOAR's original airborne survey system was designed to utilize an acquisition computer and multiple display computer stations connected via a network. QNX does not seem to be discussed or used within the computer science research community. The purpose of this discussion is to evaluate QNX as a reasonable choice for a real-time distributed computing environment.
According to Coulouris et al. [1994], six key characteristics describe the usefulness of a distributed computing system. If QNX shares all of these characteristics then it can be considered a useful distributed operating system and a reasonable choice for this project.
Since QNX accommodates each of these key distributed computing attributes, the last issue to consider is QNX's responsiveness to time-critical tasks. Using QNX to administer the distributed network aboard SJB is a valid choice.
The real-time visualization system aboard SJB is due to be updated and improved. The 'plumbing' of the system needs to be tight so that data leaks will not depreciate the benefits gained by using real-time displays. Knowing that the data routing system is optimally transferring data across the network allows real-time data graphics on the visualization monitors to be used as quality control indicators and not just graphics to entertain the aircraft crew.
Part of the motivation behind redesigning the packet routing system was to ensure maximum data throughput at the real-time visualization stations for real-time quality control. The current system aboard SJB utilizes three out of four visualization computers for monitoring and visualizing data acquired via serial input lines. Serial data (SD) throughput at these display stations, using the original packet routing system, is usually ~30% of the total amount of SD acquired. The new packet routing system was able to route 100% of the SD packets (under SJB's normal operating load) to each of the experimental visualization stations. Consequently, the demands of the SD visualizations would be better met with the new packet routing system.
Radar data (RD) comprises ~90% of all collected data. Currently, the original packet routing system can route all RD packets to the RD visualization station aboard SJB. This high throughput comes at the cost of low SD throughput. Thus, the new packet routing system better parallelizes RD and SD flows so that all real-time visualizations proceeds concurrently with high data throughput at each display station.
The new packet routing system generates a normal distribution of packet latencies measured at all experimental visualization stations with increasing RD packet rates. The central tendency values (mean, mode, median) are all relatively close in value. Packet latencies measured at the visualization stations when using the original packet routing system are not normally distributed. The central tendency measurements for these runs are much more varied, thus not indicative of a normal distribution. For the new packet routing system, variances increase exponentially with increasing RD packet rates (except for RD packet latencies when a higher thread priority is used for transferring RD packets). The variances calculated when using the original packet routing system are always high. These observation suggests that the new system is more stable and predictable and better able to handle the current and future increases in radar acquisition rates.
During high packet acquisition rates, packets are dropped at seemingly periodic intervals. Several additional analyses could be performed using the new packet routing system to further understand the patterns observed in packet loss. What causes the bursts of packet loss observed during high data acquisition rates? Are these bursts of data loss periodic? Is the periodicity due to the system periodically using the network for housekeeping tasks? Bursts of packet loss and periodic packet losses were identified during Internet transfers over the Mbone by Yajnik et al. [1996]. Their explanation is that periodic packet loss is caused by periodic updates by network routers. Within a QNX distributed network, nameloc, the process that keeps track of global network names, probes each QNX node once per time period. This time period is set at system boot up. The nameloc process executes at a higher priority than the packet routing processes, so it will preempt a packet routing process when it is scheduled to execute. This periodic probing may contribute to clusters of packet loss observed during performance testing of the new packet routing system.
Time series methods have been used to search for structure in Ethernet traffic [e.g., Yajnik et al., 1996; 1999; Moon et al., 1998] and may be useful for identifying patterns of packet loss during packet routing. These methods include autocorrelation and cross-correlation of time series.
Autocorrelation is a method for identifying time-related patterns within a data set. Normally, this method compares a time series with itself lagged one time interval, two time intervals, and so on, where lag is the amount of offset between the two series being compared. Time series exhibiting temporal patterns show high autocorrelation coefficients; random data sets (those with no patterns) have autocorrelation coefficients close to zero. This method can be employed to evaluate packet arrival patterns to determine if periodicity exists with these data sets. However, it is not possible to directly calculate the autocorrelation function of the time series, since packets arrive at unequal time intervals. Moon et al. [1998] calculate the autocorrelation of delay. In other words, the packet sequence number replaces time and the autocorrelation coefficient is calculated for packet latency by lagging the packet sequence numbers. Alternatively, an average latency could be calculated for a specific time interval (e.g., 50 milliseconds) and the autocorrelation function for average packet latency could be calculated using a time lag. Structure in the autocorrelation function may provide clues about what is causing the clusters of packet drops from an individual mailbox queue.
Cross-correlation is a method for comparing different time series. For example, cross-correlation can be used to compare patterns in serial and radar packet latencies. The delay method of Moon et al. [1998] will not work in cross-correlation of radar and serial packets because the packet rates are so different. An average latency per time interval is best for calculating the cross-correlation coefficients for these time series, but the time interval needs to be chosen carefully. The time interval needs to be large enough to accommodate the radar packet rate and small enough to avoid smoothing the time series. Relationships of packet loss to other system processes influencing the network could be investigated with cross-correlation methods. For example, high cross-correlation coefficients between bursts of packet loss and network probing by nameloc could indicate effects of system generated network traffic on packet throughput at the visualization nodes.
The ultimate goal of using these correlation methods is to predict when packet loss bursts will occur. Modeling packet loss may suggest how to avoid dropping so many packets, or how to avoid long bursts of packet loss. Hopefully, a model could optimize a set of parameters to minimize packet loss for a given delivery rate at the visualization stations. These parameters likely include message queue size, process and message priorities, scheduling algorithms, and network process parameters.
Once a optimal set of parameters are determined, additional performance might be achieved using the Communication-threads to smooth out the packet transfer rate to the visualization processes. Additionally, another message queue can be introduced at the display end of the packet routing system to assist with smoothing. Differential caching has been shown to successfully modulate digital video traffic to different clients [Sen et al., 1999]. A similar scheme can be implemented using multiple message queues between the acquisition computer and the display stations.
Finally, different hardware configurations may route packets more efficiently. Faster processors would increase the speed of the acquisition and display computers. Different network topologies and network speeds may transfer packets with fewer collisions. Nevertheless, the hardware limit is always reached, and this is where the software model, discussed previously, could be used to further improve real-time performance.
Packet loss and packet delay is currently being studied using data transmissions over the Internet. Such a large and varied environment makes it difficult to measure and quantify traffic patterns from a single localized source. With QNX it is easy to build a private real-time distributed network. This network may provide a better medium for understanding packet loss patterns during network data transfers and the largest factors inhibiting real-time data flow.
In this thesis, a real-time visualization display, running under the QNX real-time distributed operating system, was developed using Photon, QNX's real-time microGUI. The real-time display was employed during two months of geophysical surveying in Antarctica and operated concurrently with the data acquisition system without any problems.
Following the field season, the distributed process that routes data packets from the acquisition process to the real-time visualization processes, via an Ethernet, was redesigned to overcome low packet throughput noted at the RTQC visualization station. The new packet routing design uses three concurrent processes. The Spool process acquires data packets, the Disk-writer process writes the acquired packets to disk, and the Coordinator process routes those packets to independently running display processes via an Ethernet. Multiple concurrent Communication-threads are created by the Coordinator to send the data packets across the Ethernet, each thread communicating with a single display process. Two packet generator programs were designed to supply serial and radar packets, separately, to the packet routing system at specified rates. The performance of original and new packet routing systems were compared under increasing packet generation rates.
Based on the operation of the real-time visualization display during the two months of geophysical surveying in Antarctica, the following conclusion can be drawn: