Why LiveGraph?

In this section we describe what motivated us to create LiveGraph and which distinctive features separate it from other data visualisation tools. We discuss LiveGraph's strengths and weaknesses and provide a short overview of alternative tools.

The problem

In our research group at the Centre for Intelligent and Complex Systems at Monash University we create a lot of different computer simulations. The simulations usually produce a lot of data, which is normally analysed after the simulation has finished. The analysis is a lengthy process. During the first stages we often create a lot of graphs which tell us various things about the data. Often the most important conclusion we draw from a data plot is that we need to re-run the simulation with a different set of parameters. This can be frustrating, as sometimes the simulations have to run for many hours, or even several days. Fortunately, we often can recognise that simulation parameters must be changed already soon after starting the simulation, provided we are able to see our graphs instantly.

We realised that almost every time we created a new simulation, we also created a new data visualisation module in order to be able to watch how the simulation variables change during the simulation’s live-time. In the vast majority of cases these visualisation modules are some kind of graph plotters. Rather than computing advanced data analysis functions (these are more interesting at a latter stage, when the results of a completed simulation are analysed), the visualisation modules were plotting graphs of variables directly known to the simulation. (For instance, number of agents, relative proportions of different populations, best or average fitness and so on.) Researchers and students wasted many, many hours on these repetitive programming tasks instead of concentrating on the actual simulation models. We realised that we needed a tool which would save us doing this work. The tool had to be so easy to use that researchers and students coming from diverse research areas, having varying programming experiences and preferring different programming languages could make advantage of it.

There are many plotting tools around, some of which offer very advanced data analysis features. However, we could not find a tool suitable for our requirements - being able to plug into various applications and plot their output in real-time. So we finally decided to develop a tool ourselves. This is how LiveGraph was born.

The solution

To be usable across a wide range of applications the plotter had to be very loosely coupled with the software which uses it. We decided to use a file stream as a communication buffer. An application would write the data it wants to be plotted in a common comma-separated-values text file. This is a really very easy requirement to an application developer, especially, as scientific simulations often generate this data anyway. An independently running Java application would monitor the data file written by the simulation and plot graphs of data contained in the file. The plotter can update the graphs automatically at very short time intervals, producing a graph of the data literally live.

Since the amounts of the data are often very large, we developed a special caching technique. As a result, the plotter does not put any significant load on the processor(s) of the machine where it is running. Another challenge was that a data file produced by a simulation often contains a lot more data than you would like to see at any one time. So we taught our plotter to automatically recognise the data series contained in a file and their labels. We created a neat graphical user interface for LiveGraph which lets you select the data series you would like to see at any one time by a single click of the mouse.
In addition we developed a Java API which provides a convenient, easy-to-use class library that can be used by Java applications when they wish to write data to files in the format accepted by LiveGraph. For those who use other programming languages we provide a definition of this file format, such that application developers can create data log files using their own routines. We hope that the open source community will contribute to the project by providing data writer APIs for more programming languages.

There are various systems available for comprehensive data analysis in a post-simulation phase. GNUPlot is a very powerful plotter system featuring its own scripting language. Computer algebra and numerical computation packages such as Mathematica, Maple or Mathlab also often offer advanced plotting facilities along with powerful statistical and mathematical tools. All these systems are very powerful, but they usually require a lot of effort in data preparation and trade off a lot of advances features for a slow-to-use user interface.

LiveGraph, on contrary, is a professional solution for exploratory data analysis. It is particularly useful when exploring the parameter space of a simulation model or while visually searching data for patters or interesting features. It allows for live monitoring of real-time data and can easily be integrated into any software. The focus lies on a very fast and easy to use interface, rather than on powerful analysis tools. If this is what you need, you certainly should consider LiveGraph as an option.

In the user manual we describe the how to use the LiveGraph plotter application in detail; in the developer manual we explain how to incorporate the functionality of LiveGraph in your application.