LiveGraph data file definition

(File format version 1.1)

The LiveGraph API reads and stores data in text-based data files. The file format is based on the widely used comma-separated-values (CSV) format. LiveGraph’s file format was defined in such a way that a standard CSV file will be accepted and correctly parsed by the application (except that the very first data line might be interpreted as column headings — see below).
On this page you will find an overview over the file structure and a formal syntax specification for LiveGraph data files followed by an example.

General structure

The data file contains data in form of a table. Each data series (i.e. a sequence of data values describing the same metric for different entities) is contained in a separate table column. Each data set (i.e. a collection of data values for different metrics of the same entity) is contained in a separate row. The very first data set contains the column headings, i.e. the labels for the data series. In addition, there are special lines such as a separator definition line, comment lines and description lines. One example of how a data file could be structured is as follows:

##,## < Data separator definition line
@Data file structure example < File description line
Observation Seconds Dead mosquitos Hungry frogs < Data series labels line
1 1 0 100 < Data set 1
2 600 1000 50 < Data set 2
3 1500 5000 0 < Data set 3
... ... ... ...   ...

Formal definition

1. File is character and line based:

LiveGraph data files are text-files (i.e. not binary files). Files are read (written) on a line-by-line basis. Only after a complete line was read and parsed (or written) will the next line be considered.

2. Empty lines are ignored:

Any empty line or a line containing only white spaces is ignored without any further consequences.

3. Data values separator definition line:

The first non-empty line in a LiveGraph data file may contain an optional data values separator definition. A data values separator is a string which will separate data values in data lines.

A data values separator definition line must start and finish with the tag "##". The entire string between the opening "##" and the closing "##" will be the treated as the separator. For instance, the line "##<*>##" defines the data values separator "<*>".

A data values separator definition may not appear anywhere else than on the first non-empty line of the data file.

If the data values separator definition is omitted the default data values separator will be the string "," (comma).

4. Comment lines:

Any line where the first non-whitespace character is "#" (except for the data values separator definition line) is treated as a comment and is ignored. Note that no comments may be placed before the optional data values separator definition line.

5. File information and description lines:

Any line where the first non-whitespace character is "@" is treated as a file information or description line. A file information line does not have any effect on the interpretation of the data contained in the file; however, it may be used by a processing application to provide information to the end-user.

6. Data series labels line:

The first non-empty line in a data file which is not a data separator definition line or a comment line or a file information line is treated as data series labels line. This line defines the number and the labels of the data columns in the file. The line is split in tokens using the data values separator. The number of tokens defines the number of data columns in the file and the tokens define the labels of the columns. Note that the labels might be empty strings. For example:

    ##;##
    @Example  1
    ID;Age;;Height
    . . .

    @Example  2
    ,ID;Height,Age,weight,
    . . .

In example 1 the data separator is defined to be ";" (semicolon). Four data series are defined here: "ID", "Age", "" and "Height" (note that the third series label here is an empty string).

In example 2 no data separator is defined, so the default separator "," (comma) is used. Note that ";" is not a separator in this case. This gives five data series with the following labels: "", "ID;Height", "Age", "weight" and "". Note that the first and the last series labels are empty strings. They are separated from the following (preceding) labels by the data separator.

7. Data lines:

Any non-empty line after the series labels line which is not a comment line or a file information line is treated as data values line. Data lines contain the actual data. They are parsed into tokens in the same way as the data series labels line, which means that some tokens may be empty strings. The LiveGraph API allows any string to be used as a token. (Note that the plotter application converts each token to a double precision floating point value; if a token is not a character string representing a valid decimal number it will be converted to a not-a-number floating point value. This is interpreted as a gap in the data series by the plotter.)

All data values on the same line are considered to belong to the same data set. The data series of each value in a given data set is determined by comparing the position index of the corresponding data token in the line to the number of the series label token in the labels line.

Examples

Here are some examples of legal LiveGraph data files:

    ##|##
    @File  info for the user
    @More  info
    #Comment
    Observation|Seconds|Dead mosquitos|Hungry frogs
    1|1|0|100
    2|600|1000|50
    3|1500|5000|0
    #Another comment
    Observation,Seconds,Dead mosquitos,Hungry frogs
    1,1,0,100
    2,600,1000,50
    3,1500,5000,0