Provisioning Data to OMS Models (Data Input/Output)

Design motivations for the conventions include:

• Support for typical scientific data I/O such as tables and properties
• Low verbosity to enhance human readability
• Support for meta data
• Low barrier to data consumption by tools outside of OMS
• Permitting the definition of a simple API to programmatically read and write data.

There are two categories of data covered by the recommeneded conventions:

• Tables, containing tabular information
• Properties, referring to key/value property data

The Data I/O format is commonly defined for both tables and properties. The format is based on CSV structure that has been extended with meta tags. A file might contain any number of tables and properties.

Both types of information can be mixed in the same file and may occur multiple times. The definitions for tables and properties are similar, both supporting meta data.

• The data file complies fully to the CSV standard.
• The file name extension is csv, standing for "comma separated values". The file might be zipped and therefore would have the extension csz.
• A csv file might contain a table, a property section, multiples of each, or a mixture of both.
• A # symbol at the beginning of the line indicates a comment line.
• Empty lines are ignored.

Keywords

Note

• All of those keywords can be followed by optional meta data.
• Keywords are case insensitive (@T is equal to@t).

Meta Data

The property section example below shows section level meta data supporting the whole "Parameter set" such as data, or createdBy, as well as key value pair property meta data such as description or single value properties such as public. It might be good practice to quote meta data values in general to account for potential commas, however it is not required.

Meta Data Examples (@S and @T)

Properties

Property Example

@S, "Parameter"
CreatedAt, "Jan 02, 1980"
CreatedBy, Joe

# Single Properties
@P, coeff, 1.0
description, "A coefficient"
public

@P, start, "02-10-1977"
description, "start of simulation"

A property section starts with the @S keyword, followed by the name of the property section. It is followed by optional meta data. Meta data keys/values can be arbitrary, and may occur in any number. A single property starts with the property keyword @P, followed by the property name and the property value. Optional meta data may also follow a single property. The property section ends at the beginning of the next section or table or the end of the file.

Property Key/Value Substitution
Properties support internal key/value substitution. This feature helps organizing property sets more efficiently, An example is shown below. A directory property idir is defined and internally used by multiple files.

...
  # Input file folder (variable)
@P, idir, "ccreek"
 Description, "Data directory"

@P, ahumFileName, "${idir}/ahum.dat"
 Description, "Absolute Humidity Data"

@P, gwFileName, "${idir}/hgeo.par"
 Description, "Hydrogeology Data"
...

The expression ${<prop_key>} will be replaced with <prop_value>, if there is somewhere else within the same property set a property defined as @P, prop_key, prop_value.

Tables

Tables can be generated using any text editor. Spreadsheet tools usually do allow the export into a CSV file.

Table Example

# table example
@T, "Example DataSet"
CreatedAt, 5/11/06
CreatedBy, JackC
# Now, there is header information
@H, time,b,c
Type, Date,Real,Real
Format, yyyy-MM-dd,#0000.00,#000.0000
,2006-05-12,0000.00,001.1000
,2006-05-13,0001.00,002.1000
,2006-05-14,0002.00,003.1000
,2006-05-15,0003.00,004.1000
,2006-05-16,0004.00,005.1000
,2006-05-17,0005.00,006.1000
,2006-05-18,0006.00,007.1000

A Table consists of three main sections:

1. The table header, indicated by @T, followed by the name of the table. The next lines may have table level meta data, one meta data entry per line. Meta data is optional.
2. The table header is followed by the column header, indicated by the @H keyword. Next to this all the column names are listed. The next lines may contain column meta data, starting with the key, followed by the values for each column (Example above shows Type and Format for the columns).
3. Data rows start with a comma ',' as the first character; values are comma separated.

A minimal table with no optional meta data looks like this:

@T, example data table
@H a, b, c
, 1,2,3
, 4,5,6
... more data