Train a NEAT-based ANN

This service trains and validates a neural network after splitting the data set into training and validation.

Procedure

1. Scaling mechanism

Normalized data set from db-name/normalized collection is retrieved and split into training and validation.
For this step, user is asked to set 2 parameters in the payload (defaults provided):

1. training_perc (default 0.9): percentage of training data. Currently no automated procedure is available.
2. scale_mechanism (default "samedistribution"): algorithm that actually splits training and validation data set. Available algorithms:
   1. "random": shuffles permutations and splits them;
   2. "samedistribution": identifies permutations that contain min and max values of output nodes. Then it shuffles the permutations, splits them, makes sure that min and max values of output nodes are in the training set and runs a Kolmogorov-Smirnov Test to check if training and validation sets have the same distribution. It repeats the entire procedure until the two sets have the same distribution.

2. Training algorithm

Standard NEAT algorithm from encog machine learning framework trains the ANN.
For this step, user is asked to set 4 parameters in the payload (defaults provided):

1. training_error (default 0.1): required score for exiting the training process;
2. max_epochs (default 49999): number of maximum training epochs;
3. population (default 1000): starting number of genomes a.k.a individual random neural networks (the larger the population the better, but it slows down the training process and requires more memory);
4. connection_density (default 1): set the initial connection density for the initial random population of genomes

The training process stops if one of the following threshold is reached:

• the score (actual error) is less than training error;
• the actual epoch is bigger than max epochs;
• if the scores difference between last n-epochs is consecutively less than a threshold.

3. Model validation

The trained neural network is then validated using the validation data set. The following errors are evaluated comparing observed and simulated results:

• absDiff
• absDiffLog
• absVolumeError
• dsGrad
• err_sum
• fhf
• flf
• ioa
• kge
• modelDev
• nashSutcliffe
• nashSutcliffeLog
• nbias
• norm_rmse
• pbias
• pearsonCorrelation
• pwrmse
• r2
• rmse
• transformedRmse

DirectedAcyclicGraph plugin failed: Could not find plugin DirectedAcyclicGraph
(com.ecyrd.jspwiki.plugin.PluginException:Could not find plugin DirectedAcyclicGraph, java.lang.ClassNotFoundException:Class 'DirectedAcyclicGraph' not found in search path!)

{
    "metainfo": {},
    "parameter": [
        {
            "name": "annName",
            "value": "test_ann"
        },
        {
            "name": "annName_out",
            "value": "test_ann"
        },
        {
            "name": "training_error",
            "value": 0.01
        },
        {
            "name": "max_epochs",
            "value": 50000
        },
        {
            "name": "training_perc",
            "value": 0.8
        },
        {
            "name": "scale_mechanism",
            "value": "SameDistribution"
        },
        {
            "name": "population",
            "value": 3000
        },
        {
            "name": "connection_density",
            "value": 1
        }
    ]
}

{
    "metainfo": {
        "status": "Finished",
        "suid": "ace4ac6f-f5c1-11e7-b48e-1d3102711e59",
        "cloud_node": "10.1.27.11",
        "request_ip": "129.82.23.199",
        "service_url": "http://csip.engr.colostate.edu:8088/csip-ann/m/train/1.0",
        "csip.version": "$version: 2.2.4 3c276d66bde6 2017-12-15 od, built at 2018-01-09 15:35 by jenkins$",
        "tstamp": "2018-01-09 21:49:43",
        "cpu_time": 5540,
        "expiration_date": "2018-01-09 21:50:19"
    },
    "parameter": [
        {
            "name": "annName",
            "value": "test_ann"
        },
        {
            "name": "annName_out",
            "value": "test_ann"
        },
        {
            "name": "training_error",
            "value": 0.01
        },
        {
            "name": "max_epochs",
            "value": 50000
        },
        {
            "name": "training_perc",
            "value": 0.8
        },
        {
            "name": "scale_mechanism",
            "value": "SameDistribution"
        },
        {
            "name": "population",
            "value": 3000
        },
        {
            "name": "connection_density",
            "value": 1
        }
    ],
    "result": [
        {
            "name": "status",
            "value": "ok"
        },
        {
            "name": "exit_reason",
            "value": "threshold_error reached: 0.01"
        }
    ]
}