Cloud Service Integration Platform

The metainfo JSON object contains variables that control model execution or provide feedback from the execution.

Figure 3.3. Variables

The result data (output JSON and output files) will be kept for a period of time after the model run successfully finished. If the model run failed or was cancelled no output will be stored. The time to keep results can be specified in the request using the "keep-results" entry within the "metadata" element.

In asynchronous execution mode, the returned metadata property "polling_interval" provides guidance for when the client should poll again for a status update. This value is model specific. Expect higher values for long running models and smaller values for models that run only for a short period. A client should not poll at intervals that are smaller than this value. Example: If a model takes on average 1 minute to finish, there is no need for the client to poll every 100 ms for a status update. The "polling_interval" in such case may be 5000 ms (5 seconds). There is no advantage for a client to use a shorter polling interval. The polling interval is provided by CSIP based on the average model runtime.

Notes:

A synchronous CSIP request returns when the service run finishes, thus blocking the calling client during service execution. CSIP services default to synchronous execution. Conversely asynchronous CSIP requests initiate a model/data service and return immediately to the client. Subsequent service calls must be made to query the service status to determine if execution is complete. Model requests service can be processed both ways, synchronously and asynchronously. The metainfo key "mode" controls the execution mode when submitting the initial request.

The example below demonstrates asynchronous service execution.

{
    "metainfo": {
            "mode": "async",
             ...
     }
   …
}

3.2.1. Synchronous Execution

The figure below shows the client calling sequence for synchronous execution. The server responds with a response JSON object which includes the client request input parameterization regardless if the model finished successfully or not. During execution the client thread is blocked. The client must account for issues like "call threading" to support a responsive user interface.

Figure 3.4. Synchronous Service Phases

After service execution completes the output data and optional report output data can be fetched at anytime before expiration from the session backend datastore.

3.2.2. Asynchronous Execution

While synchronous execution may be acceptable for short running CSIP services (execution time < 10 sec) it is unpractical for long running requests. Such services may execute for minutes, hours, or even days. The figure below shows the principal flow of calls for such asynchronous execution. If mode is set to "async", the initial service call (POST) submits the JSON request for execution and returns immediately.

Figure 3.5. Asynchronous Service Phases

The response metainfo element contains the suid, a unique key, that uniquely identifies the service request. The service status indicates its state of execution. In addition, the metainfo contains "first_poll" and "next_poll" properties. The client should wait "first_poll" milliseconds before first querying for a service result.

Example 3.13. Result for submitted model execution

{
  "metainfo": {
    "status": "Submitted",
    "suid": "182b8505-7534-11e1-b93f-1d1c56c85325",
    "tstamp": "2012-03-23T22:04:00+0000",
    "next_poll": "2000",
    "first_poll": "25000",
  },
  "parameter": [
..
  ]
}

A REST query is being executed using the suid. The call shown below is constructed using the '/csip/q' path and the appended suid.

Example 3.14. Async Query

GET /csip/q/182b8505-7534-11e1-b93f-1d1c56c85325 HTTP/1.1
Host: csip.engr.colostate.edu:8083
Accept: application/json

The query result shows that the model is still running, and has not finished. Subsequent queries for the service status as in the example should be done after "polling_interval" milliseconds.

Example 3.15. Query Result

{
  "metainfo": {
    "status": "Running",
    "suid": "182b8505-7534-11e1-b93f-1d1c56c85325",
    "tstamp": "2012-03-23T22:04:00+0000",
    "polling_interval": "2000",
    "first_poll": "25000",
  },
  "parameter": [
..
  ]
}

Upon completion the service status indicates Finished and the service output is provided in the result JSON section as fetched from CSIP. The suid must be retained by the client during service execution as it serves as a token to query the service status and its results.

CSIP services may require input files to execute. Not all service parameters can and should be mapped into JSON format. A service wrapping a legacy scientific model might consume input files directly with no modifications. In some cases, service input requirements may be so extensive their representation in JSON is impractical to implement.

CSIP services can easily consume client requests with files attached. Files are attached to CSIP HTTP/POST requests as multipart/form-data. See Example 3.16, “Input with multipart/form-data”.

POST /m/swat/1.0 HTTP/1.0
Host: localhost
Content-type: multipart/form-data, boundary=AaB03x
Content-Length: $requestlen

--AaB03x
content-disposition: form-data; name="param"; filename="req.json"  
Content-Type: application/octet-stream

$param
--AaB03x
content-disposition: form-data; name="file1"; filename="hru.dat"   
Content-Type: application/octed-stream
$file1
--AaB03x
content-disposition: form-data; name="file2"; filename="input.std" 
Content-Type: application/octed-stream

$binarydata
--AaB03x--

$ curl -X POST -H "Accept:application/json" \
        -F param=@/tmp/req.json \
        -F file1=@/tmp/hru.dat \
        -F file2=@/tmp/input.std \ 
        ...
          "http://localhost:8080/csip-swat/m/swat/1.0" 

More examples on how to attach files to the HTTP POST request using different clients and languages are shown in Section 4, “Invocation Examples”.

Legacy scientific models typically produce data output files. CSIP streamlines providing model output as key value pairs in the results JSON array to the client. Such output can be obtained from the model directly, or parsed from model output files. In some cases it is beneficial for the client to access the native model output files. For this use case, CSIP provides a query service to obtain the files. An example is shown below.

A model finishes and provides in the result JSON array a list of file pointers. A client can fetch the files as referenced by the provided URLs. The file size is provided in bytes.

{
  "metainfo": {
    "status": "Finished",
    "suid": "182b8505-7534-11e1-b93f-1d1c56c85325",
    "tstamp": "2012-03-23T22:04:00+0000",
    "polling_interval": "2000",
    "first_poll": "25000",
    "cputime": "28155",
    "expiration_date": "2012-03-23T22:09:28+0000"
  },
  "parameter": [
    {
      "name": "wepsrun",
      "value": "weps.run"
    },
    {
      "name": "climate",
      "value": "cli_gen.cli"
    },
    {
      "name": "wind",
      "value": "win_gen.win"
    },
    {
      "name": "management",
      "value": "S Wheat-Beet-SB CMZ 1.man"
    },
    {
      "name": "soils",
      "value": "Heldt_48_90_CL.ifc"
    }
  ],
  "result": [
    {
      "name": "sci",
      "value": "http://localhost:8080/csip-erosion/q/182b8505-7534-11e1-b93f-1d1c56c85325/sci_energy.out"
      "size": "21729"
    },
    {
      "name": "stir",
      "value": "http://localhost:8080/csip-erosion/q/182b8505-7534-11e1-b93f-1d1c56c85325/stir_energy.out"
      "size": "91638"
    },
  ]

A client parses the result JSON to obtain the URLs for downloadable files. It can then issue an HTTP GET request to fetch the files.

GET /csip-erosion/q/182b8505-7534-11e1-b93f-1d1c56c85325/stir_energy.out HTTP/1.1
Host: localhost:8080
Accept: */*

Example 3.18, “GET Request for result file download” shows the GET request to download the stir_energy.out file.

When client data or model service requests fail the service provides an error message. This message is embedded into the metainfo section of the JSON response.

The figure below shows a response containing an error message. The presence of an error key in metainfo indicates an execution problem.

{
    "metainfo": {   
        "status":"Failed",
        "error": "Insufficient input data"
    }
    "parameter":[
      ...
    ]
}

The method of submitting multiple sets of parameters for model execution is called an ensemble run. Each individual parameter set represents an individual request. CSIP splits up parameter sets and executes a single run for each parameter set. Service requests will execute in parallel using the configured number of worker threads.

An ensemble run completes all individual requests runs finish. The result JSON contains the results of the entire ensemble request. Like the array of parameters, results are returned as an array of result sets in request order. The first result set relates to the first parameter set.

{
   "metainfo": {   
       "parametersets":25
   }
   "parameter":[
    }
     "name:"p1",
     "value":"23",
      ...
    {,
    {
     "name":"p1",
     "value:"24",
      ...
    }
    ...
   ]
}

Ensemble runs like single runs can be executed in asynchronous and synchronous mode.

CSIP services can be automatically tested. Unit tests can easily be set up to test individual services or collections. Tests may run at the command line or within unit test frameworks such as JUnit ^[2] or TestNG^[3]. CSIP provides a separate library for testing (csip-test.jar). It can be directly executed or added to the JUnit test environment.

Service test setup follows simple naming conventions and is fully descriptive. Each service end point may be tested with one or more JSON requests files. Each JSON request may have file attachments as input for the service. The CSIP test environment automatically submits the JSON to the service endpoint and attaches all input. After successful service invocation the response is stored at the client. If the service produces result files they will be downloaded and stored on the client side. "Golden" response files can be used to automatically compare the retrieved response against known valid values.

All tests for one service endpoint reside in one folder. The folder contains the test descriptor, the JSON request(s), optional request input(s), and optional golden files.

The following directory structure is used for CSIP tests:

Service test description follows a simple naming convention. A service test must have the "service.properties" file. Each service test consists of a base name and a set of suffixes for different aspects of the test (*-req.json, *-res.json, etc.). The CSIP test framework will recognize the file's bases and generate the service outputs accordingly.

The file service.properties contains property settings controlling service test execution. The url entry is the only required entry which specifies the service endpoint to test.

# service endpoint, required
url=http://localhost:8080/csip-rzwqm/m/rzwqm/1.0

# number of concurrent tests, optional, defaults to '1'
concurrency=4

# timeout for a single test, optional, defaults to '3000'
timeout=5000

# test method: "keysonly" | "keyvalue" | "keyvalueorder" | "none", 
# defaults to 'none'
verify=keysonly

# should this service test be ignored? ,optional, defaults to 'false'  
ignore=false

# should result files be downloaded?, optional, defaults to 'true'
download_files=false

Example: Service Test

A service for temperature conversion might be tested using the following setup. A test folder was created using the structure as seen in Example 3.22, “Service Test Example”. There are 2 tests defined within "tempservice_V1_0". Both are using a reference file.

work
+--csip
   +-- tests
       +-- tempservice_V1_0
       |    service.properties 
       |    tempconv1-req.json 
       |    tempconv1-ref.json
       |    tempconv2-req.json  
       |    tempconv2-ref.json
       |
       +-- otherservice_V2_0/
            service.properties  
            ...

The file service.properties in tempservice_V1_0 contains the test properties as shown in Example 3.23, “Test settings for 'tempconv' in service.properties”. The key/value pairs of the response results should be compared against the reference file.

# service endpoint
url=http://localhost:8080/csip-example/m/tempconv/1.0
# verify the results
verify=keyvalue

The test can be executed with the command below. The command line argument '/work/csip/tests/tempservice_V1_0' is the path to the folder of the tempservice tests.

> java -jar csip-test.jar /work/csip/tests/tempservice_V1_0
=================
 Total:     2
 Failed:    0
 Succeeded: 2
 Skipped:   0
 Ignored:   0

>_

As a result, the two tests run successfully against the 'tempconv' endpoint. Upon completion, the output summary of the test command is provided.

Integration into existing unit testing frameworks is easy. The csip-test.jar file has to be added to the CLASSPATH used for testing. The Example 3.24, “JUnit based Service Testing” shows the use of service test example within a JUnit test. Similar to the command line interface, a service test is executed with the path to the test directory as an argument.

import csip.test.ServiceTest;
import junit.framework.Assert;
import org.junit.Test;

public class STest {
  @Test
  public void stest() throws Exception {
    ServiceTest.Results r = ServiceTest.run(
          "/work/csip/tests/tempservice_V1_0");
    Assert.assertTrue(r.getTotal() == r.getSucceeded());
  }
}

JUnit Assert methods can be used to validate the test result.

For simple service usage, CURL is a useful tool to submit an http request and receive a response. CURL is a simple command line tool for Linux, Windows, OSX, and other operating systems. A typical usage of CURL for CSIP is shown below.

The example fetches the model parameterization requirements for the temperature conversion as a JSON object using an HTTP GET (default curl operation)

curl -H "Accept: application/json" 
    "http://localhost:8080/csip-example/m/temp/1.0"

The following example executes the temperature service by sending an HTTP POST request by submitting the request data.

The file temp.json contains the service request data, the parameter data "temp" with a value of 25.

{  
   metainfo: {},
   parameter: [ 
     { name: "temp",
       value: 25 
     }
   ]
}

The command below performs a POST request against the CSIP endpoint.

curl -X POST \
     -H "content-type: application/json" \
     -H "accept: application/json" \
     "http://localhost:8080/csip-example/m/temp/1.0" -d @temp.json

The "-X" flag selects the POST request method to use when communicating with the CSIP server. The header flag "-H" specifies the content type as "application/json" and the JSON request data is obtained from the file "temp.json" (using the "-d" flag).

There are different methods for invoking CSIP services from Java. In the examples below we present four approaches to write CSIP REST clients, including an example using the CSIP library itself.

import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.io.BufferedReader;

...
URL url = new URL("http://localhost:8080/csip/m/temp/1.0");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
conn.setUseCaches(false);
conn.setDoInput(true);
conn.setDoOutput(true);
OutputStream os = conn.getOutputStream();
os.write(("{\n"
        + "    \"metainfo\": {\n"
        + "    },\n"
        + "    \"parameter\": [ \n"
        + "     {\n"
        + "      \"name\": \"temp\",\n"
        + "      \"value\": 25\n"
        + "     }\n"
        + "    ]\n"
        + "}").getBytes());
os.flush();
os.close();

if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
    throw new RuntimeException("Failed.");
}

InputStream is = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
StringBuilder response = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
     response.append(line);
     response.append('\n');
}
connection.disconnect();

System.out.println("Response: " + response.toString());

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Entity;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.UriBuilder;

...
String req = new String("{\n"
      + "    \"metainfo\": {\n"
      + "    },\n"
      + "    \"parameter\": [ \n"
      + "     {\n"
      + "      \"name\": \"temp\",\n"
      + "      \"value\": 25\n"
      + "     }\n"
      + "    ]\n"
      + "}");

String url = "http://localhost:8080/csip/m/temp/1.0";

String resp = ClientBuilder.newClient()
         .target(UriBuilder.fromUri(url).build())
         .service.request(MediaType.APPLICATION_JSON)
         .post(Entity.entity(req, MediaType.APPLICATION_JSON_TYPE), String.class);

System.out.println(resp);

import java.io.BufferedReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.HttpClientBuilder;

...
StringEntity e = new StringEntity(
              "{\n"
            + "    \"metainfo\": {\n"
            + "    },\n"
            + "    \"parameter\": [ \n"
            + "     {\n"
            + "      \"name\": \"temp\",\n"
            + "      \"value\": 25\n"
            + "     }\n"
            + "    ]\n"
            + "}", ContentType.APPLICATION_JSON);

String url = "http://localhost:8080/csip/m/temp/1.0";

// create the client and post the request:
HttpPost post = new HttpPost(url);
post.setEntity(e);
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = client.execute(post);
 
BufferedReader rd = new BufferedReader(
       new InputStreamReader(response.getEntity().getContent()));
 
StringBuilder result = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
    result.append(line);
    result.append('\n');
}
System.out.println("Response: " + result.toString());

import org.codehaus.jettison.json.JSONObject;
import csip.Client;
...

String req = "{\n"
           + "    \"metainfo\": {\n"
           + "    },\n"
           + "    \"parameter\": [ \n"
           + "     {\n"
           + "      \"name\": \"temp\",\n"
           + "      \"value\": 25\n"
           + "     }\n"
           + "    ]\n"
           + "}";
String url = "http://localhost:8080/csip/m/temp/1.0";
JSONObject res = new Client().doPOST(url, new JSONObject(req));

System.out.println(res.toString(4));

The Microsoft .NET Framework provides a layered, extensible, and managed implementation of Internet services that can be quickly and easily integrated into your applications. Both examples, C# and VB.NET use the same .NET Framework classes for performing a CSIP service call.

The following example describes the steps used to execute send input data to a CSIP service and fetch its response.

The .NET Framework provides protocol-specific classes derived from WebRequest and WebResponse for "http:" URIs. Both classes are the core if this example.

The following example shows the CSIP client in C# for the .NET platform.

using System;
using System.Net;
using System.IO;
using System.Text;

namespace httpclient
{
 class MainClass
 {
  public static void Main(string[] args)
  {
   string json = "{ " +
    "  metainfo: {}," +
    "  parameter: [ " +
    "    { name: \"temp\"," +
    "      value: 25 " +
    "    } " +
    "  ] " +
    "}";

   // Create a request using a URL that can receive a post. 
   WebRequest request = WebRequest.Create(            
              "http://localhost:8080/csip/m/temp/1.0");
   request.Method = "POST";                           

   // Create POST data and convert it to a byte array.
   byte[] byteArray = Encoding.UTF8.GetBytes(json);
   // Set the ContentType property of the WebRequest.
   request.ContentType = "application/json";
   request.ContentLength = byteArray.Length;          
   // Get the request stream.
   Stream dataStream = request.GetRequestStream();
   dataStream.Write(byteArray, 0, byteArray.Length);  
   dataStream.Close();     

   // Get the response.
   WebResponse response = request.GetResponse();      

   StreamReader reader = new StreamReader(response.GetResponseStream());
   string responseFromServer = reader.ReadToEnd();    

   Console.WriteLine(((HttpWebResponse)response).StatusDescription);
   Console.WriteLine(responseFromServer);             

   reader.Close();
   response.Close();
  }
 }
}

This example explain the use of VB.NET for posting a a CSIP model request to a server using the .NET framework classes WebRequest and WebResponse as is was shown in the previous section for C#. Because the .NET framework classes being used are the same, the examples only differ with respect to language constructs.

Imports System.Net
Imports System.IO

Public Class Application
 Public Shared Sub Main()
  
  Dim URL As String = 
    "http://localhost:8080/csip/m/temp/1.0"
  
  Dim req As String = "{ " +
    "  metainfo: {}," +     "  parameter: [ " +
    "    { name: ""temp""," +
    "      value: 25 " +
    "    } " +
    "  ] " +
    "}"
  Dim res As String = ""
  
  Try
    Dim client As HttpWebRequest = Webrequest.Create(URL)
    client.Method =  "POST"
    client.ContentType = "application/json"
    Dim encoding As New Text.ASCIIEncoding() 
    Dim postByteArray() As Byte = encoding.GetBytes(req)
    
    client.ContentLength = postByteArray.Length
    Dim postStream As Stream = client.GetRequestStream()
    postStream.Write(postByteArray, 0, postByteArray.Length)
    postStream.Close()

    Dim clentResponse As HttpWebResponse = client.GetResponse()
    
    If clentResponse.StatusCode = HttpStatusCode.OK Then
      Dim responseStream As StreamReader = _
        New StreamReader(clentResponse.GetResponseStream())
      res = responseStream.ReadToEnd()
    End If
    clentResponse.Close()
     
  Catch e As Exception
    res = "CSIP error occurred: " & e.Message
  End Try
     
  System.Console.WriteLine(res)
 End Sub
End Class

The VB.NET example employs exception handling for error management in tis example.

Python is an object-oriented, interactive, general purpose programming language that has characteristics such as high level dynamic data types, dynamic typing and a very clear syntax ^[6]. The httplib2 python library^[7] is a comprehensive HTTP client library that handles caching, keep-alive, compression, redirects, and many kinds of authentication. The httplib2.py library supports many features left out of other Python HTTP libraries. This library is installed using the pip command.

#pip install httplib2

CSIP model execution using the Python/httplib2 is shown below. The http2 library and the json library must be imported.

import httplib2, json

req = json.dumps(
 { 'metainfo': {}, 
   'parameter' : [ 
   { 
    'name' : 'temp', 
    'value': 25
   }]
 })

headers = {"content-type": "application/json", 
           "accept": "application/json"}
url = "http://localhost:8080/csip/m/temp/1.0"

http = httplib2.Http()
res, content = http.request(url, 'POST', 
                            headers=headers, body=req.encode())

print res.status, res.reason
print content

The json.dumps() method serializes the argument to a JSON formatted string. Note that we have to use the encode() function from json to encode the string before using it as the POST body.

C is a general-purpose, imperative computer programming language, while C++ adds object-orientation to C. The most complete library for C/C++ programs is libcurl^[8], a free and easy-to-use client-side URL transfer library. Curl builds and works identically on all operating systems. libcurl is the most used C-based highly portable transfer library in the world.

libcurl is often pre-installed on linux and unix based systems. However, package managers such as apt, yum, rpm, etc. can be used to fetch and install the binary packages as needed.

libcurl introduces the "easy" interface. All operations in the easy interface are prefixed with 'curl_easy'. The easy interface provides for single transfers with a synchronous and blocking function call. The example CSIP client program below can be compiled using gcc and must be linked against the libcurl library by using the -lcurl flag.

// install libcurl
// compile: gcc Call.c -lcurl

#include <string.h>
#include <curl/curl.h>

int main() {
  CURL *curl;
  CURLcode res;
  struct curl_slist *headers = NULL;

  curl = curl_easy_init();
  if (curl) {
    const char *req = 
        "{ "
        "  metainfo: {},"
        "  parameter: [ "
        "    { name: \"temp\","
        "      value: 25 "
        "    } "
        "  ] "
        "}";
    
    const char *url = "http://localhost:8080/csip/m/temp/1.0";

    headers = curl_slist_append(headers, "content-type:application/json");
    headers = curl_slist_append(headers, "accept:application/json");

    curl_easy_setopt(curl, CURLOPT_URL, url);
    curl_easy_setopt(curl, CURLOPT_POST, 1L);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, req);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, strlen(req));
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);

    res = curl_easy_perform(curl);
    if (CURLE_OK != res) {
      printf("Error: %s\n", strerror(res));
      return 1;
    }
    curl_easy_cleanup(curl);
    curl_slist_free_all(headers);
  }
  return 0;
}

The program initializes curl, creates header information, sets various options, and performs the POST request. Finally, all resources are released.

The code above will also compile with a C++ compiler (e.g. g++), which uses the same library.

JavaScript is an object-oriented programming language commonly used to create interactive effects within web browsers. The presented JavaScript CSIP example uses jQuery, a fast, small, and feature-rich JavaScript library^[9]. JQuery simplifies HTML document traversal and manipulation, event handling, animation, and Ajax with an easy-to-use API that works across a multitude of browsers.

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
...

var reqData = 
 '{' +
  '"metainfo": {},'+
  '"parameter": [' +
    '{' +
    '  "name": "temp",' +
    '  "value": 25' +
    '}' +
  ']' +
 '}';

// ajax call to the service
$.ajax({
  type: "POST",
  headers:{'accept': 'application/json'},
  contentType: "application/json",
  url: "http://localhost:8080/csip/m/temp/1.0",
  data: reqData,
  success: function (data, textStatus, xhr) { 
      success(data);
  },
  error: function(jqXHR, textStatus, errorThrown){
      alert("Error! " + errorThrown);
  }
});
...

Simple AJAX calls in JavaScript can be performed to invoke CSIP services as shown in Example 3.35, “CSIP service call in JavaScript”. This fragment constitutes population of a request object (reqData) and the client service invocation.

The success callback function is passed the returned data, which will be a JSON string depending on the MIME type of the response. It is also passed in the text status of the response.

Tomcat application server security defaults permit Javascript REST clients to only invoke services on the originating web server that provides the Javascript content to the user. Ajax requests are subject to the same origin policy; the request can not successfully retrieve data from a different domain, subdomain, port, or protocol. The CORS (Cross-origin resource sharing) setting in tomcat can be enabled to overcome this limitation.

Groovy is a dynamic language with features similar to those of Python, Ruby, Perl, and Smalltalk. It can be used as a scripting language for the Java Platform, is dynamically compiled to Java Virtual Machine (JVM) bytecode, and interoperates with other Java code and libraries.

Groovy's HTTPBuilder ^[10]is the easiest way to manipulate HTTP-based resources in a Java VM. HTTPBuilder is a wrapper for Apache's HttpClient with some Groovy syntactical extensions. The request/response model is also inspired by Prototype.js.

HTTPBuilder is the main API class used to make requests and parse responses as shown below.

@Grab(group='org.codehaus.groovy.modules.http-builder', 
      module='http-builder', version='0.7' )

import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.POST
  
def http = new HTTPBuilder('http://localhost:8080')
http.request(POST) {
    uri.path = '/csip/m/temp/1.0'
    requestContentType = JSON
    body = [metainfo:[dummy:0], parameter:[[name: 'temp', value: 25]]]

    response.success = { resp, json ->
        println "Success! ${resp.status}"
        println "Response: ${json}"
    }

    response.failure = { resp ->
        println "Request failed with status ${resp.status}"
    }
}

The @Grab command manages installation of the http-builder library using the Grape tool. HttpBuilder is not part of the Groovy distribution. The request content type is set to JSON, while the JSON content is provided in Groovy syntax. The response handler is a closure which prints out the response body on successful execution.

Go is an open source programming language developed by Google and many other contributors from the open source community ^[11]. It is a fast, statically typed, compiled language with support for garbage collection and run-time reflection.

The Go package "GoRequest" ^[12]provides a simplified HTTP client library for HTTP client implementations. In the example, this library is used for exercising the CSIP service call for temperature conversion. The library is installed using the go get command. The gorequest library simplifies the submission of JSON request data compared to standard Go libraries. JSON data can be provided directly.

package main

// install go get github.com/parnurzeal/gorequest
// go run Call.go

import (
  "fmt"
  "github.com/parnurzeal/gorequest"
)

func main() {
  request := gorequest.New()
  resp, body, errs := request.Post(
        "http://localhost:8080/csip/m/temp/1.0").
  Set("accept","application/json").
  Set("content-type","application/json").
  Send(`{
    "metainfo": {},
    "parameter": [ 
       {
         "name": "temp",
         "value": 25
       }
      ]
     }`).
   End()
   fmt.Println("response Status:", resp.Status)
   fmt.Println("result:", body)
   fmt.Println("errors:", errs)  
}

The Go client source code shown above creates a CSIP POST request and sends the JSON parameter request to the simpleservice endpoint. The response status, body, and errors are printed out.

Ruby is a dynamic, reflective, object-oriented, general-purpose programming language. Most Ruby distributions contain 'Net::HTTP' ^[13]classes which are part of the Ruby Standard Library. Net::HTTP provides a rich library which can be used to build HTTP user-agents. Net::HTTP is designed to work closely with URI. URI::HTTP#host, URI::HTTP#port and URI::HTTP#request_uri are used within Net::HTTP.

require 'net/http'
require 'json'

def do_csip
    uri  = URI('http://localhost:8080/csip/m/temp/1.0')
    http = Net::HTTP.new(uri. host)
    req  = Net::HTTP::Post.new(uri.path, initheader = 
                   {'content-type' =>'application/json',
                    'accept' =>'application/json'})
    req.body = { 
                metainfo: {},  
                parameter: [
                    { name: "temp", 
                      value: 25 } 
                ] 
               }.to_json
    res = http.request(req)
    puts "response #{res.body}"
rescue => e
    puts "failed #{e}"
end

do_csip

A POST request can be made using the Net::HTTP::Post class. The example above creates a JSON encoded request body containing the temperature conversion input. The function do_csip performs the request and prints out the response.

It is recommended for the client to follow some rules when executing CSIP Services:

A CSIP client call provides input to the service using JSON content. The 'parameter' section of the JSON file contains all literal input. The custom CSIP service, however, does not have to process, parse, and extract the data from JSON directly. This is accomplished from within CSIP infrastructure (ModelDataService). A service should always use the methods as shown in Example 4.5, “Parameter Input Methods” to obtain parameter and metadata. ModelDataService methods handle the correct conversion of data types and validation of values as applicable.

boolean hasParam(String name)                       
int getParamCount() 
Set<String> getParamNames() 

String getStringParam(String name)                   
String getStringParam(String name, String def)   
int getIntParam(String name)   
int getIntParam(String name, int def)   
double getDoubleParam(String name)   
double getDoubleParam(String name, double def)   
boolean getBooleanParam(String name)   
boolean getBooleanParam(String name, boolean def)  
long getLongParam(String name)   
long getLongParam(String name, long def)   
JSONOjetct getJSONParam(String name)   
JSONOjetct getJSONParam(String name, JSONOjetct def)   

String getParamUnit(String name)                     
String getParamDescr(String name)   
JSONObject getParamGeometry(String name)

All parameter input should be obtained in the preProcess() or process() method of a service call. Usually, the request parameter data is placed into service fields or passed to the service's internal data structures.

A detailed listing of parameter methods can be found in ???.

Example: Accessing Model Parameter

A ModelDataService receives a JSON request that contains a parameter called "precip". This parameter has a value "14.4" and unit "inches".

{
    "parameter" : [
      ...
      {
         "name" : "precip",
         "value": 14.9
         "unit" : "inch"
      },
      ...
    ]
}

The above "precip" request parameter of a request can be read in by the service using the getDoubleParam() and getParamUnit() methods.

@Override
protected void preProcess() throws Exception {  
    ...
    double precip = getDoubleParam("precip"));   
    String precip_unit = getParamUnit("precip"));   
    ...
}

If a JSON parameter cannot be converted into the desired type, a ServiceException is thrown.

Service metainfo is a part of every service request. This is not direct input to the model, it rather provides context data to the service. As an example, a flag controlling the verbosity of service output, or requesting the use of a certain model backend would be part of the metainfo section and not parameter.

The Example 4.6, “Metainfo methods” shows all methods available in CSIP to fetch metainfo content. It is not needed to parse the JSON request directly.

Set<String> getMetainfoNames()           
int getMetainfoCount()
boolean hasMetainfo(String name)

String getStringMetainfo(String name)    
int getIntMetainfo(String name) 
double getDoubleMetainfo(String name) 
boolean getBooleanMetainfo(String name) 

Metainfo is typically processed in the preProcess() or the process() method of a service call to control service behavior. A detailed listing of all metainfo methods can be found in ???.

Example: Accessing MetaInfo

Within the metainfo section of a service request there can be any number of custom key/value entries. As an example, the metainfo section contains a verbosity flag, that controls the level of output for the service.

{ 
    "metainfo": {
        "verbose": true,
        ...
    }
    ...
}

Metainfo parameters are obtained from the JSON request object using the get???MetaInfo() methods as shown below.

if (hasMetainfo("verbose") {
    boolean verbose = getBooleanMetainfo("verbose")
}  

If a metainfo element does not exist or has the wrong type, a ServiceException is thrown.

The use of custom metainfo content should be carefully considered. Metainfo is not input data to parameterize the service- rather it provides context and configuration data to direct HOW the service should run. Metainfo should be considered optional, as service defaults should enable the service to run when metainfo is absent.

Client provided metainfo is read-only in CSIP. The CSIP infrastructure appends useful metainfo to describe status information, execution time, compute node, etc. which is sent back to the client in the response JSON .

CSIP Model service requests support file attachments as service input. Files are attached as FormDataMultiPart in the POST request. The CSIP infrastructure manages files by extracting them from the request, storing them on the workspace of the service session, and publishing URIs to make them available. Example 4.7, “File input methods” lists the service methods supporting file access.

Set<File> getFileInputs()        
int getFileInputsCount()         
File getFileInput(String name)   
boolean hasFileInput(String name)

All files that are part of the request are copied unchanged into the session workspace. If the incoming file is an archive (*.zip, *.gz, *.tar, *tgz, ...) CSIP automatically uncompresses and extracts it into the workspace. It is good practice to compress large files to reduce network I/O requirements.

Example: File attachment

Files are included in the CSIP client request as HTTP mulitpart attachments. The CSIP service infrastructure automatically handles recognition and extraction of input files into the service session workspace. Input files are unpacked from the request and stored in their original form. Using the process() methods of a service the files can referenced by name.

The curl call below invokes a CSIP service and attaches a file. The file "scott.kmz" is attached in this example.

curl -X POST -H "Accept:application/json" \
  "http://localhost/csip-lamps/m/lamps/1.0" -F file1=@c:/scott.kmz

The submitted file can now be referenced in the service implementation.

protected void preProcess() {
    ...
    File gem = getFileInput("scott.kmz")
    ...
}

If the getFileInput() method returns a reference to the file, it is guaranteed to exist.

Report generation can optionally be incorporated in post processing. It allows sever-side preparation of additional service output that can optionally be fetched by the client. The report typically contains secondary output that has lower significance than the primary service result output. Reports are generated in the report() method of a service.

The list of all putReport() methods is shown in Example 4.10, “Report generation API”.

    void putReport(String name, String val, String unit, String descr) 
    void putReport(String name, String val, String unit) 
    void putReport(String name, String val) 
    void putReport(String name, int val, String unit, String descr) 
    void putReport(String name, int val, String unit) 
    void putReport(String name, int val) 
    void putReport(String name, double val, String unit, String descr) 
    void putReport(String name, double val, String unit) 
    void putReport(String name, double val) 
    void putReport(String name, boolean val, String unit, String descr) 
    void putReport(String name, boolean val, String unit) 
    void putReport(String name, boolean val) 
    void putReport(String name, JSONObject val, String unit, String descr) 
    void putReport(String name, JSONObject val, String unit) 
    void putReport(String name, JSONObject val) 
    void putReport(File file) 

    void putReport(File file, String descr) 
    void putReport(File... file)  

The putReport() helper methods have semantics similar to the putResult() methods.

Example: Report Generation

Report generation must be implemented within the report() method. Report entries are added using the putReport() family of helper methods. The example shows a service code fragment creating report entries.

...
double runoff;

public void report() {
   putReport("peakrunoff", 2.34, "cfs");
   putReport(new File("summary-report.pdf"), "summary");
}
...

CSIP services can bundle executables for different architectures to enable flexible deployment to different operating systems. The variable ${arch} can be used in the string referencing the resource to resolve the actual architecture at service runtime. This is shown below.

  @Resource(file="/bin/${arch}/tr20.exe", type=EXECUTABLE, id="tr20")

When the executable is requested using the getResourceExe() method, the "arch" variable will be replaced with the host machine's actual architecture fingerprint:

Example: To deploy the same service to various architectures, a tr20.exe native binary must exist as /bin/win-amd64/tr20.exe and /bin/lin-amd64/tr20.exe, respectively. The executable must be compiled for both architectures. Only one resource definition is sufficient to reference the executable. At service runtime, the correct executable is selected based on the service hosting architecture.

Automated execution of a model within a service simplifies the integration of existing executables into services. Annotation-only based services should be sufficient for most external models requiring a fixed set of inputs.

The executable is annotated with the ID "auto". An example of a ModelDataService is shown in Example 4.18, “SWAT External Executable Service”. Here, no custom service code is needed.

An external model data service must subclass ModelDataService. All service annotations should be added to this subclass. In addition to @Resource, the @Name, @Description, @Path, and @Polling annotations should be used. @Path is required.

import csip.ModelDataService;
import csip.annotations.*;
import static csip.annotations.ResourceType.*;
import javax.ws.rs.Path;
import oms3.annotations.*;

/**
 * SWAT Service. Plain execution of the original executable.
 *
 * @author od
 */
@Name("SWAT")
@Description("Soil Water Assessment Tool model service. (SWAT2012 Rev. 627)")
@Path("m/swat/1.0")
@Polling(first = 5000, next = 2000)
@Resources({
  @Resource(file = "/bin/lin-amd64/swat2012_627.exe", type = EXECUTABLE, id = "auto"),
  @Resource(file = "output.* *.out chan.deg fin.fin " + 
                   "watout.dat input.std stdout.txt stderr.txt", type = OUTPUT)        
})
public class V1_0 extends ModelDataService {
  // done.
}

The same auto-execution approach can be applied to Java and OMS models.

Model services may require significant time to execute, based on the underlying model and logic. It is not uncommon that services run for seconds, minutes or hours per service call. Asynchronous service invocations for longer running services allow clients to stay responsive during service execution. Clients must periodically check the status of the service completion as described in Section 3.2.2, “Asynchronous Execution”.

To support a reasonable polling frequency for clients, the service should describe when and how often a client should poll. The @Polling annotation, or the methods listed in the examples provide two alternatives to indicate:

A custom model service can either overwrite the methods in Example 4.19, “Polling methods” , or use the annotations in Example 4.20, “Polling Annotations” to specify the desired polling frequency. An accurate polling indication by the model service and its respectful handling at the client side avoids flooding the network with unnecessary requests. For example: If a model runs for 10 minutes, it is not necessary to query the execution status every second after asynchronous submission.

    long getNextPoll() 
    long getFirstPoll()

Polling methods are preferred if the service execution time varies based on the size of the input data or other factors. In this case the service can provide a dynamic estimate for a polling frequency based on the size or nature of the inputs.

public @interface Polling {
    long first() default -1;
    long next() default -1;
}

Polling annotation are preferred if service execution time is mostly constant, it does not depend on the size of the inputs.

The long return values for first/next polling should be specified in milliseconds.

@Polling(first = 180000, next = 2000)
public class MService1 extends ModelDataService {
...
}

public class MService1 extends ModelDataService {

  //....

  protected long getFirstPoll() {
     return no_years_in_data * 100;
  }

  protected long getNextPoll() {
     return 1000;
  }
}

Logging is enabled and configured by default for each CSIP session. The session ID (suid) is used to identify the logging output. This support separation and filtering of logging by session. The CSIP configuration property "csip.logging.enabled" is a boolean that can modified at any time to disable or enable logging.

The ModelDataService class provides a protected logger called "LOG" that can be accessed in a custom service implementation Example 4.25, “Logging Example”. The LOG field is final, thus it cannot be altered. The default log level is set to "INFO", however it can be set using the configuration property "csip.logging.level". Table 5.1, “CSIP System Properties” describes all logging related properties in detail.

public void process() throws Exception {
    //...
    if (LOG.isLoggable(Level.INFO) {
        LOG.info("connecting do database ...");
    }
    // ..

CSIP uses standard Java logging. Predefined log levels include: OFF, ALL, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST.

Logging information can be captured using a separate server as it is described in Section 2.5, “Multi-Server Deployment”. The session user interface provides a link to the log record for each individual session. This way logging information can be obtained from any client for debugging and tracing purposes without having direct access to the application server. Logging for a CSIP deployment is centrally configured. Each CSIP deployment can specify its own logging infrastructure configuration.

The table below lists CSIP properties that can be customized for specific CSIP deployments .

It is recommended to perform configuration changes right after context loading and before any the first CSIP service request is made. Changes of the session backend are only supported at initialization to avoid inconsistencies for session management.

Property changes should be described in a JSON file containing proper key/value pairs, according to Example 5.1, “Example CSIP configuration file (service-conf.json)”. An HTTP/POST request is used to load the JSON config file and apply the settings to the CSIP server.

{     
    "csip.session.backend" : "redis",
    "csip.session.redis.server" : "192.168.1.100",
    "csip.session.ttl" : 600
}

All property values can be passed in as native types or as a string. The entry csip.session.ttl can be set with 600 or "600".

The curl command below performs the configuration update by issuing the HTTP POST request with the file service-conf.json. The service path to be used is always "../c/conf".

$ curl -X POST -d @service-conf.json http://localhost:8080/csip-test/c/conf

The configuration service returns the current configuration or an error message.

On a single server deployment all CSIP components run on the same server. Sessions are managed using the Hazelcast backend and remain available for 10 minutes after compleition. Session archival and logging are not enabled. This is the default CSIP deployment configuration.

Figure 5.1. Singe server deployment

{     
    "csip.session.backend" : "hazelcast",
    "csip.session.ttl" : "600",
    "csip.logging.enabled" : false,
    "csip.archive.enabled" : false,
    "csip.hazelcast.server" : true
}

This CSIP instance uses an internal hazelcast server for session management. There is no need to start and manage it manually.

Server Configuration

In this single server configuration all sessions are managed using the redis backend and will stay available for 5 minutes after finish. Session are archived to the redis server after 5 minutes. Logging is disabled.

Figure 5.2. Singe server deployment

The configuration is shown below.

{     
    "csip.session.backend" : "redis",
    "csip.session.redis.server" : "10.0.1.100",
    "csip.session.ttl" : "300",
    "csip.archive.enabled" : true,
    "csip.archive.server" : "10.0.1.100"
    "csip.archive.ttl" : 604800
}

Archived session data is retained for a week (604800 sec) before deletion.

Server Configuration

In this single server configuration all sessions are managed using the redis backend and will stay available for 5 minutes. Session data is deleted afterwards. Logging is enabled.

Figure 5.3. Singe-Server Deployment for Development

{     
    "csip.session.backend" : "redis",
    "csip.session.redis.server" : "10.0.1.100",
    "csip.session.ttl" : "600"
    "csip.archive.enabled" : false,
    "csip.logging.enabled" : true
    "csip.logging.server" : 10.0.1.100,
    "csip.logging.level" : ALL,
}

This single server configuration is suitable for local development of CSIP services since archives are not required. The log level is set to ALL to capture all logging output. If development occurs on the local machine, the ~.server properties can be set to localhost.

Server Configuration

In this single server configuration all sessions are managed using the Redis backend and stay available for 10 minutes after completion. Sessions are archived to the local Redis server. Logging is enabled.

Figure 5.4. Single-Server Deployment with Archival

The CSIP configuration is shown below.

{     
    "csip.session.backend" : "redis",
    "csip.session.redis.server" : "10.0.1.100",
    "csip.session.ttl" : "600"
    "csip.archive.enabled" : true,
    "csip.archive.server" : 10.0.1.100,
    "csip.archive.ttl" : -1,
    "csip.logging.enabled" : true,
    "csip.logging.server" : 10.0.1.100,
    "csip.logging.level" : WARNING,
}

This single server configuration is most suitable for local deployment of CSIP services. The log level is set to WARNING to capture only relevant logging output. Note that the same Redis instance is used for session management, archival, and logging. This simplifies data management but may represent a bottleneck when service traffic is high. Redis memory resources should be carefully monitored since archive entries will not expire.

Server Configuration

Multi-server deployment addresses production needs such as scalability and fail over. Infrastructure components are partitioned so that they reside on separate machines to eliminate resource contention.

Figure 5.5. Multi-Server Deployment: separate Logging and Archival

Typically multiple CSIP servers are used to service modeling and data service requests. The configuration below uses one Tomcat and two Redis servers for session management, archival, and logging.

{     
    "csip.session.backend" : "redis",
    "csip.session.redis.server" : "10.0.1.101",
    "csip.session.ttl" : "600"
    "csip.archive.enabled" : true,
    "csip.archive.server" : 10.0.1.101,
    "csip.archive.ttl" : -1,
    "csip.logging.enabled" : true,
    "csip.logging.server" : 10.0.1.102,
    "csip.logging.level" : WARNING,
}

The archive entries never expire, only warning messages are logged.

Server Configuration

A typical production level deployment accounts for scalability, fail over, and redundancy of resources. CSIP can be configured to cover all of those aspects.

Figure 5.6. Multi Server Deployment in Production

[TBD]

The CSIP session UI for a given context can be accessed using the following URL:

GET http://<host>/<context>/c/session

The session UI is shown in Figure 5.7, “Session User Interface”. The table shows all relevant information for a session.

Figure 5.7. Session User Interface

The first column lists the simulation id (suid) for a given service session with request and response JSON links. The response JSON is available once the session is completes. The status, client IP, and worker node IP is provided followed by the time of the request, the time when the service session (and its output data) will expire, the execution time of the service, and the service end point. Finally, request attachment filenames are listed and the session log is provided.

The rows in this view are colored according to the status. Color changes with the status.

For the request, response, requesting IP address, and log, hyperlinks provide additional information to the user when clicked (e.g. IP geolocation ^[15], or log listing).

Query parameters can be appended to the session URL to control the appearance of the table. As a default the most recent session is shown on top of this table.

Note that the session view is not updated on a periodic basis. Users van harness browser extensions to refresh the content so often (e.g. every 5 seconds).

The Logging user interface allows access to log records on a "per-session" basis. Sessions are listed with their session id and a link to the log record Figure 5.8, “Logging User Interface”. Note that those sessions are not ordered. A user search for sessions by id can fetch the associated log entry.

Figure 5.8. Logging User Interface

The logging UI link is identical to the session UI link. Logging entries do not expire with the session. They remain in the logging server until they are expired based on the value in "csip.logging.ttl", which defaults to 24 hours. Alternatively, they can be deleted explicitly.

The Archive UI provides a complete listing of all archived sessions Figure 5.9, “Archive User Interface”. It shows a table, where each row represents a session archive. The session id (suid), the completion status at the end of service execution, the originating request IP address, the time of archival, the service endpoint, the session files, and a link to the log are provided in each session row.

Figure 5.9. Archive User Interface

The archive table can be sorted similar to the session UI using the previously described col/order query parameter in the URL. As a default, archived sessions are sorted by archival date with the most recent archive shown on top.