You are not logged in. Click here to log in.

Application Lifecycle Management
Login
 Projects
 Reports
 Wiki
 Documents
 Trackers
 SCM Repositories
Last ModifiedRecently Visited Items
Search In Project
Search inClear
anonymous
Tags:  not added yet

FICUS-OMS Framework Extensions

Table of Contents

Python Binding for OMS

The Python binding is an OMS module that allows to run Python scripts as OMS components. In this way, Python OMS components can be integrated into bigger multi-languages modeling solutions.

Because still under testing, the Python binding is temporarily implemented in a forked OMS repository only, available You must login to see this link. Register now, if you have no user account yet..

Intro

Jep

The Python binding is based on Jep back-ends.

Jep (Java Embedded Python) uses the native Python interpreter embedding CPython in Java through JNI. It is multi-thread safe (actually the provided documentation describes it as safe in a heavily threaded environment). It is much faster than alternatives because based on the native Python interpreter.

Requirements

The Python binding module is part of the OMS docker image. You need to install Docker and Git following this procedure to process Python OMS compliant components

Python OMS compliant components

How to modify a Python script to become an OMS-compliant component

A standard Python script needs to be slightly modified in order to get parsed by the Python binding module.

First, adjust the Python script following these simple rules:

  1. Identify inputs and outputs of the Python components;
  2. Declare inputs and outputs at the very start of the Python script;
  3. If the script IS NOT split in functions, bind it into a main function excluding input/output declarations and packages loading;
  4. If the script IS split in functions, identify the main function that has to be executed;
  5. The outputs must also be declared global at the very start of the main function;
  6. point 5 is not required if input variables are also output variables (see Listing 2) 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
 
import sys
import os

# @In("int")
inVar

# @Out("int")
outVar

# @Execute
def execute():
    global outVar

    ...python script...
    outVar = inVar + 3
    ...python script...

    return

Listing 1: Simple Python OMS-compliant script
 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
 
import sys
import os

# @In("int")
# @Out("int")
simpleInOut

# @Out("int")
onlyOutVar

# @Execute
def execute():
    global onlyOutVar

    ...python script...
    simpleInOut += 35
    outVar = simpleInOut + 3
    ...python script...

    return

Listing 2: Input variable is also output variable

Second, add OMS annotations into the modified script:

  1. Because the Python language doesn't have data types, the optional field for @In @Out annotations is REQUIRED. The optional field specifies which Java data type that variable corresponds to;
  2. The @Execute annotation must be located above the signature of the main function;
  3. No annotation required by other functions and scripts called from the main one.

Available data types

The bi-directional data transfer between Java and Python supports the following data types:

Java-side Python-side Python import
int int
Integer Integer
double double
Double Double
String String
int jarray(length, JINT_ID, initialValue) from jep import jarray, JINT_ID
double jarray(length, JDOUBLE_ID, initialValue) from jep import jarray, JDOUBLE_ID
String jarray(length, JSTRING_ID, initialValue) from jep import jarray, JSTRING_ID
List<Integer> []
List<Double> []
List<String> []
List<Object> []
Map<Integer, Object> dictionary
Map<Object, Object> dictionary

Main example

This is a very basic example that shows how to write a Python OMS-compliant component and how to connect it with two Java classes. You must login to see this link. Register now, if you have no user account yet.. A usage example of each available data type is provided.

Flow chart

This is a simple flow chart of the modeling solution. The orange boxes represent Java components, the green boxes represent Python components.

DirectedAcyclicGraph plugin failed: Could not find plugin DirectedAcyclicGraph
(com.ecyrd.jspwiki.plugin.PluginException:Could not find plugin DirectedAcyclicGraph, java.lang.ClassNotFoundException:Class 'DirectedAcyclicGraph' not found in search path!)

Input.java

This Java component allocates and initializes each variable. 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
package py_simpleTest;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List
import java.util.Map;
import oms3.annotations.Execute;
import oms3.annotations.Out;

public class Input {

    // Integer
    @Out
    public int intVal;
    @Out
    public int[] intVectVal;
    @Out
    public Integer integerVal;

    // Double
    @Out
    public double dbVal;
    @Out
    public double[] dbVectVal;
    @Out
    public Double doubleVal;

    // String
    @Out
    public String strVal;
    @Out
    public String[] strVectVal;

    // List
    @Out
    public List intList;
    @Out
    public List doubleList;
    @Out
    public List strList;

    // Map
    @Out
    public Map intMap;

    @Execute
    public void inputProcessing() {

        // Integer
        intVal = 3;
        intVectVal = new int[]{3,4,5};
        integerVal = 33;

        // Double
        dbVal = 3.54;
        dbVectVal = new double[]{3.33,4.44,5.55};
        doubleVal = 3.35;

        // String
        strVal = "Testing jep";
        strVectVal = new String[]{"Testing", "jep"};

        // List
        intList = new ArrayList();
        intList.add(4);
        intList.add(19);
        intList.add(1);
        intList.add(11);
        doubleList = new ArrayList();
        doubleList.add(23.41);
        doubleList.add(23.19);
        doubleList.add(98.33);
        strList = new ArrayList();
        strList.add("test");
        strList.add("jep");

        // Map
        intMap = new HashMap();
        intMap.put(54, "Pasini");
        intMap.put(99, "Cassano");
        intMap.put(21, "Bayliss");

    }
}

inout1.py

This Python OMS-compliant component simply manipulates the input variables and returns the new modified ones. 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
from jep import jarray, JDOUBLE_ID, JINT_ID, JSTRING_ID

# @In("int")
intVal
# @Out("int")
valint
# @In("int[]")
intvectval
# @Out("int[]")
valvectint
# @In("Integer")
integerval
# @Out("Integer")
valinteger

# @In("double")
dbval
# @Out("double")
valdb
# @In("double[]")
dbvectval
# @Out("double[]")
valvectdb
# @In("Double")
doubleval
# @Out("Double")
valdouble

# @In("String")
strval
# @Out("String")
valstr
# @In("String[]")
strvectval
# @Out("String[]")
valvectstr

# @In("List")
intlist
# @Out("List")
listint
# @In("List")
doublelist
# @Out("List")
listdouble
# @In("List")
strlist
# @Out("List")
liststr

# @In("Map")
intmap
# @Out("Map")
mapint

# @Execute
def execute():

    # Output integer
    global valint
    global valvectint
    global valinteger

    # Output double
    global valdb
    global valvectdb
    global valdouble

    # Output string
    global valstr
    global valvectstr

    # Output list
    global listint
    global listdouble
    global liststr

    # Output map
    global mapint

    # Integer manipulation
    valint = intval + 3
    valinteger = integerval + 3
    valvectint = jarray(len(intvectval), JINT_ID, 0)
    for i in range(len(intvectval)):
        valvectint[i] = intvectval[i] * 3

    # Double manipulation
    valdb = dbval * 3.4
    valdouble = doubleval * 8.54
    valvectdb = jarray(len(dbvectval), JDOUBLE_ID, 0.0)
    for i in range(len(dbvectval)):
        valvectdb[i] = dbvectval[i] * 11.756

    # String manipulation
    valstr = strval + ": works!"
    valvectstr = jarray(4, JSTRING_ID, "")
    for i in range(len(strvectval)):
        valvectstr[i] = strvectval[i]
    valvectstr[2] = ":"
    valvectstr[3] = "works"

    # List manipulation
    listint = [1,2,3,4]
    for i in range(len(listint)):
        listint[i] = intlist[i] * listint[i]

    tmpdouble = [1.43,2.156,3.199]
    listdouble = []
    for i in range(len(doublelist)):
        listdouble.append(doublelist[i])

    for i in range(len(doublelist)):
        tmpval = tmpdouble[i] * doublelist[i]
        listdouble.append(tmpval)

    liststr = ["work"]
    liststr.insert(0, strlist[1])
    liststr.insert(0, strlist[0])

    # Dictionary manipulation
    mapint = intmap
    mapint[18] = "Manning"

Output.java

This Java class checks each variable with expected values. It returns error when actual and expected values differs. 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
 
package py_simpleTest;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List
import java.util.Map;
import oms3.annotations.*;

public class Output {

    // Integer
    @In
    public int intVal;
    @In
    public int[] intVectVal;
    @In
    public Integer integerVal;

    // Double
    @In
    public double dbVal;
    @In
    public double[] dbVectVal;
    @In
    public Double doubleVal;

    // String
    @In
    public String strVal;
    @In
    public String[] strVectVal;

    // List
    @In
    public List intList;
    @In
    public List doubleList;
    @In
    public List strList;

    // Map
    @In
    public Map intMap;

    @Execute
    public void exec() {
        integerCheck();
        doubleCheck();
        stringCheck();
        listCheck();
        mapCheck();
    }

    private void integerCheck() {
        System.out.println("Checking for Integer data structures interchange...");
        if (intVal != 6)
            throw new RuntimeException();

        int[] tmp = new int[]{9,12,15};
        for (int i=0; ilength; i++) {
            if (tmp[i]!=intVectVal[i]) {
                throw new RuntimeException();
            }
        }
        if (!integerVal.equals(36))
            throw new RuntimeException();
        System.out.println("Integer data structures correctly interchanged.");
    }

    private void doubleCheck() {
        System.out.println("Checking for Double data structures interchange...");
        if (dbVal != 12.036)
            throw new RuntimeException();

        double[] tmpd = new double[]{39.14748,52.19664,65.2458};
        for (int i=0; ilength; i++) {
            if (tmpd[i]!=dbVectVal[i]) {
                throw new RuntimeException();
            }
        }
        if (!doubleVal.equals(28.609))
            throw new RuntimeException();
        System.out.println("Double data structures correctly interchanged.");
    }

    private void stringCheck() {
        System.out.println("Checking for String data structures interchange...");
        if (!strVal.equals("Testing jep: works!"))
            throw new RuntimeException();
        String[] tmps = new String[]{"Testing","jep",":","works"};
        for (int i=0; ilength; i++) {
            if (!strVectVal[i].equals(tmps[i])) {
                throw new RuntimeException();
            }
        }
        System.out.println("String data structures correctly interchanged.");
    }


    private void listCheck() {
        System.out.println("Checking for List data structures interchange...");
        List intFinalList = new ArrayList();
        intFinalList.add(4*1);
        intFinalList.add(19*2);
        intFinalList.add(1*3);
        intFinalList.add(11*4);
        if (!intList.equals(intFinalList))
            throw new RuntimeException();

        List doubleFinalList = new ArrayList();
        doubleFinalList.add(23.41);
        doubleFinalList.add(23.19);
        doubleFinalList.add(98.33);
        doubleFinalList.add(33.4763);
        doubleFinalList.add(49.99764);
        doubleFinalList.add(314.55768);
        if (!doubleList.equals(doubleFinalList))
            throw new RuntimeException();

        List strFinalList = new ArrayList();
        strFinalList.add("test");
        strFinalList.add("jep");
        strFinalList.add("work");
        if (!strList.equals(strFinalList))
            throw new RuntimeException();
        System.out.println("List data structures correctly interchanged.");
    }

    private void mapCheck() {
        System.out.println("Checking for Map data structures interchange...");
        Map intFinalMap = new HashMap();
        intFinalMap.add(54, "Pasini");
        intFinalMap.add(99, "Cassano");
        intFinalMap.add(21, "Bayliss");
        intFinalMap.add(18, "Manning");
        if (!intMapList.equals(intFinalMap))
            throw new RuntimeException();
        System.out.println("Map data structures correctly interchanged.");
    }

}

The Simulation file (.sim) excercises the components above

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
import static oms3.SimBuilder.instance as OMS3

def home = oms_prj

/*
 * Python component connected to two Java components.
 * 'input' component instantiates and initializes each variable 
 * and passes them further to 'component1' which manipulates each
 * variable accordingly to its data type. Then 'component1' passes
 * each variable further to 'output' which compares them to expected values.
 */
OMS3.sim {

    resource "${home}/lib"
    build()

    model {
        components {
            "input"           "py_simpleTest.Input"
            "output"          "py_simpleTest.Output"
            "component1"      "py_simpleTest.inout1"
        }
        connect {
            // componentname,outfieldname" -> "componentname,infieldname"
            "intput.intVal"        "component1.intval"
            "intput.intVectVal"    "component1.intvectval"
            "intput.integerVal"    "component1.integerval"
            "intput.dbVal"         "component1.dbval"
            "intput.dbVectVal"     "component1.dbvectval"
            "intput.doubleVal"     "component1.doubleval"
            "intput.strVal"        "component1.strval"
            "intput.strVectVal"    "component1.strvectval"
            "intput.intList"       "component1.intlist"
            "intput.doubleList"    "component1.doublelist"
            "intput.strList"       "component1.strlist"
            "intput.intMap"        "component1.intmap"

            "component1.valint"        "output.intVal"
            "component1.valvectint"    "output.intVectVal"
            "component1.valinteger"    "output.integerVal"
            "component1.valdb"         "output.dbVal"
            "component1.valvectdb"     "output.dbVectVal"
            "component1.valdouble"     "output.doubleVal"
            "component1.valstr"        "output.strVal"
            "component1.valvectstr"    "output.strVectVal"
            "component1.listint"       "output.intList"
            "component1.listdouble"    "output.doubleList"
            "component1.liststr"       "output.strList"
            "component1.mapint"        "output.intMap"
        }
        parameter {
        }
    }
}

Other examples

Another example is made available at the You must login to see this link. Register now, if you have no user account yet. repository for exercising generic data types List<Object>, Map<Object, Object> for bi-directional interchange Java-Python.

@TODO

R Binding for OMS

The R binding is an OMS module that allows to run R scripts as OMS components. In this way, R OMS components can be integrated into bigger multi-languages modeling solutions.

Because still under testing, the R binding is temporarily implemented in a forked OMS repository only, available You must login to see this link. Register now, if you have no user account yet..

Intro

Rserve

The R binding is based on Rserve back-ends developed by Simon Urbanek as part of REngine(generic java interface to R).

Rserve supports remote connection, authentication and file transfer, because it is designed on a client-server concept. One Rserve instance can serve multiple clients (applications) simultaneously ensuring separated data spaces for each one. Because R is not multi-thread safe, the synchronization of multiple concurrent connections is managed by Rserve. These features allow distributed computing on multiple machines and CPUs. Furthermore, to gain speed while minimazing the amount of transferred data, Rserve communicates with applications in a binary form.

Requirements

The R binding is part of the OMS docker image. You need to install Docker and Git following this procedure to process R OMS compliant components

R OMS compliant components

How to modify an R-script to become an OMS-compliant component

A standard R script needs to be slightly modified in order to get parsed by the R binding module.

First, adjust the R script following these simple rules:

  1. Identify inputs and outputs of the R component;
  2. Declare inputs and outputs at the very start of the R script;
  3. The result must be assigned to each output variable through the double arrow assignment operator <<-;
  4. If the script IS NOT split in functions, bind it into a main function excluding input/output declarations and packages loading;
  5. If the script IS split in functions, identify the main function that has to be executed;
  6. Comment every print() or show() commands. Rserve doesn't handle them at the moment;
  7. Functions cannot be declared into the call of another function. For example: a <- compute(a, function(x) { x*3 }) doesn't work. You need to first declare the function and then pass it to the called function. 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
 
library(raster);

# @Execute
process <- function() {

    ...R script...

    simpleComputation <- function(x) {
        x * 3
    }

    a <- compute(a, simpleComputation)

    ...R script...

}

Listing 1: Example about how to declare and call a nested function

Second, add OMS annotations into the modified script:

  1. Because the R language doesn't have data types, the optional field for @In @Out annotations is REQUIRED. The optional field specifies which Java data type that variable corresponds to (see the example below);
  2. The @Execute annotation must be located above the signature of the main function;
  3. No annotation required by the other inner functions and scripts called from the main one.

Available data types

The bi-directional data transfer between Java and R supports the following Java standard data types:

  • int
  • double
  • String
  • int[]
  • double[]
  • String[]

Moreover, it supports the transfer of some objects, mapped in the following table:

Java-side R-side
GridCoverage2D Raster
CoverageStack RasterStack
List<GridCoverage2D> list() of Raster

For implementing these features, the raster package has been considered R-side, while the GridCoverage2D class from Geotools has been fully integrated Java-side. The CoverageStack object has been implemented on purpose. The latter (as the J2R and R2J classes, which manages the bi-directional data transfer) is automatically generated into the src/ directory of the OMS project when both of the following events happens:

  • the source code of the OMS project is built through ant all command;
  • an R component involving a Raster object or a RasterStack object is recognized by the building system into the source path.

If an R component is recognized into the source path, but there is no input/output of Raster objects or RasterStack objects, the CoverageStack class and the J2R and R2J classes are not generated. In this case, just the Java wrapper for the R component is generated.

NA check

The Double.NaN value is correctly converted into NA while transferring double values from Java to R and vice-versa

The Main Example

This is a very basic example that shows how to write three R OMS compliant components and how to connect them. It is made available You must login to see this link. Register now, if you have no user account yet.. Suppose you already have a Java component which provides the raster reading and another one which provides the raster writing.

Flow chart

This is a simple flow chart of the modeling solution. The orange boxes represent Java components, the blue boxes represent R components.

DirectedAcyclicGraph plugin failed: Could not find plugin DirectedAcyclicGraph
(com.ecyrd.jspwiki.plugin.PluginException:Could not find plugin DirectedAcyclicGraph, java.lang.ClassNotFoundException:Class 'DirectedAcyclicGraph' not found in search path!)

stack.R

The first R component simply takes two input maps and return a RasterStack. 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
 
library(raster)

# @In("GridCoverage2D")
inMap1;

# @In("GridCoverage2D")
inMap2;

# @Out("CoverageStack")
outStack;

# @Execute
process <- function() {

    outStack <<- stack(x=c(inMap1,inMap2))

}

Listing 2: First component stack.R

transform.R

The second R component transforms the NA values to 0 into each map of the RasterStack and returns a List or Rasters. 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
library(raster)

# @In("CoverageStack")
rasterStack

# @Out("List")
rasterList

# @Execute
process <- function() {

    rasterList <<- transform(rasterStack)

}

transform <- function(rasterStack) {
    tmpList <-c()
    for(layer in 1:nlayers(rasterStack)) {
        map <- rasterStack[[layer]]
        map[is.na(map)] <- 0
        tmpList <- c(tmpList, map)
    }
    return(tmpList)
}

Listing 3: Second component transform.R

split.R

The third component splits the input list into two Rasters 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
 
library(raster)

# @In("List")
rasterList

# @Out("GridCoverage2D")
raster1

# @Out("GridCoverage2D")
raster2

# @Execute
process <- function() {

    raster1 <<- rasterList[[1]]
    raster2 <<- rasterList[[2]]

}

Listing 4: Third component split.R

The Simulation file (.sim) excercises the components above

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
import static oms3.SimBuilder.instance as OMS3

def home = oms_prj

/*
 * Two connected components.
 * Parameter passes from 'sim' file into 'Comp1', 'Comp1' 
 * modifies it and passes it further to 'Comp2' which prints it out.
 */
OMS3.sim {

    resource "${home}/lib"
    build()

    model {
        components {
            "reader1"     "edu.colostate.engr.alm.RasterReader"
            "reader2"     "edu.colostate.engr.alm.RasterReader"
            "stack"         "edu.colostate.engr.alm.stack"
            "transform"  "edu.colostate.engr.alm.transform"
            "split"          "edu.colostate.engr.alm.split"
            "writer1"     "edu.colostate.engr.alm.RasterWriter"
            "writer2"     "edu.colostate.engr.alm.RasterWriter"
        }
        connect {
            // componentname,outfieldname" -> "componentname,infieldname"
            "reader1.gc" "stack.inMap1"
            "reader2.gc" "stack.inMap2"

            "stack.outStack" "transform.rasterStack"

            "transform.rasterList" "split.rasterList"

            "split.raster1" "writer1.gc"
            "split.raster2" "writer2.gc"
        }
        parameter {
            // feed the beginning of the pipeline!
            "reader1.file" "path/to/inFile1.tif"
            "reader2.file" "path/to/inFile2.tif"

            // feed the end of the pipeline!
            "writer1.file" "path/to/outFile1.tif"
            "writer2.file" "path/to/outFile2.tif"

        }
    }
}

Listing 5: The simulation file.

Other examples

Other examples are made available at the You must login to see this link. Register now, if you have no user account yet. repository (where examples and documentation about the usage of Standard Data Types are gathered) and the You must login to see this link. Register now, if you have no user account yet. repository (where examples and documentation about the usage of Raster and RasterStack objects are gathered).

@TODO

  • Massive testing under each Operating System (still memory issues under Windows 7 operating system);
  • Rserve cannot handle print() and show() commands;
  • R2J & J2R => check robustness (e.g. the crs and proj of maps from/to Lists/Stacks must be the same for each map);
  • Input and output of the RUG model must read and write each raster format. Right now, GeoTiff is the only available;
  • Get R component line that returns error;
  • Redesign of R2J: better implementation of the R2J class is required for future developments;
  • Is there a better memory management R-side for parallel loops?;
  • @In @Out => same variable but different data type doesn't work because of the Java wrapper;
  • Testing of multithreaded requests to a single Rserve instance => This seems working. Other tests required;
  • Stop simulation button doesn't kill Rserve service and R threads. Furthermore, if an R component fails, a running parallel R component doesn't stop.
Shortcuts
Requirements Management:

Requirements:

Work Items:

QA & Test Management:

Release Management: