1.1. Introduction
This tutorial is designed to introduce the concept of building simple workflows within Kepler 2.5.
Kepler is a workflow engine and design platform for analysing and modelling scientific data. Kepler provides a graphical interface and a library of pre-defined components to enable users to construct scientific workflows which can undertake a wide range of functionality. It is primarily designed to access, analyse, and visualise scientific data but can be used to construct whole programs or run pre-existing simulation codes.
Kepler builds upon the mature Ptolemy II framework, developed at the University of California, Berkeley. Kepler itself is developed and maintained by the cross-project Kepler collaboration.
The main components in a Kepler workflow are actors, which are used in a design (inherited from Ptolemy II) that separates workflow components ("actors") from workflow orchestration ("directors"), making components more easily reusable. Workflows can work at very levels of granularity, from low-level workflows (that explicitly move data around or start and monitor remote jobs, for example) to high-level workflows that interlink complex steps/actors. Actors can be reused to construct more complex actors enabling complex functionality to be encapsulated in easy to use packages. A wide range of actors are available for use and reuse.
1.1.1. 1 Installing Kepler
Requirements
- Kepler take ~300 MB. Please make sure that you have at least this amount of free space available.
- The required version of JRE is 1.8. Please check if you have it installed on your system before taking further steps described in this tutorial.
Installing Kepler
Download and install kepler from: https://kepler-project.org/users/downloads
You can always access tutorial workflows at following address: tutorial.tar
Now you can start Kepler application and proceed to tutorial examples.
1.1.2. 2. Executing simple workflows
In order to execute workflow, you have to load workflow XML file into Kepler. During this tutorial session we will use following workflows:
Workflow description | Location |
---|---|
"Hello world" | $HOME/tutorial/workflow/basic/Hello_World.xml |
"Hello world debug" | $HOME/tutorial/workflow/basic/Hello_World_Debug.xml |
"If-else-simple" | $HOME/tutorial/workflow/basic/if_else_simple.xml |
"If-else-simple-expression" | $HOME/tutorial/workflow/basic/if_else_simple_expression.xml |
"Simple-Loop" | $HOME/tutorial/workflow/basic/simple_loop.xml |
"Array-Copy-Loop" | $HOME/tutorial/workflow/basic/array_loop_copy_files.xml |
"Feedback-Loop" | $HOME/tutorial/workflow/basic/array_loop_composite_repeat_feedback.xml |
"Loop with relation" | $HOME/tutorial/workflow/basic/loop/ssimple_loop_relation.xml |
"Loop without DDF Boolean Select" | $HOME/tutorial/workflow/basic/loops/loop-no-ddf-b-s.xml |
"Loop based on variables" | $HOME/tutorial/workflow/basic/loops/loop-variables.xml |
"Loop using encapsulated Python script" | $HOME/tutorial/workflow/basic/python/python_script.xml |
"Series plotting based on looped computations" | $HOME/tutorial/workflow/basic/python/xyplotter.xml |
1.1.2.1. 2.1 Hello world workflow
1.1.2.1.1. 2.1.1 Using existing "Hello world" workflow
- know how to start Kepler
- know how to load simple workflow
- know how to execute workflow
- know how to animate workflow
Film available: http://www.youtube.com/watch?v=1xsPH6Mnzx0
In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:
Start Kepler application by issuing:
cd ~/kepler/ ./kepler.sh
Open "Hello world" workflow by issuing: File -> Open and navigate to:
$HOME/tutorial/workflow/basic/Hello_World.xml
After workflow is opened, press "Play" button
workflow should generate output within Display actorAnimating workflows
In Kepler it is possible to animate workflows during execution. In order to animate workflow you have to turn on animations. You can do this by choosing: Tools -> Animate at Runtime...
Demo movie for this feature can be found at following location: animation
1.1.2.1.2. 2.1.2 Using existing "Hello world - with debug" workflow
- know how to start Kepler
- know how to load simple workflow
- know how to execute workflow
- know how to listen to the actor
Film available: http://www.youtube.com/watch?v=EVGSXC4kcks
In this exercise you will execute simple Kepler workflow with Debug information. In order to this follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Open "Hello world debug" workflow by issuing: File -> Open and navigate to:
$HOME/tutorial/workflow/basic/Hello_World_Debug.xml
- After workflow is opened, "Right-click" Expression actor and choose "Listen to actor"
- Press "Play" button
Workflow should generate output within Display actor and should print debug information generated by Expression actor
1.1.2.1.3. 2.1.3 Building "Hello world" from the scratch
- know how to start Kepler
- know how to build simple workflow
- know how to connect elements
- know how to add elements to the workflow
- know how to search for the actors within Kepler's library
Film available: http://www.youtube.com/watch?v=DXXYnuDjnWw
In this exercise you will build simple "Hello World" workflow and execute it. In order to get this task done, follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
- Type in "SDF" into "Search" field and press "Search" button
- Drag and Drop SDF director into workflow
- Right-click on SDF director and choose "Configure Director"
- Set number of iterations to "1"
- Type in "String" into "Search" field and press "Search" button
- Add "String Constant" actor to the workflow
- Right-click "String Constant" actor and choose "Configure Actor"
- Type "Hello world!" into "value" field
- Commit changes
- Type in "Display" into "Search" field and press "Search" button
- Add "Display" actor into workflow
Connect "String Constant" actor with "Display" actor
Intermediate results
- After workflow is ready, press "Play" button
workflow should generate output within Display actor
1.1.2.2. 2.2 Relations, Paths and Synchronization
- know how to add elements into workflow
- know hot to use expressions
- know how to synchronize workflow's execution
- know how to use parametrs
- know how to use relations
Film available: http://www.youtube.com/watch?v=OCO9L5MzUrM
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
- Add DDF director into workflow
- Add Constant into workflow and set it's value to "true" (double click Constant and enter "true")
- Set Constant "firingCountLimit" to "1" (Right Click -> Configure Actor -> firingCountLimit Text Field)
- Add Relation next to the Constant
- Connect Relation and Constant::output
- Add Parameter and rename it to "a" (Right click -> Customize name)
- Set value of a to "1" (double click a)
- Add Parameter and rename it to "b" (Right click -> Customize name)
- Set value of b to "2" (double click b)
- Add Constant into workflow and rename it to "Send a to output"
- Set Send a to output value to "a"
- Add Constant into workflow and rename it "Send b to output"
- Set *Send b to output" value to "b"
- Connect Send a to output::trigger" with *Relation
- Connect Send b to output::trigger" with *Relation
- Add Relation to workflow and connect it with Send a to output
- Add Relation to workflow and connect it with Send b to output
- Add Display to workflow and connect relation connected to Send a to output
- Set Display "Display name" (Right click -> Customize Name) to "Value a"
- Add Display to workflow and connect it with other relation
- Set Display "Display name" (Right click -> Customize Name) to "Value b"
- Add Expression to the workflow
- Add input port input_a to the Expression (Right click -> Configure Ports -> Add, select checkbox "in")
- Add input port input_b to the Expression (Right click -> Configure Ports -> Add, select checkbox "in")
- Connect Expression::input_a with relation bound to Send a to output
- Connect Expression::input_b with relation bound to Send b to output
- Add Display to the workflow and set it's "Display name" to "Result"
- Connect Result::input with *Expression::output"
Set Expression value to "a+b" (Double click Expression)
At this point your workflow should be similar to the one belowIntermediate results
- Execute workflow
Simple modification in order to make Kepler workflow fail - Set Expression value to "a/b"
- Set b value to "0"
1.1.2.3. 2.3 If-else workflow
1.1.2.3.1. 2.2.1 Using existing "if-else" workflow
- know how to use different paths for data flow
- know how to split workflow execution path
- know how to use Boolean Switch actor
Film available: http://www.youtube.com/watch?v=rr03bekyiDU
In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Open "If-else" workflow by issuing: File -> Open and navigate to:
$HOME/tutorial/workflow/basic/if_else_simple.xml
- After workflow is opened, press "Play" button
workflow should generate output within Display actor
1.1.2.3.2. 2.2.2 Building "if-else" from the scratch
- know how to use different paths for data flow
- know how to split workflow execution path
- know how to use Boolean Switch actor
Film available: http://www.youtube.com/watch?v=3M7IFyzSTAY
In this exercise you will build "if-else" workflow.
You should complete previous examples before starting this one
In this example it is assumed that you already know how to use actor/director browser (left panel) and how to put actors into workflow (right panel)
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
- Drag and Drop "DDF Director" into workflow
- Drag and Drop "String Constant" actor into workflow
- Change it's name to "String Hello" (Right-click -> Custimize name)
- Change it's value to "Hello world will test if-else" (Right-click -> Configure Actor -> value)
- Change it's firingCountLimit to "1" (Right-click -> Configure Actor -> firingCountLimit)
- Drag and Drop "Parameter" actor into workflow
- Change it's name to "a" (Right-click -> Customize name)
- Change it's value to "1" (Double click -> value)
- Drag and Drop "Parameter" actor into workflow
- Change it's name to "b" (Right-click -> Customize name)
- Change it's value to "2" (Double click -> value)
- Drag and Drop "Boolean Switch" actor into workflow
- Drag and Drop "Display" actor into workflow next to "Boolean Switch" actor
- Change it's name to "Display if"
- Drag and Drop "Display" actor into workflow below "Display if" actor
- Change it's name to "Display else"
- Drag and Drop "Constant" actor into workflow below "Boolean Switch" actor
- Change it's value to "a < b" (Right-click -> Configure Actor -> value)
- Change it's firingCountLimit to "1" (Right-click -> Configure Actor -> firingCountLimit)
- After all actors are at the workflow's area, you have to connect them
- Connect Boolean Switch::trueOutput with Display if::input
- Connect Boolean Switch::falseOutput with Display else::input
- Connect Boolean Switch::input with String Hello::output
Connect Boolean Switch::control with Constant::output
At this point your workflow should be similar to the one belowIntermediate results
- Press "Play" button
workflow should generate output within "Display else" actor - Change value of Constant to "a > b" and execute workflow once again
- Save the workflow (e.g. as ~/my_workflow.xml) - we will need it in next excersise
1.1.2.3.3. 2.2.2 Building "if-else-expression" from the scratch
- know how to use different paths for data flow
- know how to split workflow execution path
- know how to use Boolean Switch actor
- know how to use Expression
- know how to use data flowing within workflow
Film available: http://www.youtube.com/watch?v=qC6eVPXW4Fs
In this exercise you will build "if-else-expression" workflow.
You should complete previous examples before starting this one
In this example it is assumed that you already know how to use actor/director browser (left panel) and how to put actors into workflow (right panel)
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Load workflow that you have previously saved (~/my_workflow.xml) or open workflow at following location:
$HOME/tutorial/workflow/basic/if_else_simple.xml
- Remove link between Display if and Boolean Switch (select link and press "Delete" or choose Edit -> Delete)
- Remove link between Display else and Boolean Switch
- Add Expression between Display if and Boolean Switch
- Set Expression Display name to "Expression if"
- Add input port in into Expression if
Set Expression if value to
in + " - this was added by Expression if"
remember to copy " as well!
- Connect Expression if::in with Boolean Switch::trueOutput
- Connect Expression if::output with Display if::input
- Add Expression between Display else and Boolean Switch
- Set Expression Display name to "Expression else"
- Add input port in into Expression else
Set Expression else value to
in + " - this was added by Expression else"
remember to copy " as well!
- Connect Expression else::in with Boolean Switch::falseOutput
- Connect Expression else::output with Display else::input
At this point your workflow should be similar to the one below
Intermediate results
- press "Play" button
workflow should generate output within "Display if" actor - Change value of Constant to "a < b" and execute workflow once again
1.1.3. 3 Loops within Kepler
In this section of tutorial we will go through basic concepts of looping within Kepler. We will execute simple loop, build it from the scratch and, at the end, we will go through more complex examples of loops.
1.1.3.1. 3.1 Executing simple loop example (classic)
- know how to build simple loops
- know how to use SampleDelay actor
- know how to create loop condition checks
- know difference between SDF and DDF Directors
Film available: http://www.youtube.com/watch?v=fJlV7Jd30cQ
In this exercise you will execute simple loop example. In order to this follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Open workflow
$HOME/tutorial/workflow/basic/simple_loop.xml
You should see following workflow loaded into Kepler
- After workflow is loaded, execute it
workflow should generate output within Display actor - You can change Constant values and see what happens after you start workflow again
1.1.3.2. 3.2 Executing simple loop example (using relation instead of SampleDelay)
- know how to build simple loops
- know how to create loop condition checks
In this exercise you will execute simple loop example. In order to this follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Open workflow
$HOME/tutorial/workflow/basic/loops/simple_loop_relation.xml
You should see following workflow loaded into Kepler
- After workflow is loaded, execute it
workflow should generate output within Display actor - You can change Constant values and see what happens after you start workflow again
1.1.3.3. 3.3 Executing simple loop example (using parameters and Variable Setter)
- know how to build simple loops
- know how to use Variable Setter actor
- know how to create loop condition checks
In this exercise you will execute simple loop example. In order to this follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Open workflow
$HOME/tutorial/workflow/basic/loops/loop-variables.xml
You should see following workflow loaded into Kepler
- After workflow is loaded, execute it
workflow should generate output within Display actor - You can change Constant values and see what happens after you start workflow again
1.1.3.4. 3.4 Executing simple loop example (without DDF Boolean Select actor)
- know how to build simple loops
- know how to create loop condition checks
In this exercise you will execute simple loop example. In order to this follow the instructions:
If Kepler is not already running start it by issuing:
cd ~/kepler ./kepler.sh
Open workflow
$HOME/tutorial/workflow/basic/loops/loop-no-ddf-b-s.xml
You should see following workflow loaded into Kepler
- After workflow is loaded, execute it
workflow should generate output within Display actor - You can change Constant values and see what happens after you start workflow again
1.1.3.5. 3.5 Building simple loop from the scratch
- know how to build simple loops
- know how to use SampleDelay actor
- know how to create loop condition checks
- know difference between DDF and SDF directors
Film available: http://www.youtube.com/watch?v=oYdOYnK7WI4
In this exercise you will build simple loop. In order to this follow the instructions:
Start Kepler application by issuing:
cd ~/kepler ./kepler.sh
- Add DDF director into workflow
- Add Constant into workflow, set it's value to 5
- Add DDF Boolean Select actor to the workflow
- Connect Constant::output with DDF Boolean Select::falseInput
- Add Relation next to DDF Boolean Select
- Connect Relation (we will call it Relation A) with DDF Boolean Select::output
- Add Expression actor to the workflow (next to Relation A)
- Add input port in into Expression
- Connect Relation A with Expression::in
Set Expression value to
in > 0
- Add Relation next to Expression (we will call it Relation B)
Connect Expression::output with Relation B
Intermediate result
- Add SampleDelay actor to workflow
- Change SampleDelay::input port direction to EAST (Right click -> Configure Ports -> Direction)
- Change SampleDelay::output port direction to WEST (Right click -> Configure Ports -> Direction)
- Connect SampleDelay::output with DDF Boolean Select::control
- Connect SampleDelay::input with Relation B
Set SampleDelay value to
{false}
- Add Boolean Switch next to Relation B
- Connect Boolean Switch::control with Relation B
Connect Boolean Switch::input with Relation A
Intermediate result
- Add Expression into workflow and set it's name to Decrease counter
- Add input port in into Decrease counter and set it's Direction to EAST
- Set output port Direction to WEST
Set Expression value to
in - 1
- Connect Decrease counter::output with DDF Boolean Select::trueInput
- Connect Decrease counter::input with Boolean Select::trueOutput
- Add Is Present next to Boolean Switch
- Connect Is Present::input with Boolean Switch::falseOutput
- Add Stop next to Is Present
- Connect Stop::input with Is Present::output
- Add Display next to Relation A
Connect Display::input with Relation A
Intermediate result
- After workflow is opened, press "Play" button
- You can change Constant values and see what happens next time you start workflow
1.1.3.6. 3.6 Executing advanced loop workflow (Composite loop + Repeat)
- know how to utilize loop concept
- know how to build advanced loop workflows
- know how to use Repeat actor
- know how to set value of the parameter
Film available: http://www.youtube.com/watch?v=muhBH7jM5dU
In this exercise you will execute advanced loop workflow. In order to this follow the instructions:
Start Kepler application by issuing:
cd ~/kepler ./kepler.sh
Open workflow
$HOME/tutorial/workflow/basic/array_loop_copy_files.xml
- After workflow is loaded
You can open composite actor by right clicking it and choosing "Open"
You can start workflow by pressing Play button
workflow should copy input files from/to$HOME/tutorial/data -> $HOME/tutorial/output
You can open terminal and verify it's execution results
ls -la ~/tutorial/data ls -la ~/tutorial/output
1.1.3.7. 3.7 Executing advanced loop workflow (Composite loop + feedback)
- know how to utilize loop concept
- know how to build advanced loop workflows
- know how to use Repeat actor
- know how to set value of the parameter
Film available: http://www.youtube.com/watch?v=4xjLcI776vg
In this exercise you will execute advanced loop workflow with feedback. In order to this follow the instructions:
Start Kepler application by issuing:
cd ~/kepler ./kepler.sh
Open workflow
$HOME/tutorial/workflow/basic/array_loop_composite_repeat_feedback.xml
- After workflow is loaded
open composite actor by right clicking it and choosing "Open"
- You can execute it by pressing Play button
workflow should generate output within Display actor
Animating workflows
In Kepler it is possible to animate workflows during execution. In order to animate workflow you have to turn on animations. You can do this by choosing: Tools -> Animate at Runtime...
Demo movie for this feature can be found at following location: animation
1.1.3.8. 3.7 Creating a loop using PythonScript actor
- know how to use Python script inside a Kepler actor
- know how to create a generic loop using PythonScript actor
In this exercise you will create a simple loop in Python and put it inside a special Kepler actor. Steps below describe the process of workflow creation from scratch. If you wish to see a final solution, please open the workflow $HOME/tutorial/workflow/basic/python/python_script.xml
Start Kepler application by issuing:
cd ~/kepler ./kepler.sh
- Instantiate a PythonScript actor by choosing menu Tools -> Instantiate Component and setting as Class name a value ptolemy.actor.lib.python.PythonScript.
- This actor starts with zero ports. They need to be added manually. Please right-click on PythonScript and choose Configure Ports.
Add an input port named in and output port named out.
Kepler's automatic type resolver may not correctly infer types of PythonScript ports due to dynamic features of Python programming language. This may lead to errors and unexpected behaviour. Thus you need to specify these types explicitly. For this tutorial, please set type of in to int and type of out to arrayType(int).
- By default Kepler initialises the script parameter of this actor to be of type Line. To develop a script in Python, it needs to be changed. Please right-click on PythonScript and choose Configure Actor. Go to Preferences and select expert mode.
- Close the window with actor's preferences and once again start with right-clicking and choosing Configure Actor. Again choose Preferences and change type of script parameter to Text.
- Now you can see a Python code displayed in several lines. Some remarks here:
- Python is a dynamic language, so no typecasting takes place,
- Do not declare any constructor.
- You only need to fill the fire() method.
- You can assume that the configured ports are already instantiated (ie. you can use names in and out to work with actor's ports)
- Let's assume a following problem to solve. The actor receives a number which will be interpreted as iteration count. In each i-th iteration, the actor will output i2. Example: in = 4, out = {0, 1, 4, 9}.
In Python, the following script will do this:
import ptolemy.data class Main: def fire(self): # read value of input token val = self.in.get(0).intValue() arr = [] for i in range(val): # create a new IntToken with each value arr.append(ptolemy.data.IntToken(i**2)) # send an ArrayToken with array of values self.out.send(0, ptolemy.data.ArrayToken(arr)) return
- You can now instantiate Constant actor. Set its firingCountLimit to 1 and value to 4. Connect it with in port of PythonScript.
- Instantiate also a Display actor and connect PythonScript's out with it.
- Finally add an SDF actor and execute the workflow.
- You will see {0, 1, 4, 9}. Now you can change the input value 4 to some other one. Or you can change the actor source code to execute a different task.
1.1.3.9. 3.8 Creating a time-loop with series plotting
- know how to provide data from the loop to the plotting actor
In this exercise you will create a simple loop containing some potentially time-consuming operations which will be plotted live. Steps below describe the process of workflow creation from scratch. If you wish to see a final solution, please open the workflow $HOME/tutorial/workflow/basic/python/xyplotter.xml
Start Kepler application by issuing:
cd ~/kepler ./kepler.sh
- Put DDF Director.
- Put Constant actor and set its firingCountLimit to 1. This will be the starting source for the example workflow.
- Put a relation symbol (Ctrl+click) next to the Constant and link them.
- Put an Expression actor and add an input port named in. Let's set the expression to in * in which indicates that we will plot the square function. Then connect the relation with in port.
- Put an XYPlotter actor and connect its inputX with the relation symbol and inputY with Expression actor's output port.
- Your workflow should look like the one presented below. It applies some function to input data and plots it. However, it works for a single value now. We need to create a loop.
- Put another Expression actor. Add an input port named in, set expression to in + 1 and connect the input port with relation. This is responsible for the step of loop.
- Create another relation symbol next to this Expression actor and connect them.
- Put next Expression actor. Again add an input port named in, set expression to in < 10 and connect the input port with the just created relation. This is responsible for loop termination when it reaches specific point.
- Put Boolean Switch actor, connect its input port to the in + 1 expression and its control port to the in < 10 expression.
- Now we want to simulate the time-consuming behaviour, so we are going to add an artificial sleep time. To do this, you need to choose from menu Tools -> Instantiate Component and set as Class name a value ptolemy.actor.lib.Sleep. A new Sleep actor will appear. It's purpose is to grab some input, wait for the specified amount of time and then send the data. For this workflow, please set its sleepTime to 1000 (the unit here is milliseconds, so we will simulate one second of time-consuming operations).
- Connect Boolean Switch trueOutput port with Sleep's input, and Sleep's output to the relation symbol at the beginning of the loop. Your workflow should look like the one below. You can run it and you will see that the output is plotted live and updated every second (ie. every time it receives new data).