Versions Compared

Old Version 18

changes.mady.by.user Unknown User (tarekkh)

Saved on

compared with

New Version Current

changes.mady.by.user Aleksandra Nowak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

DPS description :

 

Deployed Toplogies:

Two topologies are deployed :

  • XSLT topology: used to transform an XML file based on XSL tranformation file.
  • Image conversion topology: used to convert TIFF type images into JP2 type images.

DPS usage tutorial :

This tutorial covers the usage of DPS

...

REST API

Here you can find out more about how Data Processing Service works.

Note

An admin should grant the user of the topology the ability to interact with a specific topology by granting this user read and write permissions on the requested

...

topology.

 

curl -i -X POST --user admin:admin -H "Content-Type: application/x-www-form-urlencoded" -d 'username=topology_user_test' http://cloud.europeana.eu/services/image_convert/permit

 

...

Submit a task to

...

DPS


Two types of DPS tasks could be sent :

  • A task to convert/transform list of files.
  • A task to convert/transform files inside a list of datasets.

Make sure that the required fields are included in your task or a method not allowed exception will be thrown and the task will not be sent!.

...

 For Image Conversion topology 
Anchor
ic
ic

...

  • Submit a list of files:

     The required fields are:

     FILE_URLS as a data entry. And the following parameters: 
    • MIME_TYPE : The mime type of the image which will be converted.
    • OUTPUT_MIME_TYPE : The expected mime type of the output image.

    • TASK_SUBMITTER_NAME : The owner of the converted image.

      Code Block
      curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData":{"entry":[{"key":"FILE_URLS","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME1 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME2 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME3 "}]},"parameters":{"entry":[{"key":"TASK_SUBMITTER_NAME","value":"topology_user_test"},{"key":"OUTPUT_MIME_TYPE","value":"image/jp2"},{"key":"MIME_TYPE","value":"image/tiff"}]}}' -i http://cloud.europeana.eu/services/image_convert/tasks

  • Submit a list of datasets:

     DATASET_URLS as a data entry. And the following parameters: 

    • MIME_TYPE : The mime type of the image which will be converted.
    • OUTPUT_MIME_TYPE : The expected mime type of the output image.

    • TASK_SUBMITTER_NAME : The owner of the converted image.

      Code Block
      curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData": {"entry": [{"key": "DATASET_URLS","value": "http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER1/data-sets/dDATASET_NAME1 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER2/data-sets/dDATASET_NAME2 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER3/data-sets/dDATASET_NAME3"}]},"parameters": {"entry": [{"key": "TASK_SUBMITTER_NAME","value": "admin"}, {"key": "OUTPUT_MIME_TYPE", "value": "image/jp2"}, {"key": "MIME_TYPE","value": "image/tiff"}]}}' -i http://cloud.europeana.eu/services/image_convert/tasks

    • If a task was submitted successfully the sender should recieve a response confirming the creation of the task, like this:

    Code Block
    201 Created
Server: Apache-Coyote/1.1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Location: http://cloud.europeana.eu/services/image_convert/tasks/TASK_ID
Content-Length: 0
Date: creation Date

    The location filed should have a url to the newely created task and at the end of this url a TASK_ID should exist. This taskID should be used to follow the progress/notification of this task.

     

     



B.    For

For XSLT topology:

  • Submit a list of files:

     The required fields are:

     FILE_URLS as a data entry. And the following parameters:
    • XSLT_URL: a URL for
    • XSLT
    • XSL file which will be used to transform  XML documents.

    • TASK_SUBMITTER_NAME: The owner of the transformed files. 

      Code Block
      curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData":{"entry":[{"key":"FILE_URLS","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME1 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME2 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME3"}]},"parameters":{"entry":[{"key":"TASK_SUBMITTER_NAME","value":"topology_user_test"},{"key":"XSLT_URL","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/XSLT_FILE_NAME"}]}}' -i http://cloud.europeana.eu/services/xslt_transform/tasks

  • Submit a list of datasets:

     DATASET_URLS as a data entry. And the following parameters: 

    • XSLT_URL: a URL for
    • XSLT
    • XSL file which will be used to transform  XML documents.

    • TASK_SUBMITTER_NAME: The owner of the transformed files.  

      Code Block
      curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData":{"entry":[{"key":"DATASET_URLS","value":"http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER1/data-sets/dDATASET_NAME1 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER2/data-sets/dDATASET_NAME2 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER3/data-sets/dDATASET_NAME3"}]},"parameters":{"entry":[{"key":"TASK_SUBMITTER_NAME","value":"topology_user_test"},{"key":"XSLT_URL","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/XSLT_FILE_NAME"}]}}' -i http://cloud.europeana.eu/services/xslt_transform/tasks

Notes:

  •  
  • If a task was submitted successfully the sender should recieve a response confirming the creation of the task, like this:  

     

    Code Block
    201 Created
Server: Apache-Coyote/1.1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Location: http://cloud.europeana.eu/services/
xslt
  • image_
transform
  • convert/tasks/TASK_ID
Content-Length: 0
Date: creation Date
  •      The location filed should have a url to the newely created task and at the end of this url a TASK_ID should exist. This taskID should be used to follow the progress 
/notification 
  •  of this task.
     
  •       
  • There is a list of optional parameters which could be used in a DPS task :          
    • REPRESENTATION_NAME: If a task designed to process specific representation inside a dataset.
    • NEW_REPRESENTATION_NAME: The name of the representation used to store the processed files.        

 Reporting 
You can 
3. Reporting:
 
To monitor the progress of your tasks.
 
Reporting is divided into two requests:   
    • Report on a task level:
       Code Block
       curl --user topology_user_test:topology_user_test http://cloud.europeana.eu/services/image_convert/tasks/TASK_ID/progress
      This call should give you information about the status of the submittted task like the status of the task (SENT-PROCESSED-CURRENTLY_PROCESSED,PENDING,DROPPED), total number of files inside a task, the number of the processed files , the exact time stating when task was sent, start started processing and finished . An example response :
       Infocode
      {"taskId":TASK_ID,"topologyName":"ic_topology","totalSize":2,"processed":2,"state":"PROCESSED","info":"Completely processed","sent_time":"2016-27-07 09:27:40","start_time":"2016-28-07 09:28:12","finish_time":"2016-28-07 09:28:12"}
           
    • detailed Detailed information about every file in a task. 
       Code Block
       curl --user topology_user_test:topology_user_test http://cloud.europeana.eu/services/image_convert/tasks/TASK_ID/notification
      An example response :
       Code Block
      [{"task_id":TASK_ID,"topology_name":"ic_topology","resource":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME1",
"state":"SUCCESS","info_text":"","additional_informations":"","result_resource":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/NEW_REPRSENTATION_NAME/versions/NEW_VERSION/files/NEW_FILENAME1"},{"task_id":TASK_ID,"topology_name":"ic_topology","resource":" http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME2",
"state":"SUCCESS","info_text":"","additional_informations":"","result_resource":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/NEW_REPRSENTATION_NAME/versions/NEW_VERSION/files/NEW_FILENAME2"}]