• Created by Unknown User (tarekkh), last modified on 08 Jul 2016

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 57 Next »

DPS description : 

A part of eCloud dedicated to provide data processing facility. The client triggers the interaction with DPS by sending a task containing list of files in a purpose of processing them. This processing is done by already deployed topologies created for specific processing purposes. DPS task should be formulated to have the necessary data needed to point to specific files inside MCS, those files will be processed by DPS topologies and then uploaded back to MCS. 

     

Deployed Toplogies:

Two topologies are deployed :

  • XSLT topology: used to transform an XML file based on XSL tranformation file.
  • Image conversion topology: used to convert TIFF type images into JP2 type images.

DPS usage tutorial :

This tutorial covers the usage of DPS-Rest APIs:

1. An admin should grant the user of the topology the ability to interact with a specific topology by granting this user read and write permissions on the requested topology. In the following command we are granting user: topology_user_test the permission to interact with ic topology.

curl -i -X POST --user admin:admin -H "Content-Type: application/x-www-form-urlencoded" -d 'username=topology_user_test' http://cloud.europeana.eu/services/image_convert/permit

 

2. Submit a task to dps;

Two types of DPS tasks could be sent :

    • A task to convert/transform list of files .
    • A task to convert/transform files inside a list of datasets.

Make sure that the required fields are included in your task or a method not allowed exception will be thrown and the task will not be sent!.

A.    For IC topology:

      • Submit a list of files:

         The required fields are:

         FILE_URLS as a data entry. And the following parameters: 
        • MIME_TYPE : The mime type of the image which will be converted.
        • OUTPUT_MIME_TYPE : The expected mime type of the output image.

        • TASK_SUBMITTER_NAME : The owner of the converted image.

          curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData":{"entry":[{"key":"FILE_URLS","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME1 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME2 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME3 "}]},"parameters":{"entry":[{"key":"TASK_SUBMITTER_NAME","value":"topology_user_test"},{"key":"OUTPUT_MIME_TYPE","value":"image/jp2"},{"key":"MIME_TYPE","value":"image/tiff"}]}}' -i http://cloud.europeana.eu/services/image_convert/tasks

      • Submit a list of datasets:

         DATASET_URLS as a data entry. And the following parameters: 

        • MIME_TYPE : The mime type of the image which will be converted.
        • OUTPUT_MIME_TYPE : The expected mime type of the output image.

        • TASK_SUBMITTER_NAME : The owner of the converted image.

          curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData": {"entry": [{"key": "DATASET_URLS","value": "http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER1/data-sets/dDATASET_NAME1 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER2/data-sets/dDATASET_NAME2 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER3/data-sets/dDATASET_NAME3"}]},"parameters": {"entry": [{"key": "TASK_SUBMITTER_NAME","value": "admin"}, {"key": "OUTPUT_MIME_TYPE", "value": "image/jp2"}, {"key": "MIME_TYPE","value": "image/tiff"}]}}' -i http://cloud.europeana.eu/services/image_convert/tasks



B.    For XSLT topology:

    • Submit a list of files:

       The required fields are:

       FILE_URLS as a data entry. And the following parameters:
      • XSLT_URL: a URL for XSLT file which will be used to transform  XML documents.

      • TASK_SUBMITTER_NAME: The owner of the transformed files. 

        curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData":{"entry":[{"key":"FILE_URLS","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME1 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME2 http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME3"}]},"parameters":{"entry":[{"key":"TASK_SUBMITTER_NAME","value":"topology_user_test"},{"key":"XSLT_URL","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/XSLT_FILE_NAME"}]}}' -i http://cloud.europeana.eu/services/xslt_transform/tasks

    • Submit a list of datasets:

       DATASET_URLS as a data entry. And the following parameters: 

      • XSLT_URL: a URL for XSLT file which will be used to transform  XML documents.

      • TASK_SUBMITTER_NAME: The owner of the transformed files.  

        curl -X POST --user topology_user_test:topology_user_test -H "Content-Type: application/json" -d '{"inputData":{"entry":[{"key":"DATASET_URLS","value":"http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER1/data-sets/dDATASET_NAME1 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER2/data-sets/dDATASET_NAME2 http://cloud.europeana.eu/api/data-providers/DATA_PROVIDER3/data-sets/dDATASET_NAME3"}]},"parameters":{"entry":[{"key":"TASK_SUBMITTER_NAME","value":"topology_user_test"},{"key":"XSLT_URL","value":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/XSLT_FILE_NAME"}]}}' -i http://cloud.europeana.eu/services/xslt_transform/tasks

Notes:

  •  If a task was submitted successfully the sender should recieve a response confirming the creation of the task, like this:  

     

    201 Created
Server: Apache-Coyote/1.1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Location: http://cloud.europeana.eu/services/image_convert/tasks/TASK_ID
Content-Length: 0
Date: creation Date

         The location filed should have a url to the newely created task and at the end of this url a TASK_ID should exist. This taskID should be used to follow the progress of this task.

     

  •       There is a list of optional parameters which could be used in a DPS task :          
    • REPRESENTATION_NAME: If a task designed to process specific representation inside a dataset.

    • NEW_REPRESENTATION_NAME: The name of the representation used to store the processed files.        


3. Reporting: 

To monitor the progress of your tasks.

Reporting is divided into two requests:  

    • Report on a task level:

       curl --user topology_user_test:topology_user_test http://cloud.europeana.eu/services/image_convert/tasks/TASK_ID/progress

      This call should give you information about the status of the submittted task like the status of the task (SENT-PROCESSED-CURRENTLY_PROCESSED,PENDING,DROPPED), total number of files inside a task, the number of the processed files , the exact time stating when task was sent, start processing and finished . An example response :

      {"taskId":TASK_ID,"topologyName":"ic_topology","totalSize":2,"processed":2,"state":"PROCESSED","info":"Completely processed","sent_time":"2016-27-07 09:27:40","start_time":"2016-28-07 09:28:12","finish_time":"2016-28-07 09:28:12"}

           

    • Detailed information about every file in a task. 

       curl --user topology_user_test:topology_user_test http://cloud.europeana.eu/services/image_convert/tasks/TASK_ID/notification

      An example response :

      [{"task_id":TASK_ID,"topology_name":"ic_topology","resource":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME1",

      "state":"SUCCESS","info_text":"","additional_informations":"","result_resource":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/NEW_REPRSENTATION_NAME/versions/NEW_VERSION/files/NEW_FILENAME1"},{"task_id":TASK_ID,"topology_name":"ic_topology","resource":" http://cloud.europeana.eu/api/records/CLOUD_ID/representations/REPRSENTATION_NAME/versions/VERSION/files/FILENAME2",

      "state":"SUCCESS","info_text":"","additional_informations":"","result_resource":"http://cloud.europeana.eu/api/records/CLOUD_ID/representations/NEW_REPRSENTATION_NAME/versions/NEW_VERSION/files/NEW_FILENAME2"}]

 


 

  • No labels