Configuration of Events¶

You can choose to be notified for object events (add, update or delete) for your space in Data Lake. It allows you to subscribe for event notifications generated when the objects of a environment or subtenant are created, updated or deleted.

Multiple subscriptions for the same path can be created when each has a different destination. Similarly, multiple subscriptions for the same destination can be created when each has a different path.

Maximum 15 subscriptions can be created for a environment or subtenant. Path in request payload should be upto folders and not upto object e.g. “myfolder/mysubfolder”

To create Event Subscription - Once this step is performed, IDL will subscribe to the intended Simple Notification Service (SNS) topic.


POST /objectEventSubscriptions

Content-Type: application/json

Request example:

AWSAzureAliyun


  {
        "path": "myfolder/mysubfolder",
        "destination": "aws-sns://arn:aws:sns:region:account-id:topicname",
        "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4"
    }


  {
      "path": "/notification/nfr",
      "destination": "Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=nfr;SharedAccessKey=0LFL2eOdFJRCWDo6bDjtapEMq0XXX45dywFEChfhj4E;EntityPath=idl-notification-nfr-topic"
    }


  {
        "path": "myfolder/mysubfolder",
        "destination": "acs:mns:region:account-id:/topics/topicname:AccessKeyId:AccessKeySecret",
        "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4"
    }

Response example:

AWSAzureAliyun


    {
        "id": "7dc53df5703e49b38670b1c468f47f1f",
        "storageAccount": "dlbucketname",
        "storagePath": "data/sub=subtenantId/myfolder/mysubfolder",
        "destination": "aws-sns://arn:aws:sns:region:account-id:topicname",
        "eTag": 1,
        "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4"
    }


  {
    "id": "7dc53df5703e49b38670b1c468f47f1f"
    "destination": "Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=idl-policy;SharedAccessKey=
    0LFL2eOdFJRCWDo6bDjtapEMq0XXX45dywFEChfhj4E=;EntityPath=idl-subscription-topic",
    "storageAccount": "datalake-rc-punazdl",
    "storagePath": "data/ten=punazdl/notification/nfr","eTag": 1
  }


  {
    "id": "163acfbe-9fb7-4081-933a-8bab41d96a4c",
    "destination": "acs:mns:cn-shanghai:1708933267332881:/topics/topicname:LTAI5tKvZSuRQ9KdJ2H8GGWt:bOBg5K2CpXLdPkdabpTCuETlCc8ere",
    "storageAccount": "datalake-integ-cdiot0-1627437476734",
    "storagePath": "data/ten=cdiot0/myFolder/mySubFolder1",
    "status": "ACTIVE",
    "name": "Subscription-6275",
    "eTag": 1
  }

Get the details of events subscribed¶

You can get the details of SNS topics to which you are already subscribed. It will list object event subscriptions for the environment or subtenant. If requester is environment, all the subscriptions for the environment as well as its all subtenants are returned. If requester is a subtenant, all the subscriptions for the subtenant are returned. If environment wants to filter results for a particular subtenant, filter query parameter subtenantId can be used. This filter query parameter is applicable only if the requester is environment.


GET /objectEventSubscriptions/

Response example:

AWSAzureAliyun


{
    "id": "7dc53df5703e49b38670b1c468f47f1f",
    "storageAccount": "dlbucketname",
    "storagePath": "data/sub=subtenantId/myfolder/mysubfolder",
    "destination": "aws-sns://arn:aws:sns:region:account-id:topicname",
    "eTag": 1,
    "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4"
}


{
    "subscriptions": [
        {
            "id": "d27c4973-35fc-46a8-bc49-7335fcc6a8d2",
            "destination": "Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=idl-policy;
            SharedAccessKey=0LFL2eOdFJRCWDo6bSkoapEMq0HUK23dywFEChfhj4E=;EntityPath=idl-subscription-topic",
            "storageAccount": "datalake-rc-punazdl",
            "storagePath": "data/ten=punazdl/EndToEndTest",
            "eTag": 1
        },
        {
            "id": "e6450b8f-1e22-4d8c-850b-8f1e22cd8c74",
            "destination": "Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=idl-policy;SharedAccessKey=0LFL2eOdFJRCWDo6bDjtapEMq0XXX45dywFEChfhj4E=;EntityPath=idl-subscription-topic",
            "storageAccount": "datalake-rc-punazdl",
            "storagePath": "data/sub=7d598ccff0d82fa826c1b8ce9ce88a4a/EndToEndTest",
            "subtenantId": "7d598ccff0d82fa826c1b8ce9ce88a4a",
            "eTag": 1
        },
        {
            "id": "e17fde2e-b752-4e6f-bfde-2eb7524e6fce",
            "destination": "Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=idl-policy;
            SharedAccessKey=0LFL2eOdFJRCWDo6bDjtapEMq0XXX45dywFEChfhj4E=;EntityPath=idl-subscription-topic",
            "storageAccount": "datalake-rc-punazdl",
            "storagePath": "data/ten=punazdl/notification/nfr",
            "eTag": 1
        }
    ],
    "page": {
        "size": 10,
        "totalElements": 3,
        "totalPages": 1,
        "number": 0
    }
}


  {
    "id": "163acfbe-9fb7-4081-933a-8bab41d96a4c",
    "destination": "acs:mns:cn-shanghai:1708933267332881:/topics/topicname:LTAI5tKvZSuRQ9KdJ2H8GGWt:bOBg5K2CpXLdPkdabpTCuETlCc8ere",
    "storageAccount": "datalake-integ-cdiot0-1627437476734",
    "storagePath": "data/ten=cdiot0/myFolder/mySubFolder1",
    "status": "ACTIVE",
    "name": "Subscription-6275",
    "eTag": 1
  }

Edit created subscription¶

You can also edit the created subscription.


PATCH /objectEventSubscriptions/7dc53df5703e49b38670b1c468f47f1f

Content-Type: application/json

Request example:

AWSAzureAliyun


    {
        "path": "myCustomPath",
        "destination": "aws-sns://arn:aws:sns:region:account-id:topicname"
    }


    {
        "path": "myCustomPath",
        "destination": "aws-sns://arn:aws:sns:region:account-id:topicname"
    }


    {
        "path": "myCustomPath",
        "destination": "acs:mns:region:account-id:/topics/topicname:AccessKeyId:AccessKeySecret"
    }

Response example:

AWSAzureAliyun


    {
        "id": "7dc53df5703e49b38670b1c468f47f1f",
        "storageAccount": "dlbucketname",
        "storagePath": "data/sub=subtenantId/myfolder/mysubfolder",
        "destination": "aws-sns://arn:aws:sns:region:account-id:topicname",
        "eTag": 1,
        "subtenantId": "204a896c-a23a-11e9-a2a3-2a2ae2dbcce4"
    }


{
    "id": "7dc53df5703e49b38670b1c468f47f1f",
    "destination": "Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=idl-policy; SharedAccessKey=0LFL2eOdFJRCWDo6bDjtapEMq0XXX45dywFEChfhj4E=;EntityPath=idl-subscription-topic",
    "storageAccount": "datalake-rc-punazdl",
    "storagePath": "data/ten=punazdl/notification/nfr",
    "eTag": 2
}


    {
      "id": "163acfbe-9fb7-4081-933a-8bab41d96a4c",
      "destination": "acs:mns:cn-shanghai:1708933267332881:/topics/topicname:LTAI5tKvZSuRQ9KdJ2H8GGWt:bOBg5K2CpXLdPkdabpTCuETlCc8ere",
      "storageAccount": "datalake-integ-cdiot0-1627437476734",
      "storagePath": "data/ten=cdiot0/myFolder/mySubFolder1",
      "status": "ACTIVE",
      "name": "Subscription-6275",
      "eTag": 1
    }

Delete the subscription¶

You can delete the created subscription if you do not want any further notifications.


DELETE /objectEventSubscriptions/7dc53df5703e49b38670b1c468f47f1f

Response example:


Status 204 No Content

Get the details of events for the given id¶

You can get the details by ID of SNS topics to which you are already subscribed.


GET /objectEventSubscriptions/7dc53df5703e49b38670b1c468f47f1f

Response Example:

Azure


{
  "id": "7dc53df5703e49b38670b1c468f47f1f",
  "destination":"Endpoint=sb://iot-sbus-rc.servicebus.windows.net/;SharedAccessKeyName=idl-policy;SharedAccessKey=0LFL2eOdFJRCWDo6bDjtapEMq0XXX45dywFEChfhj4E=;EntityPath=idl-subscription-topic",
  "storageAccount": "datalake-rc-punazdl",
  "storagePath": "data/ten=punazdl/notification/nfr",
  "eTag": 2
}

Send event notification¶

To enable Integrated Data Lake Service to send event notifications on provided SNS topic, the required permissions SNS:GetTopicAttributes and SNS:Publish should be granted.

The following examples explain the functionality:

Allowing access specifically to Integrated Data Lake Service’s AWS account (Recommended approach):

Replace "<IDL AWS Account Id>" with Integrated Data Lake Service’s AWS account id, which is 463822690055
Replace "<SNS Topic ARN>" with arn of the SNS topic subscribing to Integrated Data Lake Service


  {
    "Version": "2008-10-17",
    "Id": "<Policy ID>",
    "Statement": [
      {
        "Sid":  "AllowAccessToIDL",
        "Effect": "Allow",
        "Principal": {
          "AWS": "<IDL AWS Account Id>"
        },
        "Action": [
          "SNS:GetTopicAttributes",
          "SNS:Publish"
        ],
        "Resource": "<SNS Topic ARN>"
      }
    ]
  }

Allowing access specifically to all AWS accounts (Not Recommended as it allows every AWS account to access SNS topic) replace "<SNS Topic ARN>" with arn of the SNS topic subscribing to Integrated Data Lake Service.


    {
      "Version": "2008-10-17",
      "Id": "<Policy ID>",
      "Statement": [
        {
          "Sid":  "AllowAccessToAll",
          "Effect": "Allow",
          "Principal": {
            "AWS": "*"
          },
          "Action": [
            "SNS:GetTopicAttributes",
            "SNS:Publish"
          ],
          "Resource": "<SNS Topic ARN>"
        }
      ]
    }

Configuration of Events¶

Subscribe to the event notification¶

Get the details of events subscribed¶

Edit created subscription¶

Delete the subscription¶

Get the details of events for the given id¶

Send event notification¶