Integrated Data Lake Client for Java¶
Introduction¶
The Integrated Data Lake Java client allows you to perform operations provided by Integrated Data Lake Service.
Further implementation of the Integrated Data Lake SDK library has been shown in a sample project that you can download and test in local or on Insights Hub application. Please refer to this repository: industrial-iot-java-sdk-examples
For more information about the service, refer to Integrated Data Lake Service.
Hint
Placeholders in the following samples are indicated by angular brackets < >
.
Integrated Data Lake Operations¶
Client Name: ObjectOperationsClient
Query objects¶
- Retrieve information of existing objects. In case a folder is specified as input path, all child objects within the folder are returned. Objects in sub-folders of the given folder are not returned. In case a file is specified as input path, only the file is returned. Only the objects of one tenant or subtenant are returned. In case a tenant wants to retrieve information of a subtenant objects, the subtenantId query parameter must be used.
// Construct the ObjectOperationsClient object
ObjectOperationsClient objectOperationsClient = ObjectOperationsClient.builder()
.mindsphereCredentials(<credentials>)
.restClientConfig(<config>)
.build();
QueryObjectsRequest requestObj = new QueryObjectsRequest();
try {
ObjectListResponse response = objectOperationsClient.queryObjects(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete an object¶
- Delete the object located at specified path.
DeleteObjectRequest requestObj = new DeleteObjectRequest();
requestObj.setPath("data/ten=dide2/e2eMetaTag-DONOTDELETE.txt");
try {
objectOperationsClient.deleteObject(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete multiple objects in bulk¶
- Create asynchronous job to delete multiple objects for the specified paths. If any of the given paths doesn't exist then job will return status as COMPLETED_WITH_ERRORS. Jobs which are completed with or without errors would be deleted periodically.Data once deleted cannot be recovered.
CreateDeleteObjectsJobRequest requestObj = new CreateDeleteObjectsJobRequest();
DeleteObjectsJobRequest deleteObjectsJob = new DeleteObjectsJobRequest();
List<DeleteObjectsJobRequestObjects> objects = new ArrayList<DeleteObjectsJobRequestObjects>();
DeleteObjectsJobRequestObjects deleteObjectsJobRequestObjects = new DeleteObjectsJobRequestObjects();
deleteObjectsJobRequestObjects.setPath("data/ten=dide2/e2eMetaTag-DONOTDELETE.txt");
objects.add(deleteObjectsJobRequestObjects);
deleteObjectsJob.setObjects(objects);
requestObj.setDeleteObjectsJob(deleteObjectsJob);
try {
DeleteObjectsJobResponse response =objectOperationsClient.createDeleteObjectsJob(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Fetch details of all the Delete Objects Jobs¶
- Create asynchronous job to delete multiple objects for the specified paths. If any of the given paths doesn't exist then job will return the status as COMPLETED_WITH_ERRORS. Jobs which are completed with or without errors would be deleted periodically.Data once deleted cannot be recovered.
GetAllDeleteObjectsJobRequest requestObj = new GetAllDeleteObjectsJobRequest();
try {
DeleteObjectsJobList response = objectOperationsClient.getAllDeleteObjectsJob(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Fetch status and details of Delete Objects Job¶
- Retrieve information of given Delete Objects Job. Only the objects of one tenant or subtenant are returned. In case a tenant wants to retrieve information on a subtenant's objects, the subtenantId query parameter must be used.
GetDeleteObjectsJobRequest requestObj = new GetDeleteObjectsJobRequest();
requestObj.setId("075aad4f-b50c-423e-a934-b6408306fbfb");
try {
DeleteObjectsJobResponse response = objectOperationsClient.getDeleteObjectsJob(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Fetch errors of Delete Objects Job¶
- Retrieve errors of given Delete Objects Job. This endpoint should be called only when /deleteObjectsJobs/{id} returns the status as COMPLETED_WITH_ERRORS. Only the objects of one tenant or subtenant are returned. In case a tenant wants to retrieve information on a subtenant objects, the subtenantId query parameter must be used.
GetDeleteObjectsJobErrorsRequest requestObj = new GetDeleteObjectsJobErrorsRequest();
requestObj.setId("a615f4eb-3de6-4fda-95b3-76524c5ffdb7");
try {
DeleteObjectsJobErrorDetailsResponse response = objectOperationsClient.getDeleteObjectsJobErrors(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Client Name: ObjectEventSubscriptionOperationsClient
List event subscriptions for the tenant or subtenant¶
- List object event subscriptions for the tenant or subtenant. If requester is tenant, all the subscriptions for the tenant as well as its all subtenants are returned. If requester is a subtenant, all the subscriptions for the subtenant are returned. If tenant wants to filter results for a particular subtenant, filter query parameter subtenantId can be used. This filter query parameter is applicable only if the requester is tenant.
// Construct the ObjectEventSubscriptionOperationsClient object
ObjectEventSubscriptionOperationsClient eventSubscriptionOperationsClient = ObjectEventSubscriptionOperationsClient.builder()
.mindsphereCredentials(<credentials>)
.restClientConfig(<config>)
.build();
QueryObjectEventSubscriptionsRequest requestObj = new QueryObjectEventSubscriptionsRequest();
try {
SubscriptionListResource response = eventSubscriptionOperationsClient.queryObjectEventSubscriptions(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Create object event subscription¶
- Allows users to subscribe for event notifications generated when the objects of a tenant or subtenant are created, updated, or deleted. Multiple subscriptions for the same path can be created when each has a different destination. Similarly, multiple subscriptions for the same destination can be created when each has a different path. Maximum 15 subscriptions can be created for a tenant or for a subtenant. Path in request payload should be upto folders and not upto object e.g. \"myfolder/mysubfolder\". Notification Content Based on the configured subscriptions,event notification messages are published to the destination. The event notification content is formatted in JSON according to this example. If object operation happened in subtenant folder, both tenantId and subtenantId will be part of the message. If object operation happened in tenant folder, only tenantId will be part of the message.
CreateObjectEventSubscriptionRequest requestObj = new CreateObjectEventSubscriptionRequest();
Subscription subscription = new Subscription();
subscription.setDestination("aws-sns://arn:aws:sns:eu-central-1:210761511742:dl-test");
subscription.setPath("myfolder/mysubfolder");
requestObj.setSubscription(subscription);
try {
SubscriptionResponse response = eventSubscriptionOperationsClient.createObjectEventSubscription(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Update object event subscription by id¶
- Update object event subscription.
PatchObjectEventSubscriptionRequest requestObj = new PatchObjectEventSubscriptionRequest();
requestObj.setId("1d8d741f-645f-4b9f-a913-0feaee4381d0");
requestObj.setIfMatch(2);
SubscriptionUpdate subscription = new SubscriptionUpdate();
subscription.setDestination("aws-sns://arn:aws:sns:eu-central-1:210761511742:dl-test");
subscription.setPath("myfolder/mysubfolder");
requestObj.setSubscription(subscription);
try {
SubscriptionResponse response = eventSubscriptionOperationsClient.patchObjectEventSubscription(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Read object event subscription by id¶
- Read object event subscription for the tenant.
RetrieveObjectEventSubscriptionRequest requestObj = new RetrieveObjectEventSubscriptionRequest();
try {
SubscriptionResponse response = eventSubscriptionOperationsClient.retrieveObjectEventSubscription(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete object event subscription by id¶
- Delete object event subscription (Unsubscribe).
DeleteObjectEventSubscriptionRequest requestObj = new DeleteObjectEventSubscriptionRequest();
requestObj.setId("1d8d741f-645f-4b9f-a913-0feaee4381d0");
requestObj.setIfMatch(3);
try {
eventSubscriptionOperationsClient.deleteObjectEventSubscription(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Client Name: ObjectAccessOperationsClient
Generate signed URLs to upload an object¶
- Generate signed URLs to upload one or more objects.
- Signed URL response for upload
// Construct the ObjectOperationsClient object
ObjectAccessOperationsClient operationsClient = ObjectAccessOperationsClient.builder()
.mindsphereCredentials(<credentials>)
.restClientConfig(<config>)
.build();
GenerateUploadObjectUrlsRequest requestObj = new GenerateUploadObjectUrlsRequest();
GenerateUrlPayload generateUrlPayload = new GenerateUrlPayload();
Path path = new Path();
path.setPath("myfolder/mysubfolder/myobject.objext");
Paths paths = new Paths();
paths.add(path);
generateUrlPayload.setPaths(paths);
requestObj.setGenerateUrlPayload(generateUrlPayload);
try {
SignedUrlResponse response = operationsClient.generateUploadObjectUrls(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Generate signed URLs to download an object¶
- Generate signed URLs to download single object.
- Generate signed URLs to download one or more objects.
- Signed URL response for download
GenerateDownloadObjectUrlsRequest requestObj = new GenerateDownloadObjectUrlsRequest();
GenerateUrlPayload generateUrlPayload = new GenerateUrlPayload();
Path path = new Path();
path.setPath("myfolder/mysubfolder/myobject.objext");
Paths paths = new Paths();
paths.add(path);
generateUrlPayload.setPaths(paths);
requestObj.setGenerateUrlPayload(generateUrlPayload);
try {
SignedUrlResponse response = operationsClient.generateDownloadObjectUrls(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Get a list of cross accounts¶
- Get list of cross accounts on which access was given.
- If requester is tenant, all the cross account for the tenant as well as its all subtenants are returned.
- If requester is a subtenant, all the cross accounts for the subtenant are returned.
- If tenant wants to filter results for a particular subtenant, filter query parameter subtenantId can be used. This filter query parameter is applicable only if the requester is tenant.
ListCrossAccountRequest requestObj = new ListCrossAccountRequest();
try {
CrossAccountListResource response = operationsClient.listCrossAccount(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Create a cross account¶
- Create a cross account on which access needs to be given for paths.
- If, in request body subtenant is denoted as , it is a special cross account. In this case, on this cross account, tenant implicitly gets Read Only access to the storage account's root level data path for itself and for all its subtenants. For this case, the authorization token should be that of the tenant.
CreateCrossAccountRequest requestObj = new CreateCrossAccountRequest();
CrossAccountRequest crossAccountRequest = new CrossAccountRequest();
crossAccountRequest.setAccessorAccountId("565037524705");
crossAccountRequest.setName("UI Account");
crossAccountRequest.setDescription("Test Accontttt");
requestObj.setCrossAccountRequest(crossAccountRequest);
try {
CrossAccount response = operationsClient.createCrossAccount(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Get details of selected cross account¶
- Get details of selected cross account.
GetCrossAccountRequest requestObj = new GetCrossAccountRequest();
requestObj.setId("d5395d2b-524e-4a28-b100-2138b6f078f4");
try {
CrossAccount response = operationsClient.getCrossAccount(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Update a cross account¶
- Update a cross account on which access needs to be managed.
UpdateCrossAccountRequest requestObj = new UpdateCrossAccountRequest();
CrossAccountUpdateRequest crossAccountUpdateRequest = new CrossAccountUpdateRequest();
crossAccountUpdateRequest.setName("abc");
crossAccountUpdateRequest.setDescription("rgreg");
requestObj.setCrossAccountRequest(crossAccountUpdateRequest);
requestObj.setId("f2904dfa-6784-423a-8573-41a77219825e");
requestObj.setIfMatch(3);
try {
CrossAccount response = operationsClient.updateCrossAccount(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete cross account¶
- Delete cross account and corresponding accesses.
DeleteCrossAccountRequest requestObj = new DeleteCrossAccountRequest();
requestObj.setId("e628bd56-8b46-4a01-bc82-82aa9ec2a880");
requestObj.setIfMatch(2);
try {
operationsClient.deleteCrossAccount(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Get a list of cross account accesses¶
This section shows two options for achieving the following behavior:
- Get list of cross account accesses.
ListCrossAccountAccessRequest requestObj = new ListCrossAccountAccessRequest();
requestObj.setId("8a1dce46-30db-4f80-ab78-9740dcdb6386");
try {
CrossAccountAccessListResource response = operationsClient.listCrossAccountAccess(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Create a cross account access¶
- Create a cross account access. Create a cross account access. If the cross account is created for tenant and all subtenants (**), then this operation is implicitly handled and not allowed through API. Maximum 5 cross account accesses can be created with ENABLED status. Maximum 10 cross accounts can be created with DISABLED status.
CreateCrossAccountAccessRequest requestObj = new CreateCrossAccountAccessRequest();
requestObj.setId("8a1dce46-30db-4f80-ab78-9740dcdb6386");
CrossAccountAccessRequest crossAccountAccessRequest = new CrossAccountAccessRequest();
crossAccountAccessRequest.setDescription("stringefewfup");
crossAccountAccessRequest.setPath("myfolder/mysubfolder");
crossAccountAccessRequest.setPermission(Permission.DELETE);
crossAccountAccessRequest.setStatus(StatusEnum.DISABLED);
requestObj.setCrossAccountAccessRequest(crossAccountAccessRequest);
try {
CrossAccountAccess response = operationsClient.createCrossAccountAccess(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Get a cross account access¶
- Get a selected access for a selected cross account.
GetCrossAccountAccessRequest requestObj = new GetCrossAccountAccessRequest();
requestObj.setId("8a1dce46-30db-4f80-ab78-9740dcdb6386");
requestObj.setAccessId("c24e4208-ff7c-4909-aee7-214dba800446");
try {
CrossAccountAccess response = operationsClient.getCrossAccountAccess(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Update a cross account access¶
- Update a cross account access. This operation is not allowed when the cross account is created for special case of tenant and all subtenants (**). Maximum 5 cross account accesses can be present with ENABLED status. Maximum 10 cross accounts can be present with DISABLED status.
UpdateCrossAccountAccessRequest requestObj = new UpdateCrossAccountAccessRequest();
requestObj.setId("2ad2371a-6ce9-4474-bb4a-25cfe1f39da3");
requestObj.setAccessId("b45993e8-6984-48ac-a0fc-953ec53a23e7");
CrossAccountAccessRequest crossAccountAccessRequest = new CrossAccountAccessRequest();
crossAccountAccessRequest.setDescription("update");
crossAccountAccessRequest.setPath("myfolder/mysubfolder");
crossAccountAccessRequest.setPermission(Permission.READ);
crossAccountAccessRequest.setStatus(StatusEnum.DISABLED);
requestObj.setCrossAccountAccessRequest(crossAccountAccessRequest);
requestObj.setIfMatch(10);
try {
CrossAccountAccess response = operationsClient.updateCrossAccountAccess(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete a cross account access¶
If the cross account is created for tenant and all subtenants (**), then this operation is implicitly handled by deletion of cross account and not allowed through API.
DeleteCrossAccountAccessRequest requestObj = new DeleteCrossAccountAccessRequest();
requestObj.setId("8a1dce46-30db-4f80-ab78-9740dcdb6386");
requestObj.setAccessId("883b6eeb-ea39-4967-b288-0658400906e6");
requestObj.setIfMatch(1);
try {
operationsClient.deleteCrossAccountAccess(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Client Name: ObjectOperationsWithAccessTokenClient
List all folders having write permission¶
This API can be accessed by tenantadmin to list all write permission folders including that of subtenants. Subtenant can access this API to get list write permission folders owned by subtenant. Size parameter value should not be more than 1000.
// Construct the ObjectOperationsWithAccessTokenClient object
ObjectOperationsWithAccessTokenClient accessTokenClient = ObjectOperationsWithAccessTokenClient.builder()
.mindsphereCredentials(<credentials>)
.restClientConfig(<config>)
.build();
ListAccessTokenPermissionsRequest requestObj = new ListAccessTokenPermissionsRequest();
try {
AccessTokenPermissionResources response = accessTokenClient.listAccessTokenPermissions(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Allows to give write permission on folder/path¶
- Allows users (tenant) to give write permission on folder/path. This API can only be accessed by tenantadmin. Tenantadmin can also provide write permission on subtenant folder. Write permission on root folder can also be given, except TSI folder
AccessTokenPermissionsRequest requestObj = new AccessTokenPermissionsRequest();
AccessTokenPermissionRequest accessTokenPermissionRequest = new AccessTokenPermissionRequest();
accessTokenPermissionRequest.setPath("customFolder");
ccessTokenPermissionRequest.setPermission(AccessPermission.WRITE);
requestObj.setWritePathPayload(accessTokenPermissionRequest);
try {
AccessTokenPermissionResource response = accessTokenClient.accessTokenPermissions(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Details of the write folder request for the given id¶
This API can be accessed by tenantadmin to get details of the request including that for subtenants. Subtenant can access this API, to get details of the request belongs to their write folder.
GetAccessTokenPermissionsRequest requestObj = new GetAccessTokenPermissionsRequest();
requestObj.setId("672eaace-60fd-4b90-865d-795b981e5ac8");
try {
AccessTokenPermissionResource response = accessTokenClient.getAccessTokenPermissions(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Generate AWS STS token¶
- Allows the users to request temporary, limited-privilege AWS credentials to get read-only or write-only access on the URI returned in the response.
- Read permission will always be on the root level.
- Path field is optional for READ permission. If value for path is not provided then it will be considered on root level
- Ensure to enable write access on the path before requesting token with write permission.
- Write access can be enabled using POST /accessTokenPermissions endpoint.
- An access token requested for a given path also automatically gives access to all subpaths of the path. For example, if an access token is requested for path /a and there are subpaths /a/b and /a/b/c, the token allows to access those too.
- An access token with write permissions can only be requested for the paths defined by resource accessTokenPermissions. An access token with read permissions can only be requested for the root path /.
GenerateAccessTokenRequest requestObj = new GenerateAccessTokenRequest();
GenerateSTSPayload generateSTSPayload = new GenerateSTSPayload();
requestObj.setStsPayload(generateSTSPayload);
try {
AccessTokens response = accessTokenClient.generateAccessToken(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete write permission on folder for the given id¶
This API can be accessed by the tenantadmin only.
DeleteAccessTokenPermissionsRequest requestObj = new DeleteAccessTokenPermissionsRequest();
requestObj.setId("4869c9ab-3430-471c-aaec-901d16d0a1a1");
try {
accessTokenClient.deleteAccessTokenPermissions(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Client Name: TimeSeriesBulkImportClient
Query all time series bulk import jobs¶
- Query all time series bulk import jobs currently existing, which are owned by the client tenant or subtenant. If requester is tenant, all the import jobs for the tenant as well as its all subtenants are returned. If requester is a subtenant, all the iport jobs for the subtenant are returned. If tenant wants to filter results for a particular subtenant, filter query parameter subtenantId can be used. This filter query parameter is applicable only if the requester is tenant.
// Construct the TimeSeriesBulkImportClient object
TimeSeriesBulkImportClient accessTokenClient = TimeSeriesBulkImportClient.builder()
.mindsphereCredentials(<credentials>)
.restClientConfig(<config>)
.build();
QueryTimeSeriesImportJobsRequest timeSeriesBulkImportClient = new QueryTimeSeriesImportJobsRequest();
try {
ImportJobListResource response = timeSeriesBulkImportClient.queryTimeSeriesImportJobs(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Create a bulk import job of time series data into data lake¶
The import takes into account time series data from the provided aspects associated to the provided assets, in the given time range.
CreateTimeSeriesImportJobRequest requestObj = new CreateTimeSeriesImportJobRequest();
ImportJobRequest importJobRequest = new ImportJobRequest();
importJobRequest.setName("string5");
importJobRequest.setDestination("myfolder/mysubfolder");
importJobRequest.setFrom("2017-12-11T13:36:00.000Z");
importJobRequest.setTo("2018-01-11T13:36:00.000Z");
List<String> aspectNames = new ArrayList<String>();
List<String> assetIds = new ArrayList<String>();
aspectNames.add("string");
assetIds.add("string");
importJobRequest.setAspectNames(aspectNames);
importJobRequest.setAssetIds(assetIds);
requestObj.setImportJob(importJobRequest);
try {
ImportJobResponse response = timeSeriesBulkImportClient.createTimeSeriesImportJob(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Retrieve details of a time series bulk import job¶
The details are only available once a job is not any longer in status PENDING.
RetrieveTimeSeriesImportJobDetailsRequest requestObj = new RetrieveTimeSeriesImportJobDetailsRequest();
requestObj.setId("2faf09913995428b90997f834da2f542");
try {
ImportJobDetails response = timeSeriesBulkImportClient.retrieveTimeSeriesImportJobDetails(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Delete time series bulk import job by id¶
Deletes the completed time series bulk import job.
DeleteTimeSeriesImportJobRequest requestObj = new DeleteTimeSeriesImportJobRequest();
requestObj.setId("9842e5ebf37943328d97c33b6814a821");
try {
timeSeriesBulkImportClient.deleteTimeSeriesImportJob(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Retrieve status of time series bulk import job¶
Retrieves the status of time series bulk import job.
RetrieveTimeSeriesImportJobRequest requestObj = new RetrieveTimeSeriesImportJobRequest();
requestObj.setId("2faf09913995428b90997f834da2f542");
try {
ImportJobResponse response = timeSeriesBulkImportClient.retrieveTimeSeriesImportJob(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Client Name: ObjectsMetadataCatalogOperationsClient
Create/Update Metadata for the object¶
- Create/Update metadata. If tenant is operating for a subtenant object, he should specify subtenant in request query parameter. If path contains special characters which require URL encoding, it should be done as appropriate. Maximum 8 tags are allowed, where each tag can be maximum 128 characters.
// Construct the ObjectOperationsClient object
ObjectsMetadataCatalogOperationsClient oCatalogOperationsClient = ObjectsMetadataCatalogOperationsClient.builder()
.mindsphereCredentials(<credentials>)
.restClientConfig(<config>)
.build();
CreateOrUpdateObjectMetadataRequest requestObj = new CreateOrUpdateObjectMetadataRequest();
requestObj.setPath("myfolder/mysubfolder/myobject.objext");
Metadata metadata = new Metadata();
List<String> tags = new ArrayList<String>();
tags.add("tag1");
tags.add("tag2");
metadata.setTags(tags);
requestObj.setMetadata(metadata);
try {
oCatalogOperationsClient.createOrUpdateObjectMetadata(requestObj);
} catch (MindsphereException e) {
// Exception handling
}
Get Metadata for the object¶
If tenant is operating for a subtenant object, he should specify subtenant in request query parameter. If path contains special characters which require URL encoding, it should be done as appropriate.
RetrieveObjectMetadataRequest requestObj = new RetrieveObjectMetadataRequest();
requestObj.setPath("myfolder/mysubfolder/myobject.objext");
try {
ObjectMetaDataResponse response = oCatalogOperationsClient.retrieveObjectMetadata(requestObj);
} catch (MindsphereException e) {
// Exception handling
}