Data Exchange Service¶
Idea¶
The Data Exchange (DE) Service allows customers to upload and download data for any purpose. It is intended for both regular users and for tools that upload data into Insights Hub.
Access¶
The Data Exchange provides its operations as a REST API. Uploading, downloading, organizing and listing folder structure can be done using API calls.
For accessing this service you need to have one of the roles listed in Data Exchange roles and scopes.
Basics¶
Customers and other services can use DE as a file storage and management provider. Note that when receiving requests through MindGate, MindGate applies its own limits in addition. This includes the timeouts and request sizes. Currently the maximum supported file size is 100 MB.
DE does not cache, analyze or scans the contents of the transferred bytes for viruses. It uses the Spring Cloud Resource Manager along with streaming mechanisms that provide multithreaded support for working with S3 storage. This type of backend communication was implemented to ensure maximum performance under minimum resource load.
Visibility spaces¶
By default, when uploading a file, the newly uploaded file inherits its parent's visibility. It can either be public or private. Public visibility means that the file is visible in tenant space. Private visibility means that the file is visible in user space only, meaning that only the user who has uploaded it can download, delete or update it.
Public space¶
Public space is accessible by all users within the tenant. Files and directories are made public, if they are uploaded or created in public directories. If the file or directory needs to be placed in root, i.e., they have no parent, the parentId attribute must be set to _PUBLIC_ROOT_ID.
Private space¶
Private space is accessible by the respective author only. Files and directories are made private, if they are uploaded or created in private directories. If the file or directory needs to be placed in root, i.e., they have no parent, the parentId attribute need to be set to _PRIVATE_ROOT_ID.
Features¶
The Data Exchange Service exposes its API for realizing the following tasks:
- Secure upload testbed or production data
- Download data from Insights Hub that you have previously uploaded
- Download results produced by other Industrial IoT services or apps
- Have a performant long term storage for your temporary or permanent data, for both users and services
- Organize files in folders
- Store confidential files at user level, making them inaccessible for other users of your tenant
Restrictions¶
- File size limits are internally limited to S3 ones, but when using DataExchange via MindGate, MindGate limits apply
- File and directory names are limited to 1024 characters
- Number of objects within a parent is limited to 10,000 items
- Recursive listing of directory contents is not provided
- All environment have 100 GB storage allocated by default irrespective of the offering
API Rate Limits for P&P Tenant¶
Data Exchange has imposed technical limits for P&P tenant to safeguard the system and to avoid system exploitation on heavy load exceeding system limits.API rate limits for Data Exchange are applicable as technical rate limits.
Example Scenario¶
A developer wants to train a model for anomaly detection using the Anomaly Detection Service. The developer uses the Data Exchange Service to upload a training set into Insights Hub. This data can then be used by the Anomaly Detection Service.