IoT File Service¶
Idea¶
The IoT File Service provides the user with file management for files related to assets. The service provides an interface to manage data, search files and download them to specified agents. Files can also be uploaded to a temporary storage and managed there.
Access¶
For accessing this service, you need to have the respective roles listed in IoT File Service roles and scopes.
A user can only interact with files within their environment and subtenants.
For accessing the Secure Data Sharing (SDS) protected APIs, you need to have appropriate Policy Definitions in place. For the list of supported APIs and Required Actions, refer here.
Basics¶
The service implements CRUD (create, read, update and delete) functionality and searching for files.
Each file is linked to an asset. An asset can have multiple files attached. The asset ID is required for each request.
Each file has a set of standard properties as metadata:
- File ID
- File name
- File path
- File type (e.g., image)
- File size
- Description
- Timestamps (created, updated)
The content of these files is not parsed by Insights Hub and it requires custom applications or analytical tools to interpret and visualize the data.
The maximum request payload that can be stored or retrieved depends on the client's network speed. The current timeout is one minute, and any file that takes longer than one minute to read or write results in an error 502 "Bad Gateway". If a file is too large to upload in one minute, use the multi part upload feature.
Supported Use Cases¶
Typical use cases that are supported within File Service are as follows:
- Storing static data related to Assets (configuration files, images, specifications, etc.)
- Supports low frequency operations on the data (once in a day, weekly or even monthly)
- Temporary storage of data to explore real time data storage in Time Series
Note
Avoid the use of File Service for high frequency and continuous read or write, analytical workflows and storing high volume real-time data (Time Series data).
Versioning¶
The service does not provide explicit versioning. If different versions of a file need to be stored, put a version identifier in the file name to make it unique.
File Deletion¶
Files are always deleted physically. There is no logical delete. Files of an asset are automatically deleted if the asset is deleted physically.
Files can be deleted from the storage in the following ways:
| Delete Mechanism | Description |
|---|---|
| Single File Delete | Delete a file for the specified asset (entity) and path (actual file + metadata). This is a synchronous call. |
| Bulk Delete | Delete files in bulk (actual files + metadata). This is an asynchronous activity in which files are deleted from storage within a day. There is an API available for bulk deletion using Time To (Live) rule. Users are required to create a Time To Live (TTL) rule to initiate bulk file deletion. The TTL rule specifies the duration after which files are automatically deleted by the system. This rule operates on a daily basis, identifying and removing files that meet the criteria outlined in the rule. |
| Asset Delete | If the underlying asset is deleted, the files related to that asset are also deleted from the storage. File deletion is triggered on asset delete notification. This is also an asynchronous activity. |
File Search¶
Files can be queried using filtering and sorting operations. For more information, refer to the Filtering documentation.
Features¶
The IoT File Service exposes its API for realizing the following tasks:
- Create a new file
- Update (overwrite) an existing file
- Read a file
- Delete a file
- Search for files by metadata
- Count and order files by metadata
- Multi part upload and download
- Bulk delete of files for an asset
Limitations¶
- For security reasons, a strict timeout is enforced. Currently, this leads to the situation that the size of a file which can be uploaded/ downloaded depends on the network bandwidth at client side.
- When uploading files to the File Service, a file name cannot contain blank spaces. Blank spaces in the file name will prevent the file form being stored.
- After deleting an asset, associated files might still be accessible for up to 30 minutes as this data is deleted asynchronously.
- After deleting a file, it will take some time (up to 60 - 120 seconds) to create the file with the same name again depending on the data stored in storage.
- After creating or deleting the file, it will take some time (up to 3 seconds) to get the file in "search file" API.
- In File Service, a maximum of 100 characters are allowed for a file name.
- Using special characters like '*' or '?' can delay the search and might result in slow search performance.
All requests pass through Industrial IoT Gateway and must adhere to the Industrial IoT Gateway Restrictions.
Usage Quota and Limits¶
IoT File Service APIs are restricted with following limitations:
| API Endpoint | Quota Parameter | XXS (S4F) | XS (Offering ID5302) | S (Offering ID5303) | M (Offering ID5304) | L (Offering ID5305) | XL (Offering ID5306) |
|---|---|---|---|---|---|---|---|
| PUT (Short Window) | core_iotfile_v3_files--_put_30s | 6 | 8 | 10 | 12 | 14 | 16 |
| PUT (Long Window) | core_iotfile_v3_files--_put_600s | 10 | 100 | 140 | 180 | 220 | 280 |
| LIST (Short Window) | core_iotfile_v3_files--_get_30s | 6 | 10 | 12 | 15 | 18 | 20 |
| LIST (Long Window) | core_iotfile_v3_files--_get_600s | 30 | 210 | 240 | 270 | 300 | 350 |
| GET (Short Window) | core_iotfile_v3_files-~_get_30s | 6 | 12 | 14 | 18 | 22 | 25 |
| GET (Long Window) | core_iotfile_v3_files-~_get_600s | 30 | 200 | 250 | 300 | 400 | 500 |
| DELETE (Short Window) | core_iotfile_v3_files--_delete_30s | 1 | 1 | 2 | 3 | 4 | 5 |
| DELETE (Long Window) | core_iotfile_v3_files--_delete_600s | 3 | 10 | 12 | 15 | 20 | 25 |
| POST Bulk Delete (Short Window) | core_iotfile_v3_filesdeletejobs_post_6h | 0 | 1 | 2 | 3 | 4 | 5 |
| POST Bulk Delete (Long Window) | core_iotfile_v3_filesdeletejobs_post_24h | 0 | 2 | 3 | 4 | 6 | 8 |
| Get Bulk Delete Jobs (Short Window) | core_iotfile_v3_files-deletejobs-~_get_30s | 0 | 1 | 2 | 3 | 4 | 5 |
| Get Bulk Delete Jobs (Long Window) | core_iotfile_v3_files-deletejobs-~_get_600s | 0 | 8 | 9 | 10 | 12 | 15 |
| Get Bulk Delete Job Detail (Short Window) | core_iotfile_v3_filesdeletejobs_get_30s | 0 | 1 | 2 | 3 | 4 | 5 |
| Get Bulk Delete Job Detail (Long Window) | core_iotfile_v3_filesdeletejobs_get_600s | 0 | 8 | 9 | 10 | 12 | 15 |
Note
Rate Limits can cause discontinuity in the use case flow. Ensure to check the rate limits before implementing File Service APIs. Also, check the alternate storage mechanisms within Insights Hub for advanced use cases.
Example Scenario¶
The application developer of a brewery wants to store files associated to assets in order to provide additional meta information (e.g., images, technical descriptions, manuals, etc.).
The developer uses the IoT File Service to upload files to an asset.