IoT Time Series Bulk Service¶

Idea¶

The IoT Time Series Bulk Service is a variant to the IoT Time Series Service. It is designed for importing big amounts of data into Insights Hub in bulk.

Access¶

For accessing this service you need to have the respective roles listed in IoT Time Series Service roles and scopes.

For accessing the Secure Data Sharing (SDS) protected APIs you need to have appropriate Policy Definitions in place. Please refer here for the list of supported APIs and Required Actions.

Basics¶

In contrast to the IoT Time Series Service, this service ingests data in batch mode and can return more than 2,000 items per response when reading the stored data. It returns as many items as possible until size and time limitations on the Industrial IoT Gateway are exceeded. If it cannot return all requested data in a single response, a link for requesting the next data set is added to the response body. The URL of this link is based on the already returned data and the original request: The start time is set to the timestamp of the last returned record and the maximum number of records is reduced by the number of already returned records.

The IoT Time Series Bulk Service accepts up to 100 consecutive files per import job.

For performance asset, all the timestamps should belong to the same day (e.g., FROM '2021-02-18T00:00:00.001Z' TO '2021-02-19T00:00:00.000Z'). For Bulk Ingest of time series, only time stamps in ISO format are supported.

Following practices are suggested for best results for Performance Assets while using IoT Timeseries Bulk Ingest Service.
1. Combine complete data for Performance Asset in single/multiple file for any Asset, Aspect and upload it in single job for bulk processing.
2. Include as much data in single job.
3. Bulk Ingest API is designed to upload historical data only unless data size of an individual job is in few MB's.
4. Small job sizes (Less than ~ 20KB) should be avoided to avail optimal performance.
5. Bulk Ingest jobs will be processed Asynchronously.

Features¶

The IoT Time Series Bulk Service exposes its API for realizing the following tasks:

• Import time series data for performance assets in bulk
• Based on Timeseries data density, the maximum number of record values for the specified time range are obtained within the 1 minute overall request time limit. It is recommended to use this while reading bulk data which can withhold response time up to 1 minute.

Usage Quota and Limits¶

The usage quota and limits details for the IoT timeseries services is mentioned in API rate limits

Limitations¶

• Currently, only the IoT File Service is supported as data source.
• Only JSON file format is supported as input file.

• The maximum timerange for READ operation for getting bulk timeseries data is 90 days.
• The maximum volume of time series data cannot exceed 1 GB for performance asset on Public Cloud. The maximum volume will be 100 MB for Private Cloud.
• Import jobs for performance assets are allowed for data in future upto 7 days only.
• For Private cloud, Import jobs for performance assets are allowed for data older than 30 minutes only.
  1. For Timeseries Data ingestion of past 7 days, use either IoT TS Bulk Services or PUT endpoint of Timeseries services. If both the methods are used, data consistency will be compromised.
  2. Timeseries data will not be retrieved via GET endpoint of Timeseries services for Data Ingested via IoT TS Bulk Services for past 7 days.
• The data in json file in Import jobs needs to be in ascending timestamp order.
• Import jobs for performance assets are rejected if timestamps with higher precision than milliseconds are used.
• The '+' character used to include timezone in timestamp must be encoded to '%2B' in time range. For example, 2017-09-21T14:08:00%2B02:00 instead of 2017-09-21T14:08:00+02:00.
• Variable with Data type as BIG_STRING is not supported by IoT TS Bulk Services.
• Bulk Service Data ingestion will be processed within 4 hours in most of the cases; this time may increase based on load.
• The Bulk Service for Read Operations for Get/Timeseries/{entity}/{propertySetName} will have a response time up to 1 minute based on query range and data density.
• The Bulk Service for Read Operations for Get/Timeseries/{entity}/{propertySetName} will not be retrieved upto 4 hours in most of the cases; if Timeseries data is ingested via PUT Endpoint of Timeseries Services.
• IoT Timeseries bulk Import is not supported for Azure Virtual Private Cloud environment

Example Scenario¶

A company is already collecting data for an existing field device. They want to correlate this with simulation data created during the engineering or testing process in order to derive relevant KPIs. The company uploads the simulation data using the IoT File Service and triggers the import with the IoT Time Series Bulk Service. After the importing the data can be consumed via this service to be used by an application that also has access to the Time Series data of the field device.