Version 4.x - What's new?

Metadata management is a crucial aspect of data management within any Data Lake application. It involves the systematic organization, storage, retrieval, and maintenance of metadata, which is essentially data about data. This includes information about the characteristics, origin, usage, and relationships of the actual data stored in the Data Lake.

Metadata within Integrated Data Lake was available in the form of tags. User could provide this tag based metadata on folders or on objects. Tags were single-word that provide a concise way to categorize content, focusing on key concepts or themes. It was a flexible way to organize the resources but they lack a standardized structure, potentially leading to ambiguity. While tags facilitate general categorization, their loose structure might not be as efficient for targeted searches or retrieval of specific information.

With Metadata Management V4 APIs, transitioning from metadata as tags to key-value pair metadata brings about a shift from a more generalized categorization approach to a structured, detailed, and context-rich representation of data attributes. This change enhances precision, searchability, and overall data management capabilities in diverse and complex information ecosystems.

The Metadata Management V4 APIs bring forth a range of new features to enhance the functionality of the Data Lake application:

• Metadata as Key-Value Pairs:

  Introduces a structured approach to metadata by utilizing key-value pairs. Users can configure metadata keys, metadata collections, and define rules that govern the behavior of these collections.

• Enhanced Security for Object Uploads and Folder Creation:

  Offers a more secure method for uploading objects or creating folders in the storage through the API Gateway. This is a notable improvement from the initial Integrated Data Lake, which initially supported only native interfaces for uploading and downloading objects.

• Metadata Inclusion during Object Upload and Folder Creation:

  Allows users to provide metadata information concurrently with the upload of an object or the creation of a folder. This ensures that metadata is associated with data at the time of its addition to the Data Lake, facilitating comprehensive and real-time information management.