Metadata Management from Integrated Data Lake¶

Metadata management is a crucial aspect of data management within any Data Lake application. It involves the systematic organization, storage, retrieval, and maintenance of metadata, which is essentially data about data. This includes information about the characteristics, origin, usage, and relationships of the actual data stored in the Data Lake.

Metadata collection operations¶

Metadata collection is the systematic process of collecting metadata keys that manage all the relevant metadata information associated with the specific dataset or system.

Metadata collections are classified into following categories:

• Global collection
• Custom collection
• Reserved collection

Creating a Metadata collection¶

This API can be used to create metadata collection.

Create a collection using the following endpoint:

POST/metadataCollections

Sample Request:

{
  "metadataCollectionId": "document_review",
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection"
}

Sample Response:

{
  "metadataCollectionId": "document_review",
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT",
  "etag": 1
}

Retrieving the list of Metadata collections¶

This API can be used to get the list of all the metadata collections. Global metadata collection (though default) is listed only when a metadata key is added into it.

Retrieve collection list using the below endpoint:

GET/metadataCollections

Sample Response:

{
  "metadataCollections": [
    {
      "metadataCollectionId": "document_review",
      "label": "Document Review",
      "description": "usage and purpose of custom Metadata Collection",
      "status": "DRAFT",
      "etag": 1
    }
  ],
  "page": {
    "size": 0,
    "totalElements": 0,
    "totalPages": 0,
    "number": 0
  }
}

Retrieving the Metadata collection details¶

This API can be used to get the details of the collection for the given id.

Retrieve collection details using the below endpoint:

GET/metadataCollections/{metadataCollectionId}

Sample Response:

{
  "metadataCollectionId": "document_review",
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT",
  "etag": 1
}

Update the Metadata collection details¶

This API can be used to update the details of the collection for the given id.

Update collection details using the below endpoint:

PATCH/metadataCollections/{metadataCollectionId}

Sample Request:

{
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT"
}

Sample Response:

{
  "metadataCollectionId": "document_review",
  "label": "Document Review",
  "description": "usage and purpose of custom Metadata Collection",
  "status": "DRAFT",
  "etag": 1
}

Deleting the Collection¶

This API can be used to delete the collection of the given id.

Delete the collection using the following endpoint:

DELETE/metadataCollections/{metadataCollectionId}

Metadata Keys Operations¶

Metadata keys are the unique identifiers used to represent specific attributes or characteristics associated with the data in a key-value pair structure.

Creating a Metadata Key¶

This API can be used to create metadata key. This API allows admin users to add metadata keys to metadata collection. By default, admin users can add keys directly to Global metadata collection. There are various configuration parameters available while defining a metadata key. These configuration parameters helps admin users to define keys to accept specific metadata values.

The configuration parameters are defined below:

• Value Type: This is the data type that defines the type of metadata value which are accepted for this key. The available value type details are described below:

  Value Type	Use Case
  Enum	Metadata values are pre-defined and user can select one of the value. The available values are seen while defining a key, which supports maximum of 300 values. The special characters supported in Enum values are - Alphanumeric characters (A-Z, a-z, 0-9) - Hyphen ("-") - Forward slash ("/") - Parentheses ("(" and ")") - Period (".") - Colon (":") - Comma (",") - Underscore ("_") - spaces(" ") but not leading and trailing
  EnumList	Metadata values are pre-defined and user can select one or more values. The available values are seen while defining a key, which supports maximum of 300 values. Additionally user can select maximum and minimum values. The special characters supported in EnumList values are - Alphanumeric characters (A-Z, a-z, 0-9) - Hyphen ("-") - Forward slash ("/") - Parentheses ("(" and ")") - Period (".") - Colon (":") - Comma (",") - Underscore ("_") but not leading or trailing - spaces(" ") but not leading and trailing
  String	User has to input one metadata value as string. Additionally, maximum and minimum length are also configured.
  StringList	User has to input one or more metadata values as string. Additionally, maximum and minimum number of accepted values are also configured.
  Long Text	User has to input long values of string e.g. "notes or information as metadata for data lake resources". Long Text support maximum of 2048 characters

• Field name and Field values: These are the additional details that admin users can use to put constraints on metadata values as described below:

  Value Type	Field Name	Field Value
  Enum	options	List of comma separated acceptable values for Enum in capital letters
  EnumList	options	List of comma separated acceptable values for Enum in capital letters
  	minSize	Minimum number of values that can be selected from the enum list. It has to be greater than 0 and less than number of options.
  	maxSize	Maximum number of values that can be selected from the enum list. It has to be greater than minSize and less than or equal to number of options
  String	options	List of comma separated acceptable values for Enum in capital letters
  	minLength	Minimum length of character set in the list that can be accepted as value. It has to be greater than 0.
  	maxLength	Maximum length of character set in the list that can be accepted as value. It has to be greater than or equal to minLength
  StringList	options	List of comma separated acceptable values for Enum in capital letters
  	minSize	Minimum number of values that can be selected from the enum list. It has to be greater than 0 and less than number of options.
  	maxSize	Maximum number of values that can be selected from the enum list. It has to be greater than minSize and less than or equal to number of options

• applyOn: This parameter is used to define where it should be applied i.e., objects (files) or folders based on various requirements.

  Options	Use Case
  OBJECTS_ONLY	Can be used for any metadata that is applicable only at object level
  FOLDERS_ONLY	Can be used to set the metadata in the entire hierarchy which should not be changed at any level e.g. Business Sensitivity or Classification Levels. To be avoided for the metadata which is updated regularly
  FOLDERS_AND_OBJECTS	Can be used where metadata is applicable on entire hierarchy but can be changed if required

• Rule Key: This indicated if this key can be used to create rules.

Recommendations:

• It is advised to keep the key in DRAFT status when created. Many details of the key cannot be updated once key is PUBLISHED.

Create a collection using the following endpoint:

POST/metadataCollections/{metadataCollectionId}/keys

Sample Request:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT"
}

Sample Response:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT",
  "etag": 1
}

Retrieving the list of Metadata keys¶

This API can be used to get the list of all the metadata keys in the provided metadata collection.

Retrieve list of keys using the below endpoint:

GET /metadataCollections/{metadataCollectionId}/keys

Sample Response:

{
  "metadataKeys": [
    {
      "key": "country_of_origin",
      "label": "Country of Origin",
      "description": "description of metadata key and its usage",
      "valueType": "String",
      "additionalDetails": [
        {
          "fieldName": "StringType",
          "fieldValues": [
            "ALPHABETIC"
          ]
        }
      ],
      "applyOn": "OBJECTS_ONLY",
      "isMandatory": true,
      "isRuleKey": false,
      "isSearchable": false,
      "status": "DRAFT",
      "etag": 1
    }
  ],
  "page": {
    "size": 0,
    "totalElements": 0,
    "totalPages": 0,
    "number": 0
  }
}

Retrieving the key details¶

This API can be used to get the details of the defined key in the collection.

Retrieve key details using the below endpoint:

GET /metadataCollections/{metadataCollectionId}/keys/{metadataKey}

Sample Response:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT",
  "etag": 1
}

Update the Metadata key details¶

This API allows the admin users to update metadata key configuration. Update details for metadata key depends on the status of the key.

The following parameters can be updated based on key status:

Update key details using the below endpoint:

PATCH /metadataCollections/{metadataCollectionId}/keys/{metadataKey}

Sample Request:

{
  "label": "Business Sensitivity",
  "description": "description of metadata key and its usage",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "isMandatory": true,
  "isRuleKey": false,
  "status": "DRAFT"
}

Sample Response:

{
  "key": "country_of_origin",
  "label": "Country of Origin",
  "description": "description of metadata key and its usage",
  "valueType": "String",
  "additionalDetails": [
    {
      "fieldName": "StringType",
      "fieldValues": [
        "ALPHABETIC"
      ]
    }
  ],
  "applyOn": "OBJECTS_ONLY",
  "isMandatory": true,
  "isRuleKey": false,
  "isSearchable": false,
  "status": "DRAFT",
  "etag": 1
}

Deleting the Keys¶

This API allows tenant admin to delete metadata key under the provided metadata collection provided in the request.

Delete the collection using the following endpoint:

DELETE /metadataCollections/{metadataCollectionId}/keys/{metadataKey}

Metadata Rules Operations¶

Metadata Rules are a set of predefined conditions that determines the metadata tied to a custom collection that is applicable to Integrated Data Lake resources.

Create a Metadata Rule¶

This API creates metadata rules for metadata collection corresponding to a metadata key.

Recommendations:

• It is advised to Keep the collection in DRAFT status while creating rule. Do not publish collection unless certain about rule key and its value used in the rule.

Create a metadata rule using the following endpoint:

POST /metadataCollections/{metadataCollectionId}/metadataRules

Sample Request:

{
  "name": "Document Review Rule",
  "key": "business_sensitivity",
  "value": "STRICTLY_PRIVATE"
}

Sample Response:

{
  "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
  "name": "Document Review Rule",
  "key": "business_sensitivity",
  "value": "STRICTLY_PRIVATE",
  "etag": 1
}

Retrieving the list of Metadata rules¶

This API lists all the metadata rules associated with a collection.

Retrieve the list of all the metadata rules using the following endpoint:

GET /metadataCollections/{metadataCollectionId}/metadataRules

Sample Response:

{
  "metadataRules": [
    {
      "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
      "name": "Document Review Rule",
      "key": "business_sensitivity",
      "value": "STRICTLY_PRIVATE",
      "etag": 1
    }
  ],
  "page": {
    "size": 0,
    "totalElements": 0,
    "totalPages": 0,
    "number": 0
  }
}

Retrieving the metadata rule details¶

This API provides the metadata rule details for the given id.

Retrieve the metadata rule details using the following endpoint:

GET /metadataCollections/{metadataCollectionId}/metadataRules/{id}

Sample Response:

{
  "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
  "name": "Document Review Rule",
  "key": "business_sensitivity",
  "value": "STRICTLY_PRIVATE",
  "etag": 1
}

Updating the metadata rules¶

This API updates the metadata rule details for the given id. Rule can be updated only in DRAFT status.

Update the metadata rule details using the below endpoint:

PATCH /metadataCollections/{metadataCollectionId}/metadataRules/{id}

Sample Request:

{
  "name": "Document Review Rule",
  "key": "business_sensitivity",
  "value": "STRICTLY_PRIVATE"
}

Sample Response:

{
  "id": "0860f696-af41-4d8d-a104-c1dd508de97a",
  "name": "Document Review Rule",
  "key": "business_sensitivity",
  "value": "STRICTLY_PRIVATE",
  "etag": 1
}

Delete the metadata rule¶

This API deletes the metadata rule provided in the request.

Delete the collection using the following endpoint:

DELETE /metadataCollections/{metadataCollectionId}/metadataRules/{id}

Attribute Based Access Control¶

Attribute Based Access Control (ABAC) is an authorization model that evaluates attributes to determine access. For more information on ABAC, refer to Attribute Based Access Control configuration.

The Attribute Based Access Control for Integrated Data Lake operations are as follows:

• Upload an Object via App Gateway with Metadata

  This PUT method is used to upload an object in the path is permitted as per the policy created for the user by Tenant Admin.

  • Access to upload a file is evaluated at the prefix (parent folder) level for the user.
  • For evaluating the access control, refer to Object and folder operation.

  Endpoint:

  PUT /api/datalake/v4/objects/{path}
  

• Download Object via App Gateway

  This GET method is used to download an object in the path is permitted as per the policy created for the user by Tenant Admin.

  • Access to download a file is evaluated at the object or file level for the user.

  Endpoint:

  GET /objects/{path}
  

• Create empty folder with Metadata

  This POST method is used to create a folder in the path is permitted as per the policy created for the user by Tenant Admin.

  • Access to create a folder is evaluated at the prefix (parent folder) level for the user.
  • For evaluating the access control, refer to Object and folder operation.

  Endpoint:

  POST /folders
  

• Delete an empty Folder

  This DELETE method is used to delete a folder in the path is permitted as per the policy created for the user by Tenant Admin.

  • Access to delete a folder is evaluated at the prefix (folder) level for the user.
  • If permitted, metadata associated with the folder are deleted.

  Endpoint:

  DELETE /folders/{path}
  

• Delete Object

  This DELETE method is used to delete an object on the path which is permitted as per the policy created for the user by Tenant Admin.

  • Access to delete file is evaluated at the object (file) level for the user.
  • If permitted, metadata associated with the folder are deleted.

  Endpoint:

  DELETE/{path}
  

• List Objects

  This GET method is used to access to list objects at a given path for the tenant is evaluated as per the policy created by the Tenant Admin.

  • User can see the folder hierarchy, if access on the parent folder is provided with propagation depth as -1. For more information on propagation depth, refer to Object and folder operation.
  • If user does not have access to any resources, list response is empty. List response will not give any error.

  Endpoint:

  GET /listObjects
  

• Add Metadata

  Users with "WRITE" action can add or update the metadata on the resources.

  Endpoint:

  /objectMetadata/{path}
  

• Get Metadata

  Users with "LIST" access can see UI or fetch API the metadata on the resources.

  Endpoint:

  /objectMetadata/{path}
  

Access Control Actions¶

The access control actions for Integrated Data Lake operations are described below:

Operation	List	Read + List	Create + List	Delete + List	Create + Read + List	Delete + Read + List	Delete + Create + Read + List
View resources (files and folders) and System metadata
View Metadata (user defined metadata)
Upload file with metadata
Download the files
Create folder with metadata
Update Metadata (independent operation)
Delete file(s) or empty folder
Search objects - File name + Metadata (system + user defined metadata)

Action Dependency Table¶

The dependencies for Integrated Data Lake actions are described below: