Metadata Search¶

Note

This section is available only for Virtual Private cloud (VPC).

The Metadata search can be used to search and analyze big volumes of objects quickly via a highly scalable text search. The search accepts request parameters such as System metadata (e.g. object name, location, created date, size) and Custom metadata (e.g. metadata keys). Filters can be combined using multiple comparative and relational operators based on the the datatype of the metadata. The search result will be available as collection of files and folders which match the provided search criteria for the provided storage path for which user has access. Search API honors the ABAC policies as applicable to the user.

If there are more results available than the defined maximum i.e. page size, a nextToken property is returned in the response. This property specifies the search request, which returns the remaining the results. It is included in every subsequent response, until the end of the list.

Supported Operators¶

The metadata types and their supported search operators are displayed in the table:

Metadata Type	Parameter Name (Datatype)	Operator	Description
System metadata	size (INT)	EQ → Equal GT → Greater than GE → Greater than or equal NE → Not equal LT → Less than LE → Less than or equal	System metadata includes parameters like object name, size, type, and last updated date
	fileName (STRING)	EQ → Equal NE → Not equal startsWith → starts with contains → contains set of characters within file name	Name of the object
	key (STRING)	EQ → Equal NE → Not equal startsWith → starts with contains → contains set of characters within file name	path of the object
	documentType (STRING)	EQ → Equal NE → Not equal	Type of the document that is file or folder. Supported values are `OBJECT` and `FOLDER`
	lastModified (DATE)	EQ → Equal NE → Not equal GT → Greater than LT → Less than GE → Greater than or equal LE → Less than or equal after → after the provided input date before → before the provided input date	Last updated date of the object (file)
Custom metadata	ENUMLIST ENUM STRING STRINGLIST	EQ → Equal NE → Not equal “==” → Equal “!=” → Not equal IN → Value within ENUMLIST NOT IN → Value NOT IN ENUMLIST	Support List of predefined values Note: Ensure that the search criteria should always be in sequence - `Value IN ENUMLIST` List should always be on the right hand side of the operator for IN and NOT IN Support ONE predefined values from the list Any character stream List of character streams Note: Ensure that the search criteria should always be in sequence - `Value IN STRINGLIST` List should always be on the right hand side of the operator for IN and NOT IN

Search API¶

Note

Search operation honors Attribute Based Access Control (ABAC). User will see only those objects in search result for which he as access.

Endpoint:

POST /search

Request:

  {
    "currentPath": "20MayFolder/",
    "systemMetadataCriteria": "system.documentType eq 'OBJECT' or system.documentType eq 'FOLDER'",
    "criteriaJoinOperator": "and",
    "customMetadataCriteria": "global.file_status  eq 'PUBLIC'",
    "nextToken": "eyJhdXRob3JpemVkUGFnZSI6Mywic2tpcCI6NCwibWF4U2l6ZSI6NH0=",
    "maxSize": 200
  }

Request Parameters Description¶

The overview of the parameters required for making data requests, detailing search paths are displayed in the table:

Parameter	Datatype	Description	Recommended Usage
currentPath	STRING	Specifies the directory or folder path where search needs to be performed	This will always be folder. Search will be performed in the entire hierarchy of this path. The result would be presented as a flat hierarchy listing all the files/folders in the path Note: `currentPath` cannot be a filename. It could possibly be root path ('/')
systemMetadataCriteria	STRING	System metadata criteria supports 5 search parameters - size - fileName - path - resourceType lastModified	Check Supported operators to create the criteria related to system metadata System metadata criteria has a limit of 3000 characters Sample: `system.fileName eq 'test.txt' and system.documentType eq 'OBJECT'`
criteriaJoinOperator	STRING	Condition operator to join the system and custom metadata criteria	Possible value are `AND` and `OR` only
customMetadataCriteria	STRING	Custom metadata support four datatypes - ENUM, ENUMLIST, STRING, STRINGLIST	Check Supported operators to create the criteria related to custom metadata Custom metadata criteria has a limit of 10,000 characters Sample: `global.accessCriteria eq 'PUBLIC'`
nextToken	STRING	Value of next token to fetch next request results. Combination of access control pages and end response
maxSize	INTEGER	Maximum number of objects in a single request	Maximum number of objects that can be fetched in single request is 1,000

Response:

{
  "storageAccount": "datalake-dev-aaab",
  "storagePath": "data/ten=tenantname/{path} OR data/sub=subtenantId{path}",
  "objects": {
    "files": [
      {
        "key": "data/ten=dide2/commands.txt",
        "lastModified": "2018-10-03T09:21:36.559Z",
        "contentSize": 1000,
        "storageClass": "HOT"
      }
    ],
    "folders": [
      {
        "key": "data/ten=dide2/e2eTest/"
      }
    ]
  },
  "page": {
    "size": 1000,
    "nextToken": "1BpNN1QT3bG+k2dJG1TgmquqCzzCjbiuYymgjL91UChw//OBKBgBrmUcZpF6fT6C428OjxaMntOk+doIH45VMRAWt40WHl58ddeTuzsMCsd9T/nMjddi2SQ=="
  },
  "isTruncated": true