API Sparse Fieldsets Guidelines¶

Introduction¶

APIs often deal with large datasets, such as material definitions. Data is typically transferred over network boundaries and it is usually impractical or infeasible to transfer entire datasets at once. Instead, API clients are expected to load only the data they need.

An API MAY provide the ability to select which members of a resource shall be returned. These members are also named "fields of the resource".

This process of selecting members of a resource is comparable to the projection in relational algebra. Others compare this function to the filtering of members on the server side. The approach is commonly named "sparse fieldset selection of a resource".

The ability to selectively return fields helps to minimize the network traffic between API client and server.

For example, applying a fields expression of

name,dimension(width,height)

to

{
  "data": {
    "name": "My Device",
    "deviceType": {
      "id": "hvac",
      "name": "HVAC device"
    },
    "dimension": {
      "width": 1.3,
      "height": 2.52,
      "depth": 0.9
    }
  }
}

would result in

{
  "data": {
    "name": "My Device",
    "dimension": {
      "width": 1.3,
      "height": 2.52
    }
  }
}

API MAY support sparse fieldsets [500]¶

An API MAY support sparse fieldsets for different endpoints. For supported endpoints, an API client can request the inclusion and/or exclusion of specific fields of the requested resource(s).

Sparse fieldsets MUST be defined via the fields query parameter [500.1]¶

When an API client specifies to return only some fields (members) from a resource, the fields query parameter MUST be used to specify the subset or respective projection on the type(s).

Value of the fields parameter SHOULD contain the fields expression [500.2]¶

The usage of the fields parameter SHOULD follow the format below.

GET /articles?fields=<expression> HTTP/1.1

Note that the query parameter in the <expression> does not specify the resource to address; this is implicitly defined by the requested resource.

The value of the fields parameter MUST follow the syntax defined in the Appendix. I.e., it can be a comma-separated list or a more complex, hierarchical expression. An empty value indicates that no fields should be returned.

Here is an example of a simple fields expression.

GET /articles?fields=title,body,comments HTTP/1.1

The sparse-fieldsets projection can also be applied to nested resources.

Here is an example of a fields expression for nested articles resources.

GET /api/blogs/1/articles?fields=title,body,comments HTTP/1.1

This query will return the articles of the blog 1, and only the fields title, body, comments of the article resource are provided.

API MUST NOT ignore the fields parameter [500.3]¶

If an API endpoint supports the fields parameter, the API MUST NOT include additional fields of the resource objects in the response when an API client requests a restricted field set. Likewise, an API MUST NOT exclude existing fields that are specified in the fields parameter.

API SHOULD specify default behavior for unspecified fields parameter [500.4]¶

The fields parameter SHOULD be optional. If the API client does not specify a fields parameter, an API

• MAY send all fields of the requested resource (most common approach), or
• MAY send a subset of fields of the requested resource, or
• MAY send no fields of the requested resource.

You might consider defining and documenting a default value for the fields parameter, so that API clients can adjust this value to their needs if required.

In general, responses MUST follow the API specification and should present the least surprise to the API clients.

If an API client provides a value for the fields parameter, the API SHOULD apply the fields expression to the maximum response. I.e., if the default response without a specified fields parameter value contains only a subset of fields, a response to a request with an explicitly defined fields parameter value can contain additional fields that are not contained in the default response, but are supported by the maximum response.

Removing fields from the default response or the maximum response in a newer API version, MUST be considered a breaking change of the API. Adding fields to the default response or the maximum response, SHOULD NOT be considered a breaking change.

Fields expressions SHOULD be treated as case sensitive [500.5]¶

The evaluation of a fields expression SHOULD be case sensitive. For instance, the following expression is valid and relates to three separate fields.

test,Test,tEst

API MUST NOT expect a specific field order [500.6]¶

The order of fields (at the same level of a hierarchy) in a fields expression MUST NOT be considered relevant. For example, the following two requests are equivalent and will result in the same response.

GET /articles?fields=name,description HTTP/1.1

GET /articles?fields=description,name HTTP/1.1

API MAY support wildcards for field selection [500.7]¶

An API MAY support wildcards in fields expressions. If the API supports wildcards to get all properties of a specific level and all subordinate levels, the * (asterisk) character MUST be used to indicate a wildcard. For example, if the request is

GET /articles?fields=* HTTP/1.1

filtering

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": {
      "version": 3.2,
      "developedBy": "IDPF"
    }
  }
}

will result in returning the full data

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": {
      "version": 3.2,
      "developedBy": "IDPF"
    }
  }
}

If the request is

GET /articles?fields=details(*) HTTP/1.1

the result is

{
  "data": {
    "details": {
      "version": 3.2,
      "developedBy": "IDPF"
    }
  }
}

As the recommended behavior for returning a field of type object is returning all its direct and subordinate fields, the following requests for getting the details object are equivalent and will result in the same response.

GET /articles?fields=details HTTP/1.1

GET /articles?fields=details(*) HTTP/1.1

Hence, supporting the * wildcard is especially useful if the API returns a restricted fieldset by default and API clients need an option to return the full fieldset without explicitly listing field names.

Missing fields SHOULD be ignored [500.8]¶

When a fields expression is applied to data, it might happen that a specified field is not found in the data. This does not make the fields expression as such invalid.

An API SHOULD ignore missing fields. Accordingly, if a fields expression has nested fields below a missing field, these nested fields SHOULD be ignored, too.

Alternatively, an API MAY treat missing fields as an invalid fields parameter (see also 500.9).

An API SHOULD return the status code 400 if the fields parameter is invalid [500.9]¶

If an invalid fields expression is set (including a fields expression consisting of space characters only), a response with the status code 400 Bad Request SHOULD be returned.

If a field is listed twice on the same level, the expression SHOULD be considered invalid:

GET /articles?fields=test(description),name,test HTTP/1.1

In the exceptional case of treating fields expressions as case-insensitive (see 500.5), also the following expression SHOULD be considered invalid.

GET /articles?fields=test(description),name,Test HTTP/1.1

A caller might set the fields parameter multiple times in the query string. How an API handles this case SHOULD be aligned with the API's overall strategy of handling duplicated query parameters. It is recommended that the API handles this case as an invalid fields parameter because the parameter values might be contradicting or would need to be merged to a single fields expression.

Appendix¶

The following sections provide additional information about fields expressions for API clients and providers, including the handling of JSON data types, examples of valid and invalid fields expressions, as well as the formal syntax of fields expressions.

JSON data types and fields selection¶

When a fields expression is applied to JSON data, the result depends on the respective JSON data types of the contained fields. The following section describes the effect of fields expressions on the result with regard to the different data types.

Strings, numbers, Boolean values and null¶

If a field with a type of string or number, a Boolean value or a null value is part of the filter expression, its value will be contained in the result.

For example, applying a fields expression of

details

to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": "More details"
  }
}

with details of type string results in

{
  "data": {
    "details": "More details"
  }
}

Likewise, applying the same expression to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": null
  }
}

where details is null, results in

{
  "data": {
    "details": null
  }
}

Applying the expression to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": true
  }
}

where details has a value of true, results in

{
  "data": {
    "details": true
  }
}

Applying the expression to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": 3.0
  }
}

where details is of type number, results in

{
  "data": {
    "details": 3.0
  }
}

Objects and arrays¶

If the field is of type object and no nested fields expression is provided for the field, the full object including all its direct and subordinate fields will be contained in the result. For example, if the fields expression is

details

(without any nested fields), all subordinate fields are listed, and filtering

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": {
      "version": 3.2,
      "developedBy": "IDPF"
    }
  }
}

will result in

{
  "data": {
    "details": {
      "version": 3.2,
      "developedBy": "IDPF"
    }
  }
}

If a nested fields expression is provided for the field, only explicitly listed fields will be included. For example, if the fields expression is

details(developedBy)

the result is

{
  "data": {
    "details": {
      "developedBy": "IDPF"
    }
  }
}

Further nesting is allowed.

If the field is of type array, all elements of the array (if any) will be included. For example, applying a fields expression of

details

to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": [
      "info",
      42
    ]
  }
}

will result in

{
  "data": {
    "details": [
      "info",
      42
    ]
  }
}

If an element is of type object, the fields expression below will be used to further filter the fields of the object. I.e., the fields expression below is not directly applied to the array but to the contained object. For example, applying a fields expression of

details

(without any nested fields) to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": [
      "info",
      42, {
        "version": 3.2,
        "developedBy": "IDPF"
      }
    ]
  }
}

will result in

{
  "data": {
    "details": [
      "info",
      42, {
        "version": 3.2,
        "developedBy": "IDPF"
      }
    ]
  }
}

If the fields expression is

details(developedBy)

the result is

{
    "data": {
    "details": [
      "info",
      42, {
        "developedBy": "IDPF"
      }
    ]
  }
}

If an element of the array is again of type array, all elements of this subordinate array (if any) will be included, and the evaluation of the elements will be done recursively per element as explained before. I.e., the fields expression below is not directly applied to the subordinate array. Instead, it is shifted to the next level that is of type object. For example, applying a fields expression of

details(developedBy)

to

{
  "data": {
    "name": "ePUB",
    "extension": ".epub",
    "details": [
      "info",
      42, {
        "version": 3.2,
        "developedBy": "IDPF"
      }, [
        true, {
          "remark": "example",
          "developedBy": "IDPF"
        }
      ]
    ]
  }
}

will result in

{
  "data": {
    "details": [
      "info",
      42,  {
        "developedBy": "IDPF"
      }, [
        true, {
          "developedBy": "IDPF"
        }
      ]
    ]
  }
}

Examples of valid fields expressions¶

Examples of valid fields expressions are

dimension(width)

name,description

connection(*)

details(metadata(version)),id

To allow for arranging the expression according to user preferences, the space character SHOULD be allowed as a separator (see SP in the ABNF notation). Thus, the following expressions are valid, too.

connection (  description )

velocity, pressure

author( * )

  details(metadata(version)),id

Examples of invalid fields expressions¶

Here are some examples of invalid fields expressions. They can help to test implementations.

Left parenthesis without preceding field name.

(name)

()

(*)

dimension(width)(height)

dimension((width))

Right parenthesis without matching left parenthesis.

description)

Parentheses that don't contain any fields.

()

( )

(,)

( , )

Misplaced comma.

dimension,

,dimension

name,,dimension

dimension(width),

dimension(,width)

dimension(width,)

Spaces in property names.

dimension(wid th)

Combining a wildcard with field names.

dimension(*,width,height)

Syntax and validation¶

The following paragraph shows the formal syntax of the fields expression as ABNF.

fields     = (*SP "*" *SP) / (field *("," field))

field      = *SP field-name *SP ["(" fields ")" *SP]

field-name = (ALPHA / DIGIT) [*(ALPHA / DIGIT / "-" / "_") (ALPHA / DIGIT)]

ALPHA       =  %x41-5A / %x61-7A

DIGIT       =  %x30-39

SP          =  %x20

Extensibility and additional characters¶

You might need to support additional characters in field names. Still, you SHOULD restrict the supported characters for field names to the required minimum (see also 101.8 for guidance on field names). To avoid conflicts with preserved characters of the current specification and to support backwards-compatibility of future versions of the guidelines, you MUST apply the following rules:

If you support one of the following characters for field names, it MUST be escaped via a \ (backslash):

• "\" backslash
• " " space
• "," comma
• "(" left parenthesis
• ")" right parenthesis
• "[" left bracket
• "]" right bracket

If an initial backslash is not followed by one of these characters, the fields expression MUST be evaluated as invalid.

If a fields expression contains an unescaped [ (left bracket) or ] (right bracket), the expression MUST be evaluated as invalid, too. This will allow for future extension.