Htsget retrieval API spec v0.2rc

Design principles

This data retrieval API bridges from existing genomics bulk data transfers to a client/server model with the following features:

Explicitly this API does NOT:

Protocol essentials

All API invocations are made to a configurable HTTP(S) endpoint, receive URL-encoded query string parameters, and return JSON output. Successful requests result with HTTP status code 200 and have UTF8-encoded JSON in the response body, with the content-type application/json. The server may provide responses with chunked transfer encoding. The client and server may mutually negotiate HTTP/2 upgrade using the standard mechanism.

Any timestamps that appear in the response from an API method are given as ISO 8601 date/time format.

HTTP responses may be compressed using RFC 2616 transfer-coding, not content-coding.


Requests to the retrieval API endpoint may be authenticated by means of an OAuth2 bearer token included in the request headers, as detailed in RFC 6750. Briefly, the client supplies the header Authorization: Bearer xxxx with each HTTPS request, where xxxx is a private token. The mechanisms by which clients originally obtain their authentication tokens, and by which servers verify them, are currently beyond the scope of this specification. Servers may honor non-authenticated requests at their discretion.


The server MUST respond with an appropriate HTTP status code (4xx or 5xx) when an error condition is detected. In the case of transient server errors, (e.g., 503 and other 5xx status codes), the client SHOULD implement appropriate retry logic as discussed in Reliability & performance considerations below.

For errors that are specific to the htsget protocol, the response body SHOULD be a JSON object (content-type application/json) providing machine-readable information about the nature of the error, along with a human-readable description. The structure of this JSON object is described as follows.

Error Response JSON fields


The type of error. This SHOULD be chosen from the list below.


A message specific to the error providing information on how to debug the problem. Clients MAY display this message to the user.

The following errors types are defined:

Error type HTTP status code Description
InvalidAuthentication 401 Authorization provided is invalid
PermissionDenied 403 Authorization is required to access the resource
NotFound 404 The resource requested was not found
UnsupportedFormat 400 The requested file format is not supported by the server
InvalidInput 400 The request parameters do not adhere to the specification
InvalidRange 400 The requested range cannot be satisfied

The error type SHOULD be chosen from this table and be accompanied by the specified HTTP status code. An example of a valid JSON error response is:

   "error": "NotFound",
   "message": "No such accession 'ENS16232164'"


All API resources should have the following support for cross-origin resource sharing (CORS) to support browser-based clients:

If a request to the URL of an API method includes the Origin header, its contents will be propagated into the Access-Control-Allow-Origin header of the response. Preflight requests (OPTIONS requests to the URL of an API method, with appropriate extra headers as defined in the CORS specification) will be accepted if the value of the Access-Control-Request-Method header is GET. The values of Origin and Access-Control-Request-Headers (if any) of the request will be propagated to Access-Control-Allow-Origin and Access-Control-Allow-Headers respectively in the preflight response. The Access-Control-Max-Age of the preflight response is set to the equivalent of 30 days.

Method: get reads by ID

GET /reads/<id>

The core mechanic for accessing specified reads data. The JSON response is a “ticket” allowing the caller to obtain the desired data in the specified format, which may involve follow-on requests to other endpoints, as detailed below.

The client can request only reads overlapping a given genomic range. The response may however contain a superset of the desired results, including all records overlapping the range, and potentially other records not overlapping the range; the client should filter out such extraneous records if necessary. Successful requests with empty result sets still produce a valid response in the requested format (e.g. including header and EOF marker).

URL parameters


A string specifying which reads to return.

The format of the string is left to the discretion of the API provider, including allowing embedded “/” characters. Strings could be ReadGroupSetIds as defined by the GA4GH API, or any other format the API provider chooses (e.g. “/data/platinum/NA12878”, “/byRun/ERR148333”).

Query parameters

optional string

Request read data in this format. Default: BAM. Allowed values: BAM,CRAM.

The server SHOULD reply with an UnsupportedFormat error if the requested format is not supported. 1

referenceName optional

The reference sequence name, for example “chr1”, “1”, or “chrX”. If unspecified, all reads (mapped and unmapped) are returned. [^b]

The server SHOULD reply with a NotFound error if the requested reference does not exist.

optional 32-bit unsigned integer

The start position of the range on the reference, 0-based, inclusive.

The server SHOULD respond with an InvalidInput error if start is specified and a reference is not specified (see referenceName).

The server SHOULD respond with an InvalidRange error if start and end are specified and start is greater than end.

optional 32-bit unsigned integer

The end position of the range on the reference, 0-based exclusive.

The server SHOULD respond with an InvalidInput error if end is specified and a reference is not specified (see referenceName).

The server SHOULD respond with an InvalidRange error if start and end are specified and start is greater than end.


A list of fields to include, see below Default: all


A comma separated list of tags to include, default: all. If the empty string is specified (tags=) no tags are included.

The server SHOULD respond with an InvalidInput error if tags and notags intersect.


A comma separated list of tags to exclude, default: none.

The server SHOULD respond with an InvalidInput error if tags and notags intersect.

Field filtering

The list of fields is based on BAM fields:

Field Description
QNAME Read names
FLAG Read bit flags
RNAME Reference sequence name
POS Alignment position
MAPQ Mapping quality score
RNEXT Reference sequence name of the next fragment template
PNEXT Alignment position of the next fragment in the template
TLEN Inferred template size
SEQ Read bases
QUAL Base quality scores

Example: fields=QNAME,FLAG,POS.

Response JSON fields


Read data format. Default: BAM. Allowed values: BAM,CRAM.

array of objects

An array providing URLs from which raw data can be retrieved. The client must retrieve binary data blocks from each of these URLs and concatenate them to obtain the complete response in the requested format.

Each element of the array is a JSON object with the following fields:


One URL.

May be either a https: URL or an inline data: URI. HTTPS URLs require the client to make a follow-up request (possibly to a different endpoint) to retrieve a data block. Data URIs provide a data block inline, without necessitating a separate request.

Further details below.

optional object

For HTTPS URLs, the server may supply a JSON object containing one or more string key-value pairs which the client MUST supply as headers with any request to the URL. For example, if headers is {"Range": "bytes=0-1023", "Authorization": "Bearer xxxx"}, then the client must supply the headers Range: bytes=0-1023 and Authorization: Bearer xxxx with the HTTPS request to the URL.

optional hex string

MD5 digest of the blob resulting from concatenating all of the “payload” data — the url data blocks.

Response data blocks

Diagram of core mechanic

Diagram showing ticket flow

  1. Client sends a request with id, genomic range, and filter.
  2. Server replies with a ticket describing data block locations (URLs and headers).
  3. Client fetches the data blocks using the URLs and headers.
  4. Client concatenates data blocks to produce local blob.

While the blocks must be finally concatenated in the given order, the client may fetch them in parallel.

HTTPS data block URLs

  1. must have percent-encoded path and query (e.g. javascript encodeURIComponent; python urllib.urlencode)
  2. must accept GET requests
  3. should provide CORS
  4. should allow multiple request retries, within reason
  5. should use HTTPS rather than plain HTTP except for testing or internal-only purposes (for security + in-flight corruption detection)
  6. Server must send the response with either the Content-Length header, or chunked transfer encoding, or both. Clients must detect premature response truncation.
  7. Client and URL endpoint may mutually negotiate HTTP/2 upgrade using the standard mechanism.
  8. Client must follow 3xx redirects from the URL, subject to typical fail-safe mechanisms (e.g. maximum number of redirects), always supplying the headers, if any. If a byte range HTTP header accompanies the URL, then the client MAY decompose this byte range into several sub-ranges and open multiple parallel, retryable requests to fetch them. (The URL and headers must be sufficient to authorize such behavior by the client, within reason.)

Inline data block URIs

e.g. data:application/vnd.ga4gh.bam;base64,SGVsbG8sIFdvcmxkIQ== (RFC 2397, Data URI). The client obtains the data block by decoding the embedded base64 payload.

  1. must use base64 payload encoding (simplifies client decoding logic)
  2. client should ignore the media type (if any), treating the payload as a partial blob.

Note: the base64 text should not be additionally percent encoded.

Reliability & performance considerations

To provide robustness to sporadic transfer failures, servers should divide large payloads into multiple data blocks in the urls array. Then if the transfer of any one block fails, the client can retry that block and carry on, instead of starting all over. Clients may also fetch blocks in parallel, which can improve throughput.

Initial guidelines, which we expect to revise in light of future experience:

Security considerations

The URL and headers might contain embedded authentication tokens; therefore, production clients and servers should not unnecessarily print them to console, write them to logs, embed them in error messages, etc.

Possible future enhancements

  1. add a mechanism to request reads from more than one ID at a time (e.g. for a trio)
  2. allow clients to provide a suggested data block size to the server
  3. consider adding other data types (e.g. variants)
  4. add POST support (if and when request sizes get large)
  5. [mlin] add a way to request all unmapped reads (e.g. by passing * for referenceName)
  6. [dglazer] add a way to request reads in GA4GH binary format 2 (e.g. fmt=proto)

Existing clarification suggestions

  1. This should probably be specified as a (comma separated?) list in preference order. If the client can accept both BAM and CRAM it is useful for it to indicate this and let the server pick whichever format it is most comfortable with. 

  2. How will compression work in this case - can we benefit from columnar compression as does Parquet?