Web component

The BioThings SDK web component contains tools used to generate and customize an API, given an Elasticsearch index with data. The web component uses the Tornado Web Server to respond to incoming API requests.

Server boot script

BioThings API ioloop start utilities.

This module contains functions to configure and start the base event loop from command line args. Command line processing is done using tornado.options, with the following arguments defined:

  • port: the port to start the API on, default 8000
  • address: the address to start the API on, default 127.0.0.1
  • debug: start the API in debug mode, default False
  • appdir: path to API configuration directory, default: current working directory

The main function is the boot script for all BioThings API webservers.

index_base.main(APP_LIST, app_settings={}, debug_settings={}, sentry_client_key=None)

Main ioloop configuration and start

Parameters:

Settings

Settings objects used to configure the www API These settings get passed into the handler.initialize() function, of each request, and configure the web API endpoint. They are mostly a container for the Config module, and any other settings that are the same across all handler types, e.g. the Elasticsearch client.

Config module

BiothingWebSettings

class biothings.www.settings.BiothingWebSettings(config='biothings.www.settings.default')[source]

A container for the settings that configure the web API

The config init parameter specifies a module that configures this biothing. For more information see config module documentation.

generate_app_list()[source]

Generates the tornado.web.Application (regex, handler_class, options) tuples for this project.

set_debug_level(debug=False)[source]

Set if running API in debug mode. Should be called before passing self to handler initialization.

validate()[source]

Validate these settings

BiothingESWebSettings

class biothings.www.settings.BiothingESWebSettings(config='biothings.www.settings.default')[source]

BiothingWebSettings subclass with functions specific to an elasticsearch backend

The config init parameter specifies a module that configures this biothing. For more information see config module documentation.

doc_url(bid)[source]

Function to return a url on this biothing API to the biothing object specified by bid.

get_es_client()[source]

Get the Elasticsearch client for this app, only called once on invocation of server.

source_metadata()[source]

Function to cache return of the source metadata stored in _meta of index mappings

Handlers

BaseHandler

class biothings.www.api.helper.BaseHandler(application, request, **kwargs)[source]

Parent class of all biothings handlers, only direct descendant of tornado.web.RequestHandler, contains the common functions in the biothings handler universe:

  • return self as JSON
  • set CORS and caching headers
  • typify the URL keyword arguments
  • optionally send tracking data to google analytics and integrate with sentry monitor
ga_event_object(data={})[source]

Create the data object for google analytics tracking.

get_query_params()[source]

Extract, typify, and sanitize the parameters from the URL query string.

initialize(web_settings)[source]

Tornado handler initialize(), Override to add settings for this biothing API. Assumes that the web_settings kwarg exists in APP_LIST

log_exceptions(exception_msg='')[source]

Logs the current exception in tornado logs and in hipchat room if available. This must be called in an exception handler

return_json(data, encode=True, indent=None)[source]

Return passed data object as JSON response. If jsonp parameter is set in the request, return a valid JSONP response.

Parameters:
  • data – object to return as JSON
  • encode – if encode is False, assumes input data is already a JSON encoded string.
  • indent – number of indents per level in JSON string
set_cacheable(etag=None)[source]

set proper header to make the response cacheable. set etag if provided.

support_cors(*args, **kwargs)[source]

Provide server side support for CORS request.

BaseESRequestHandler

class biothings.www.api.es.handlers.base_handler.BaseESRequestHandler(application, request, **kwargs)[source]

Parent class of all Elasticsearch-based Request handlers, subclass of BaseHandler. Contains handling for Elasticsearch-specific query params (like fields, size, etc)

get_cleaned_options(kwargs)[source]

Separate URL keyword arguments into their functional category. Incoming kwargs are separated into one of 4 categories, depending on how the argument controls the pipeline:

  • control_kwargs - These are arguments that control the pipeline execution (raw, rawquery, etc)
  • es_kwargs - These are arguments that get passed directly to the Elasticsearch client during query
  • esqb_kwargs - These are arguments that go to the Elasticsearch query builder (fields, size, etc)
  • transform_kwargs - These are arguments that go to the Elasticsearch result transformer (jsonld, dotfield, etc)
initialize(web_settings)[source]

Tornado reqeust handler initialization. Initializations common to all Elasticsearch-specific request handlers go here.

return_raw_query_json(query)[source]

Return valid JSON if rawquery option is selected. This is necessary as queries can span multiple lines (POST)

BiothingHandler

class biothings.www.api.es.handlers.biothing_handler.BiothingHandler(application, request, **kwargs)[source]

Request handlers for requests to the annotation lookup endpoint

get(bid=None)[source]

Handle a GET to the annotation lookup endpoint.

initialize(web_settings)[source]

Tornado handler .initialize() function for all requests to the annotation lookup endpoint. Here, the allowed arguments are set (depending on the request method) for each kwarg category.

post(ids=None)[source]

Handle a POST to the annotation lookup endpoint

QueryHandler

class biothings.www.api.es.handlers.query_handler.QueryHandler(application, request, **kwargs)[source]

Request handlers for requests to the query endpoint

get()[source]

Handle a GET to the query endpoint.

initialize(web_settings)[source]

Tornado handler .initialize() function for all requests to the query endpoint. Here, the allowed arguments are set (depending on the request method) for each kwarg category.

post()[source]

Handle a POST to the query endpoint.

MetadataHandler

class biothings.www.api.es.handlers.metadata_handler.MetadataHandler(application, request, **kwargs)[source]

Request handlers for requests to the metadata endpoint.

get()[source]

Handle a GET to the metadata endpoint. Also handles /metadata/fields.

initialize(web_settings)[source]

Tornado handler .initialize() function for all requests to the metadata endpoint. Here, the allowed arguments are set (depending on the request method) for each kwarg category.

Elasticsearch Query Builder

class biothings.www.api.es.query_builder.ESQueries(es_kwargs={})[source]

A very simple class to object-ize Elasticsearch Query DSL. This should be replaced by the official Elasticsearch equivalent. Also contains a simple JSON validator after generating all queries (dump to string and re-read)

bool(query_kwargs)[source]

Given query_kwargs, validate and return a bool query.

match(query_kwargs)[source]

Given query_kwargs, validate and return a match query.

match_all(query_kwargs)[source]

Given query_kwargs, validate and return a match_all query.

multi_match(query_kwargs)[source]

Given query_kwargs, validate and return a multi_match query.

query_string(query_kwargs)[source]

Given query_kwargs, validate and return a query_string query.

raw_query(raw_query)[source]

Given query_kwargs, validate and return a raw query (queries that don’t fit the same query_template).

class biothings.www.api.es.query_builder.ESQueryBuilder(index, doc_type, options, es_options, scroll_options={}, userquery_dir='', regex_list=[], default_scopes=['_id'])[source]

Class to return the correct query given the request endpoint, method, and URL params.

Parameters:
  • index – The Elasticsearch index to run the query on
  • doc_type – The Elasticsearch document type of the query
  • options – Options from the URL string relevant to query building
  • es_options – Options for Elasticsearch query stage
  • scroll_options – Options for scroll requests
  • regex_list – A list of (regex, scope) tuples for annotation lookup
  • userquery_dir – The directory containing user queries for this app
  • default_scopes – A list representing the default Elasticsearch query scope(s) for this query
add_query_filters(_query)[source]

Given a query, add any other filters

annotation_GET_query(bid)[source]

Return an annotation lookup GET query for this bid.

Parameters:bid – Biothing ID, used to lookup the annotation
annotation_POST_query(bids)[source]

Return an annotation lookup POST query for these bids.

Parameters:bids – Biothing IDs, used to lookup the annotations
get_query_filters()[source]

Override me to add more query filters

metadata_query()[source]

Return a metadata query

query_GET_query(q)[source]

Return a query endpoint GET query for this query string q.

Parameters:q – query string specifying the query
query_POST_query(qs, scopes)[source]

Return query endpoint POST queries for these query strings qs.

Parameters:
  • qs – Query strings to query
  • scopes – Scope of query strings qs
scroll(scroll_id)[source]

Return the next batch of results from a scroll query.

Parameters:scroll_id – ID of the batch to yield hits from

Elasticsearch Query

class biothings.www.api.es.query.ESQuery(client, options={})[source]

This class contains functions to execute the query section of all handler pipelines. The inputs to it are an Elasticsearch client (from BiothingESWebSettings), and any options from the URL string. Each handler calls a different query function, though they all do essentially the same thing: get the query generated in the ESQueryBuilder stage of the pipeline (query_kwargs), and run it using the supplied Elasticsearch client.

annotation_GET_query(query_kwargs)[source]

Given query_kwargs from ESQueryBuilder, return results of annotation lookup GET query on ES client.

annotation_POST_query(query_kwargs)[source]

Given query_kwargs from ESQueryBuilder, return results of annotation lookup POST query on ES client.

get_biothing(query_kwargs)[source]

Return a biothing using the Elasticsearch client.get function

metadata_query(query_kwargs)[source]

Given query_kwargs from ESQueryBuilder, return results of metadata query on ES client.

query_GET_query(query_kwargs)[source]

Given query_kwargs from ESQueryBuilder, return results of query GET on ES client.

query_POST_query(query_kwargs)[source]

Given query_kwargs from ESQueryBuilder, return results of query POST on ES client.

scroll(query_kwargs)[source]

Given query_kwargs from ESQueryBuilder, return results of a scroll on ES client - returns next batch of results.

Elasticsearch Result Transformer

class biothings.www.api.es.transform.ESResultTransformer(options, host, doc_url_function=<function ESResultTransformer.<lambda>>, jsonld_context={}, data_sources={}, output_aliases={}, app_dir='', source_metadata={})[source]

Class to transform the results of the Elasticsearch query generated prior in the pipeline. This contains the functions to extract the final document from the elasticsearch query result in Elasticsearch Query. This also contains the code to flatten a document (if dotfield is True), or to add JSON-LD context to the document (if jsonld is True).

Parameters:
  • options – Options from the URL string controlling result transformer
  • host – Host name (extracted from request), used for JSON-LD address generation
  • doc_url_function – a function that takes one argument (a biothing id) and returns a URL to that biothing
  • jsonld_context – JSON-LD context for this app (optional)
  • data_sources – unused currently (optional)
  • output_aliases – list of output key names to alias, unused currently (optional)
  • app_dir – Application directory for this app (used for getting app information in /metadata)
  • source_metadata – Metadata object containing source information for _license keys
clean_annotation_GET_response(res)[source]

Transform the results of a GET to the annotation lookup endpoint.

Parameters:res – Results from Elasticsearch Query.
clean_annotation_POST_response(bid_list, res, single_hit=True)[source]

Transform the results of a POST to the annotation lookup endpoint.

Parameters:
  • bid_list – List of biothing id inputs
  • res – Results from Elasticsearch Query
  • single_hit – If True, render queries with 1 result as a dictionary, else as a 1-element list containing a dictionary
clean_metadata_response(res, fields=False)[source]

Transform the results of a GET to the metadata endpoint.

Parameters:res – Results from Elasticsearch Query.
clean_query_GET_response(res)[source]

Transform the results of a GET to the query endpoint.

Parameters:res – Results from Elasticsearch Query.
clean_query_POST_response(qlist, res, single_hit=True)[source]

Transform the results of a POST to the query endpoint.

Parameters:
  • qlist – List of query inputs
  • res – Results from Elasticsearch Query
  • single_hit – If True, render queries with 1 result as a dictionary, else as a 1-element list containing a dictionary
clean_scroll_response(res)[source]

Transform the results of a GET to the scroll endpoint

Parameters:res – Results from Elasticsearch Query.