This site has been deprecated. Do not edit pages here. Please visit the new Pico Labs documentation.


Skip to end of metadata
Go to start of metadata

The simplest way to use cloud-based data inside KRL is with dataset and datasource declarations. Both declarations are made in the ruleset’s global section. The difference between the two is timing.  A dataset request is processed once, when the ruleset begins execution. On the other hand, datasource declarations simply declare an operation that can be used to make queries to the datasource in other parts of the ruleset on-demand.

Datasets

Global declarations can declare a dataset for use in the rules of the ruleset. Dataset declarations are preceded by the keyword dataset and contain a name to bind the data to as well as a URL where the data resides. These are separated by the <- symbol.

In this example, the data at the location specified by the URL is downloaded immediately and bound to the variable temps.

Doing this each time a ruleset runs would be wasteful and inefficient for data that changes infrequently. Consequently, the dataset declaration supports an optional caching specification. If a caching specification is not given the data is assumed to not be cachable[i]. The caching specification consists of the keyword cachable followed by an optional time period specified as follows.

In the specification <period> can be seconds, minutes, hours, days, or weeks. If the time period is not specified, the caching period defaults to 24 hours.

The following shows examples of dataset declarations.

Dataset declarations are best used with small to medium sets of data that don’t change often and can be cached.

Recall the dual execution environment that KRL enables on the server and the browser. This also means that data you load will be transferred to the browser as well. The runtime tries to do this as efficiently as possible and uses a separate script to load the data so that the browser can cache it.  Different instances of KRE may handle the caching times on the browser differently, so be sure to understand the caching settings of the KRE instance you are working to ensure good performance from dataset declarations.

Datasources

Datasource declarations are like datasets in that they are a way of accessing remote data, but they are queried when needed rather than loaded all at once like datasets. Similar to datasets, datasources are declared in the global block using a similar syntax:

Later, a data source can be queried as part of any KRL expression. For example, we could query the datasource declared above like so:

Note that we prepend the namespace datasource: to the name of the datasource and give additional arguments to be added to the query:

  • If the parameter is a string, then it is concatenated with the URL root given in the datasource declaration without modification. The string you supply, when appended to the datasource root URL, must result in exactly the URL that you intend to call.
  • If the parameter is a map of name-value pairs, the map will be converted into a properly formatted query string and appended to the datasource URL. Naturally, the map can be computed rather than simply being made up of literals.  The composition of the URL is done in an effort to create a valid URL[ii].

Datasources use the same syntax for specifying cachability as datasets.

Data Formats

When you use a dataset or datasource declaration, KRL will assume that the data is returned as JSON and try to parse it accordingly. If the parse fails, the data will be returned as a raw string. You can give dataset and datasource declarations hints about the format of the data being requested if it’s not JSON, and KRL will try to parse that data format and return the resulting JSON structure.

Aside from JSON, KRL also supports parsing XML and—as a special case—RSS. The format hint is given by appending the format hint to the name in the declaration separated by a colon like so:

Of course, there is no canonical way to convert XML and RSS to JSON. For specifics of how these formats are converted to JSON data structures, see the KRL documentation.



[i] To avoid server overload, a minimum caching period (currently 20 minutes) is imposed on all remote data sources in the current implementation of KRE. Specifying caching periods shorter than the minimum will have no effect.

[ii] If the URL given in the datasource declaration contains a question mark, then the parameter string created from the map is appended using an ampersand, if it does not, the string is appended using a question mark. 

Labels: