WebSockets, HTML5 and kdb+

Glen Smith kdb+ 5 Comments

The aim of this post is to introduce the idea of using HTML5 and kdb+ together. HTML is a markup language used to build web pages and HTML5 is the latest version. It comes with many new features but one we will focus on is WebSockets which allows us to easily send and receive data from kdb+.

WebSockets allows web applications to maintain bidirectional communications with server side processes over one TCP socket. The method that was widely used before WebSockets is AJAX. AJAX consisted of continually polling a connection to receive new data, opening a new connection, downloading data and then closing the connection. With WebSockets, once the connection between the client and server is established it stays open and data can be sent down to the client in real time. kdb+ has been supporting WebSockets since v3.0.

What does this mean for kdb+?

By allowing access to database through a web browser we are able to use HTML and JavaScript to do things that the kdb+ console cannot, such as create front end user interfaces or data visualizations. It makes the database accessible beyond the console and anyone with a device that has a modern web browser can access the data.

Let’s get started

This is a brief introduction on using WebSockets and JavaScript, if you would like to learn more about HTML and JavaScript check out the courses on Codecademy. Before we start be sure you have your development environment set up:

  1. Download this q script
  2. Download this index.html file and open it in your favourite text editor for inspection
  3. Start up a q server on a port of your choice and load the q script
  4. Open the index.html file in either Chrome or Firefox
  5. Open the developer console by pressing F12, this is used to see the console log messages and for debugging

Data handling

We need to establish a method for sending and receiving data between kdb+ and the browser. There are different methods and it depends on what you want to be able to do from the web browser. The easiest method would be to build a string and evaluate it on the kdb+ side but the best option is to separate the components of the function into the function name and arguments. This is a great option to build from and allows for more functionality such as checking the function name against a dictionary of allowed functions before executing it. The data sent from the browser will be formatted in JSON and so we will use Arthur Whitney’s json.k library for encoding and decoding JSON. As the binary data will be serialized in kdb+ we need the ability to deserialize and serialize in the browser, this is achieved by using Kx’s c.js library.

For the example below we will use a basic version of the separated function components option. The evaluate function is run on the received JSON data, it evaluates the func property and then applies the arguments to it. The diagram below shows the keys chema of the JSON data being sent over WebSockets from the Browser to kdb+.

WebSocketJSONKeySchema

 

Data handling in kdb+

For this basic example, handing data that is sent over WebSockets is easy. The WebSocket handler .z.ws deserializes and decodes the JSON data into a dictionary. The dictionary values are used to evaluate the function with its arguments, then encoded into JSON, serialized and then sent back down the connection handle.


\l json.k

dataformat:{`name`data!(x;y)}

evaluate:{dataformat[x`func;(value x`func) @ value x _ `func]}

.z.ws:{neg[.z.w] -8!.j.j @[evaluate;.j.k -9!x;{'"error: ",x}]}

Data handling in the browser

A WebSocket connection to the URL argument is created and assigned to the ws variable, this enables us to change the methods and properties of the WebSocket connection. The binaryType property is set to force an ArrayBuffer conversion when a binary message is received. The event handlers allow us to execute a function when an event occurs. In the example below once the WebSocket opens, the onopen event handler executes and data is sent. Once a message is received the onmessage event handler executes and the message is shown in the console window. The ws.sendcmd function formats the message correctly before it is sent using the ws.send function.

 


var ws = new WebSocket('ws://localhost:1234');
// Required by c.js 
ws.binaryType = 'arraybuffer';
// WebSocket event handlers
ws.onopen = function () {
     console.log("Open");
     // Sends a JSON encoded message
     ws.sendcmd("sum",2,3);
};
ws.onclose = function () {
     console.log("Closed")
};
ws.onmessage = function (event) {
     var data = JSON.parse(deserialize(event.data));
     console.log(data.data);
};
ws.onerror = function (error) {
     console.log("Error " + error.data);
};
// Used to send function calls to kdb+. e.g. ("sum",1,2,3)
ws.sendcmd = function (args) {
  if(arguments.length<=1) return false;
  var len = arguments.length,data = {},i;
  // First argument used as func property, rest are used as arg properties  
  data['func'] = arguments[0];
  for(i = 1;i<len;i++){
    data['arg'+i] = arguments[i];
  }
  ws.send(serialize(JSON.stringify(data)));
};

Conclusion

  • This tutorial has taught you how to send and receive data to kdb+ via WebSockets
  • There is so much potential for building web applications that interact with kdb+
  • For a more thorough explanation of using HTML5 and kdb+ together check out our document
Glen SmithWebSockets, HTML5 and kdb+

Comments 5

  1. Pingback: WebSockets, HTML5 and kdb+

  2. David Hall

    Have you been able to connect to tickerplant and subscribe using websockets?

    I tried to send (.u.sub[trade;AAPL]) for example. there was only one response and it was:

    ["trade",[]]

    and I used this .z.ws, which works fine for all req / response … but not sub ..

    .z.ws:{neg[.z.w] -8!value -9!x;}

    1. Post
      Author
      Glen Smith

      Hi David,

      I went ahead and tried doing so using our Torq framework which has WebSocket pub sub functionality built in already.

      It would be a better decision to leave the tickerplant alone and just create a new subscriber process which will hold all your WebSocket logic.

      In your subscriber process you will connect to the tickerplant via IPC, wait for the web client to connect and subscribe via .u.sub to the specified symbol and table.

      So when the web client connects they call a function that runs .u.sub against the tickerplant via IPC, runs .html.wssub on the required tables and these will be published to the web client by .html.pub

      To see how we use this in Torq, check out:

      Pub Sub functionality – html.q
      Publisher – heartbeat.q
      Subscriber – monitor.q

      If you need further details, don’t hesitate to let me know!

      All the best,
      Glen

  3. Kingsley Jones

    Thanks for these excellent blog posts!

    I did not realize kdb+ was now free in 32 bit form. That, and your tutorials on this material, are exaclty what is needed to realize the potential in a host of “embedded Kx” applications.

    kdb+ as a time-series store and aggregation engine is perfect for downstream python/js apps…

    I will certainly give it a go for my internal development efforts.

    1. WooiKent Lee

      Thanks for this great feedback Kingsley! We are glad that our blog posts are helping you in your interests!

      Do feel free to browse around and try our challenges too! 😀

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax