MATLAB, kdb+ and Streaming Data

Liam O'Connor Uncategorized

Introduction

Recent projects at AquaQ have required the ability to receive streaming data to MATLAB from a vanilla kdb+ tickerplant. The aim therefore was to create a simple method for this with the ability to work over as many platforms as possible. The examples included in this blog use the R2021a version of MATLAB and kdb+ 4.0.

Java API for kdb+ 

The Java API for kdb+ is a Java library which acts as an interface between kdb+ and Java. The API is included within a java file ‘c.jar’ which can be found on KX System’s github repository here. The source c.java file, is formatted as a single ‘c’ outer class, which contains a number of inner classes whose role are to decode data so that it is readable from kdb+ to Java and vice versa. For example, one of these inner classes is the ‘Dict’ class which converts kdb+ dictionaries to Java Objects. This same class is also later used to recreate these Java Objects from arrays of keys and values in MATLAB. 

Setup

Before we begin using our Java utility within MATLAB, we must add the Java files to MATLAB’s Java class file path which can be added dynamically by running the following on MATLAB’s command line: 

>> javaaddpath('<pathtofile>') 

Alternatively, we can permanently alter the Java class path in MATLAB by manually editing the classpath file. 

>> edit('classpath.txt')

Connecting to a kdb+ Process

An initial connection to a kdb+ process can be established by instantiating the outer c class. For the purpose of this blog post, I will start a q process on port number 5001. We call the c class from the KX Java package and open a connection to this q process on the MATLAB command line by running: 

>> q=kx.c(host,port,username:password) 

This should return a response from the MATLAB command line indicating that the connection has been successful: 

>> q=kx.c("localhost",5001,"admin:admin")

q  =

kx.c@1b73be9f

% This connection can then be closed by running:
>> close(q)

Now that we have established and opened a secure connection to our kdb+ process, we can now perform a basic query by calling the k class from our connection like so: 

>> q.k('2+2')

ans =
     4

Pulling Data Sets and Executing Stored Procedures

The Java API for kdb+ works by converting kdb+ datatypes into Java Objects which can then be read by MATLAB. For example, for dictionaries and tables, these are the ‘Dict’ and ‘Flip’ classes. An example of a query which returns a kdb+ dictionary as a response can be seen below: 

>> q.k('`a`b!(1 2;2 3)')

ans =

kx.c$Dict@4c398c80

This response indicates that our kdb+ dictionary has been received by MATLAB as a Java Object of type ‘kx.c$Dict’. We can use MATLAB’s dot indexing in order to retrieve the key-value pairs from this Java Object. If we define this response as some variable ‘dict’, we can retrieve its keys by using the ‘.x’ dot index: 

>> dict.x

ans = 

  java.lang.String[]:

    'a'
    'b'

Similarly, we can retrieve the values by using the ‘.y’ index : 

>> dict.y

ans =

  java.lang.Object[]:

    [2x1 int64]
    [2x1 int64]

Dictionaries do not exist in MATLAB, its closest neighbour being Map Containers. These are data containers which map values to unique keys similar to kdb+ dictionaries. A simple MATLAB script can be created which converts the ‘kx.c$Dict’ object into these map containers. 

Retrieving table data can be done in a similar way to that of dictionaries by passing a query to our kdb+ process which returns a kdb+ table which is then received by MATLAB as a ‘kx.c$Flip’ Java object. Similar to that of the dictionaries, we can view the keys or column names of this ‘Flip’ data type by using the x dot index and we can observe its values by using the y dot index.

Kdb+ table data can be formatted into MATLAB tables, however it is not straightforward due to the differences in data types between the two. Therefore, it can be helpful to create a script which indexes into the column data from the returned Flip objects and convert their respective Java types into MATLAB data types. 

After running the Flip object through this script, we can observe the data in a more concise way that is readable to MATLAB: 

Tables & Dictionaries

Sending data from MATLAB to kdb+ can be done in a similar manner to how data is received. This can also only be done after a connection has been established. We can call keywords or functions defined on the kdb+ process and pass MATLAB data as arguments. We can also create complex kdb+ objects such as dictionaries and tables, using the ‘Dict’ and ‘Flip’ classes we saw earlier, and then push these over to the kdb+ process. To create these objects we need to use the MATLAB command javaObject in the following format:  

>>javaObject(classname,arguments) 

For example, the Dict class requires two arguments in order to create the Java Object, these being its keys and value pairs. These can be defined like so for example: 

>> keys = {'a','b'}

keys = 

  1x2 cell array

    {'a'}    {'b'}
>> values = {100*rand(1,10),int32(randperm(10))}

values =

  1x2 cell array

    {1x10 double}    {[6 1 7 4 9 5 8 3 10 2]}

We can then create our dictionary object by using the  javaObject command: 

>> Dict=javaObject('kx.c$Dict',keys,values)

Dict =

kx.c$Dict@32193bea

Seeing that this has returned a kdb+ Java Object, we can now define this and send to our kdb+ process by calling the kdb+ “set” keyword and passing the Dictionary object as an argument.  

>> q.k("set", "Dictionary", Dict)

ans = 

    'Dictionary'

If we now look at the kdb+ process, we can see that the object has been received and defined as ‘Dictionary’: 

q)Dictionary
a| 65.57407 3.571168 84.91293 93.39932 67.87352 75.77401 74.31325 39.2227 65...
b| 2        10       4        5        3        8        7        1       6  ..

An example of how this can be done with kdb+ table ‘flip objects’ can be seen on the github repository

Listening to Streamed Data

For the purposes of this blog, AquaQ has created a Java utility which allows MATLAB to act as a listener to streamed data from a kdb+ process such as a vanilla kdb+ tickerplant. We can add the Java files from this utility to our java class path, allowing it to be used in MATLAB. This utility allows for the MATLAB process to subscribe to the kdb+ tickerplant and receive the data continuously in the form of Java Objects

We receive a message from the tickerplant that the listener has been notified and we confirm this by displaying to the screen that we have received a Java Object. The full instructions for how to use this Java utility can be found on the github repository.

We begin by defining the handler as below

>> handler = com.aquaq.kdb.matlab.ticker.TickerEventHandler

After this, we must then define a callback function. An example of a callback function can be seen below:

>> set(handler, 'TickerEventCallBack', @(h,e)disp(tableselect(e.k(3)))) 

The callback function defined above indexes into the tickerplant response in order to access the table data, it then calls a script to format this into a MATLAB table and then display this to the screen. Now we have defined our handler, we can detail our tickerplant configuration by using the command below:

>> f = com.aquaq.jlab.TickerController(host, port, username:password, handler)

We can then start the feed by running the following:

>> f.start() 

This should then receive the data from the kdb+ process and display the data to the screen. The feed can then be stopped by issuing the command:

>> f.stop()

Conclusion

This blog has shown how we can establish a connection from MATLAB to a kdb+ process using the Java API for kdb+ in order to send and receive datasets and complex kdb+ data objects. We have also created a Java utility which can be used to connect to a streaming data source such as a tickerplant to allow for the receiving and processing of continuous data within MATLAB.

If you’re interested in learning more about any of the technologies mentioned in this post or want more information on the interaction between kdb+ and MATLAB, get in touch at info@aquaq.co.uk

Liam O'ConnorMATLAB, kdb+ and Streaming Data