TorQ in 2020

2020 was a strange year for most of us. We’ve used it as an excuse to put a lot of work into extending TorQ- here’s a brief recap of some highlights.

The first large contribution in 2020 was to provide the option of integrating Datadog into TorQ. Datadog is an extremely useful application monitoring service capable of tracking anything from servers to databases and also visualising analytics on dashboards. As a result, it has never been easier to monitor the health of your stacks!

TorQ 3.6.0 updated the framework to be compatible with kdb+ 4.0. As usual we like to leverage new features where we can, and we like the code profiler wrapped into kdb+ 4.0.

May 2020 saw the inclusion of TorQ’s Data Quality System (DQS). The DQS is a set of four processes which work to automatically and comprehensively monitor the data entering your TorQ stack. Furthermore, the DQS may be coupled with a monitoring tool such as Datadog to provide even further insight.

It is now quicker and easier than ever to download and install TorQ thanks to our new installation script! This way you can spend less time worrying about deployments, versioning and rollbacks and spend more time on the important stuff.

Our most recent enhancements of TorQ have been a brand new test framework based on k4 unit and a new segmented tickerplant. TorQ’s new testing framework aims to provide a way of unit testing extensive areas of code both rigorously and consistently. As a result, tests are much more readable as well as being easier to create and amend. Additionally, the segmented tickerplant has been added to provide a more flexible and customisable tickerplant process, especially regarding logging and subscription.

Along with extensions to the TorQ framework, this year we have also released a new Grafana adapter for kdb+ which offers lower overhead, higher throughput and better support for the visualisation of large datasets. This adapter is packaged as a simple plugin which is listed on the Grafana plugin directory.

There’s plenty lined up for 2021 as well…

We are working on a data access API for TorQ to allow seamless querying across kdb+ processes (on disk or in-memory) whilst optimizing for the properties of the underlying tables. We are extending it to also interface with Google BigQuery which will allow users to query both kdb+ and BigQuery data through the TorQ gateway. This will be extremely useful for users with large datasets that would like to take advantage of the query performance of kdb+ whilst reducing the internal storage costs of data. The BigQuery interface will be extendable to other Cloud Data Warehouses as well as traditional databases.

If you would like more information regarding the 2020 additions or on other AquaQ projects, please get in touch.