Kafka with kdb+ and Java

Paul Doherty kdb, kdb+ Leave a Comment

Kafka is a distributed messaging service used to achieve a low-latency, high-throughput transfer of data using persistent logs to prevent loss of data. This blog will: Illustrate the various components used in Kafka. Show how Kafka can be used to share data between different technologies, for example, a Java producer and a kdb+ consumer. Highlight a use case for Kafka …

Paul DohertyKafka with kdb+ and Java

Grafana kdb+ Adaptor Update

Dylan Creighton kdb, kdb+ 1 Comment

One of the most requested features to our kdb+ Grafana adaptor was the ability to create panels from the results of server side functions. This most recent update to the adaptor continues to build on the SimpleJSON datasource created by Grafana by working around the limitation of the single drop down menu for queries, whilst also allowing parameters to be supplied. The adaptor is also integrated into TorQ …

Dylan CreightonGrafana kdb+ Adaptor Update

Grafana – kdb+ adaptor

Fiona Morgan kdb, kdb+ 2 Comments

AquaQ Analytics are often involved in projects using different visualisation tools connected to kdb+.  We have recently spent some time reviewing and then implementing a simple Grafana-kdb+ adaptor. This open-source script has been developed using the inbuilt Grafana SimpleJSON datasource and comes with the capability of table and time series visualisation and manipulation. Instructions for installation and usage can be …

Fiona MorganGrafana – kdb+ adaptor

AquaQ releases q-REST Web Interface

Michael McParland kdb, kdb+, TorQ Leave a Comment

AquaQ Analytics is pleased to announce the release of q-REST v1.0 – a RESTful web interface for kdb+. This open source package has been designed as a convenient way for web clients to make requests to a kdb+ instance using the JSON format. The package can be easily integrated with Swagger UI, Docker and standard authentication methods such as SSO, allowing users to easily implement the …

Michael McParlandAquaQ releases q-REST Web Interface

TorQ Release v3.3 – kdb+ 3.6

Karan Patel kdb, kdb+, TorQ Leave a Comment

We are delighted to announce release v3.3 of TorQ, the latest installment of our kdb+ framework; which can be found here. The TorQ v3.3 release has upgraded to run with kdb+ 3.6, whilst maintaining backward compatibility with prior versions.  The main changes are outlined below. Concurrent Synchronous Requests As part of this update, the gateway in TorQ supports the deferred response …

Karan PatelTorQ Release v3.3 – kdb+ 3.6

As You Bike It: Mapping Belfast Bike Usage

Ryan McCarron data capture, datablog, kdb, kdb+, TorQ Leave a Comment

We recently noted the ability to use our TorQ package to collect data on our local Belfast Bikes transport network. So how does our cycle network look after collecting for a while? We recorded data on the status of each station in the network for 6 months, and we can now use this data to observe how the population of …

Ryan McCarronAs You Bike It: Mapping Belfast Bike Usage

kdb+ Feedhandler Development

Stephanie Woodings data capture, datablog, kdb, kdb+ Leave a Comment

Setting up a kdb+ database is relatively straight forward.  The TorQ Starter Pack is a good place to start.  Two of the trickier bits are: Estimating how much memory your dataset will require Writing applications (generally called “feedhandlers”) to insert data into the database For (1) we’ve written previously on some database set up utilities used to estimate the size of a dataset …

Stephanie Woodingskdb+ Feedhandler Development

Alteryx and kdb+ Integration

Michael Dougherty datablog, kdb, kdb+ 1 Comment

Integrating kdb+ via TorQ with Alteryx Designer We decided to take a look at Alteryx (https://www.alteryx.com/), the company, their designer product and also see what is possible when rapidly integrated with TorQ, our open source kdb+ framework. Alteryx is based in Irvine, California, U.S. It offers a data science platform geared toward citizen data scientists. The platform’s self-service data preparation …

Michael DoughertyAlteryx and kdb+ Integration

TorQ Release v3.1 – Kafka, Data Replay and Subscriber Cutoff

Jamie Grant data capture, kdb, kdb+, TorQ 1 Comment

We are delighted to announce release v3.1 of TorQ, the latest instalment of our kdb+ framework. The first of our great new additions is kafka.q, which provides q language bindings for the ‘distributed streaming platform’ Apache Kafka, a real time messaging system with persistent storage in message logs. An application architecture built around Kafka could dispense with a tickerplant component, …

Jamie GrantTorQ Release v3.1 – Kafka, Data Replay and Subscriber Cutoff

Integrating kdb+ with Apache Kafka

Jamie Grant kdb, kdb+ Leave a Comment

We’ve noticed a few clients and listbox/google group members asking questions about Apache Kafka recently, so we decided to take a closer look and add it to TorQ. Apache Kafka bills itself as a ‘distributed streaming platform’ – what that seems to mean is that it is a real time messaging system with persistent storage in message logs. The distributed …

Jamie GrantIntegrating kdb+ with Apache Kafka