TorQ Roadmap 2022

Jonny Press cloud, kdb+, TorQ

We are currently assisting a number of clients with the move towards cloud, and cloud capabilities feature heavily in the development pipeline for TorQ, our kdb+ production framework. The move to cloud can be a “lift-and-shift” approach, or it can be an opportunity to re-engineer the solution. The case for re-engineering parts or all of an incumbent solution are strong. …

Jonny PressTorQ Roadmap 2022

CI/CD & TorQ-Pipeline

Liam O'Connor kdb+, TorQ

Many application development lifecycles have moved to more modern cycle approaching full automation. The purpose of this blog is to demonstrate TorQ-Pipeline, an example using TorQ of a modern kdb+ CI/CD (Continuous Integration/Continuous Delivery/Deployment) pipeline. Before DevOps culture, the software cycle was heavily siloed. The goal of CI/CD is to break down the barriers between the different teams involved in …

Liam O'ConnorCI/CD & TorQ-Pipeline

MATLAB, kdb+ and Streaming Data

Liam O'Connor kdb+

Introduction Recent projects at AquaQ have required the ability to receive streaming data to MATLAB from a vanilla kdb+ tickerplant. The aim therefore was to create a simple method for this with the ability to work over as many platforms as possible. The examples included in this blog use the R2021a version of MATLAB and kdb+ 4.0. Java API for kdb+  …

Liam O'ConnorMATLAB, kdb+ and Streaming Data

Our database is currently busy, please hold…

Gary Davies kdb+

In a data driven world, the ability to access and retrieve data efficiently is critical. This means data needs to be readily available to users, and in the kdb+ world this can present a challenge. kdb+ being somewhat single threaded can result in heavy users acting greedily on data processes. In AquaQ we are hearing more from our clients about …

Gary DaviesOur database is currently busy, please hold…

Automating Data Egress from kdb+ into Google BigQuery

Matthew Clark data capture, kdb+, TorQ Leave a Comment

kdb+ tick data capture software has been largely popular in banks, hedge funds and other financial institutions for years. Its kdb+ framework permits incredibly fast querying of data largely due to its vectorised querying language and efficient storage methods. A large cost component in any kdb+ implementation is always storage costs. Storage costs have been a big driving factor towards …

Matthew ClarkAutomating Data Egress from kdb+ into Google BigQuery

AquaQ’s BigQuery API: Accessing Google’s BigQuery Data in kdb+

Matthew Clark kdb+ Leave a Comment

Introduction Google’s BigQuery is a fully scalable, serverless, cloud data warehouse of ever-increasing popularity. It boasts impressive ease of use as well as low-cost, quick configuration and “super-fast”[1] SQL querying across data. Making use of worker processes (like kdb+ secondary processes) and Google’s processing infrastructure, BigQuery’s querying speeds tower above many other SQL data warehouses. For a SQL-based storage and …

Matthew ClarkAquaQ’s BigQuery API: Accessing Google’s BigQuery Data in kdb+

AquaQuarantine 2021 Tech Talk Content

Jonny Press datablog, kdb+ Leave a Comment

The videos and slide decks from the AquaQuarantine 2021 Tech Talks are below. Content for the 2020 series can be found here. Shakti Refinitiv Tick History in Google BigQuery Chronicle Queue Vaex, Arrow, Parquet, kdb+ kdb+ Depth Data Storage Formats Shakti In this talk Fintan Quill, Shakti Director of Sales Engineering, will discuss this new parallel data system. With Shakti, database, language, …

Jonny PressAquaQuarantine 2021 Tech Talk Content

AquaQuarantine Tech Talk Series 2021

Jonny Press data, datablog, kdb+ Leave a Comment

We’ve decided to bring back our data-oriented talks for another run, with some help from our friends. Please use the sign up form below if you would like to join us. The content for our previous webinars is available here. Vaex, Arrow, Parquet Thursday 11th March, 4pm GMT Vaex is an open source python library for analysing large on-disk datasets …

Jonny PressAquaQuarantine Tech Talk Series 2021

Level 2 Storage Formats

Andrew West data, kdb, kdb+ 2 Comments

Storing an order book can be tricky due to the large number of orders and prices that continually change throughout the day. Here we investigate and compare three common methods of storing an order book, assessing their benefits and limitations. To do that, the TorQ-CME pack was used to process CME MDP 3.0 FIX data on Soybean Futures for 10 …

Andrew WestLevel 2 Storage Formats

A C++ Utopia – Some comments on interfacing kdb+ and C++

Matt Hughes data, kdb+ Leave a Comment

Recently quite a few projects at AquaQ have involved the manipulation of large numeric vectors in kdb+, and passing this data to and from external applications or tools written in C or C++. A practical example would relate to the use of machine learning libraries written in C++ that require output to be written to a kdb+ tickerplant or database. …

Matt HughesA C++ Utopia – Some comments on interfacing kdb+ and C++