In a data driven world, the ability to access and retrieve data efficiently is critical. This means data needs to be readily available to users, and in the kdb+ world this can present a challenge. kdb+ being somewhat single threaded can result in heavy users acting greedily on data processes. In AquaQ we are hearing more from our clients about …
Automating Data Egress from kdb+ into Google BigQuery
kdb+ tick data capture software has been largely popular in banks, hedge funds and other financial institutions for years. Its kdb+ framework permits incredibly fast querying of data largely due to its vectorised querying language and efficient storage methods. A large cost component in any kdb+ implementation is always storage costs. Storage costs have been a big driving factor towards …
AquaQ’s BigQuery API: Accessing Google’s BigQuery Data in kdb+
Introduction Google’s BigQuery is a fully scalable, serverless, cloud data warehouse of ever-increasing popularity. It boasts impressive ease of use as well as low-cost, quick configuration and “super-fast”[1] SQL querying across data. Making use of worker processes (like kdb+ secondary processes) and Google’s processing infrastructure, BigQuery’s querying speeds tower above many other SQL data warehouses. For a SQL-based storage and …
AquaQuarantine 2021 Tech Talk Content
The videos and slide decks from the AquaQuarantine 2021 Tech Talks are below. Content for the 2020 series can be found here. Shakti Refinitiv Tick History in Google BigQuery Chronicle Queue Vaex, Arrow, Parquet, kdb+ kdb+ Depth Data Storage Formats Shakti In this talk Fintan Quill, Shakti Director of Sales Engineering, will discuss this new parallel data system. With Shakti, database, language, …
AquaQuarantine Tech Talk Series 2021
We’ve decided to bring back our data-oriented talks for another run, with some help from our friends. Please use the sign up form below if you would like to join us. The content for our previous webinars is available here. Vaex, Arrow, Parquet Thursday 11th March, 4pm GMT Vaex is an open source python library for analysing large on-disk datasets …
Level 2 Storage Formats
Storing an order book can be tricky due to the large number of orders and prices that continually change throughout the day. Here we investigate and compare three common methods of storing an order book, assessing their benefits and limitations. To do that, the TorQ-CME pack was used to process CME MDP 3.0 FIX data on Soybean Futures for 10 …
A C++ Utopia – Some comments on interfacing kdb+ and C++
Recently quite a few projects at AquaQ have involved the manipulation of large numeric vectors in kdb+, and passing this data to and from external applications or tools written in C or C++. A practical example would relate to the use of machine learning libraries written in C++ that require output to be written to a kdb+ tickerplant or database. …
Chronicle Software integrates Chronicle Queue with kdb+ – Delivering high speed trading solution in partnership with AquaQ Analytics
London, Belfast, Singapore – 05 February 2021 – Chronicle Software (Chronicle), a specialist provider of low-latency technology solutions for the financial services industry, announced the integration of their latency-optimised messaging framework Chronicle Queue with the high-performance database kdb+. Chronicle has completed the integration project in partnership with AquaQ, the consultancy and product development company. The integration enables users of …
AquaQ Parquet kdb+ converter
Following on from the columnar and data formats blog post, we have decided to investigate creating a custom API for parquet. Traditionally, we would do this through python, however, we are interested to see what the performance benefits are. Kdb-Parquet is a library that can convert kdb+ tables to and from the Apache Parquet table format. The library provides a …
Real-time airline traffic dashboards with TorQ-Air
TorQ-Air is an application built on the TorQ framework that allows us to capture data through the Lufthansa developer API and perform both real-time and historical analytics on flight data. The solution includes a front-end built on Kx-dashboards, showing a number of real-time and historical stats. Download and installation instructions can be found on the TorQ-Air github repo. Founded in …