The videos and slide decks from the AquaQuarantine 2021 Tech Talks are below. Content for the 2020 series can be found here.
- Refinitiv Tick History in Google BigQuery
- Chronicle Queue
- Vaex, Arrow, Parquet, kdb+
- kdb+ Depth Data Storage Formats
In this talk Fintan Quill, Shakti Director of Sales Engineering, will discuss this new parallel data system. With Shakti, database, language, connectivity and stream processing are together in one platform. Created by Arthur Whitney, Shakti builds on a singular approach to software design which enables it to capture tens of millions of messages per second in real-time, while simultaneously analyzing and storing trillions of records of historical data.
Interfacing kdb+ with Refinitiv’s TickHistory in Google BigQuery
Financial institutions capture and store market data independently. Each institution maintains, manages and pays for their own infrastructure and staging of data for various lines of business and workloads. Refinitiv, an LSEG Business, have ported their Tick History database of 500+ venues dating back to 1996 to Google BigQuery, allowing clients to access query ready data rather than having to download and process files. This hosted data utility model allows clients to reason over a normalised, point in time dataset and just return the results of their analytics. AquaQ has been working with Refinitiv and Google to develop a single TorQ API that makes this data more accessible to the kdb+ community of financial institution end users, via a cloud hybrid approach to market data. The data behind the TorQ API can be split with some data residing, as it does today, within the financial institution’s kdb+ database, and some data pulled directly from Refinitiv’s hosted Tick History. The TorQ API does the heavy lifting, distributing the query appropriately across the financial institutions kdb+ store and BigQuery.
Sebastian Fuchs (Refinitiv), Martin Bradford (Refinitiv), Gary Davies (AquaQ Analytics) and Matthew Clark (AquaQ Analytics) will discuss and demonstrate the offering.
Chronicle Queue is a persisted low-latency messaging queue. It is utilised across a large number of institutions in latency sensitive applications. Peter Lawrey of Chronicle Software and Mark Friel of AquaQ Analytics will give an overview of the application and demonstrate the open source adaptor to kdb+ that has been developed in partnership.
Vaex, Arrow, Parquet
Vaex is an open source python library for analysing large on-disk datasets which are usually stored in Parquet or Arrow formats. It uses a number of techniques similar to kdb+. Matt Doherty of AquaQ Analytics has been executing some exploratory data analytics using Vaex and will give an overview of the technology and show some comparisons with kdb+ in relation to both querying and data storage formats.
kdb+ Depth Data Storage Formats
Market data only ever increases in granularity and volume. There are different approaches to storage which change both the ease of writing analytics as well as the query performance. Andrew West of AquaQ Analytics will discuss various strategies and optimisations that can be deployed in kdb+ installations.