AquaQ’s BigQuery API: Accessing Google’s BigQuery Data in kdb+

Matthew Clark kdb+ Leave a Comment

Introduction Google’s BigQuery is a fully scalable, serverless, cloud data warehouse of ever-increasing popularity. It boasts impressive ease of use as well as low-cost, quick configuration and “super-fast”[1] SQL querying across data. Making use of worker processes (like kdb+ secondary processes) and Google’s processing infrastructure, BigQuery’s querying speeds tower above many other SQL data warehouses. For a SQL-based storage and …

Matthew ClarkAquaQ’s BigQuery API: Accessing Google’s BigQuery Data in kdb+

TorQ Data Access API

Dan Sheppard TorQ Leave a Comment

Accessibility is important for all software, it provides a pleasant user experience as well as a seamless interaction with non-native processes. The Data Access API provides TorQ with a sleek user interface as well as accessible and extensive functionality to both the inexperienced kdb+ user and non kdb+ processes. Whilst experienced kdb+ developers will likely prefer to write queries directly, …

Dan SheppardTorQ Data Access API

AquaQuarantine 2021 Tech Talk Content

Jonny Press datablog, kdb+ Leave a Comment

The videos and slide decks from the AquaQuarantine 2021 Tech Talks are below. Content for the 2020 series can be found here. Shakti Refinitiv Tick History in Google BigQuery Chronicle Queue Vaex, Arrow, Parquet, kdb+ kdb+ Depth Data Storage Formats Shakti In this talk Fintan Quill, Shakti Director of Sales Engineering, will discuss this new parallel data system. With Shakti, database, language, …

Jonny PressAquaQuarantine 2021 Tech Talk Content

Comparing Arrow, Parquet and kdb+ File Formats

Michael Turkington data Leave a Comment

We have been investigating comparisons of performance and compression between the kdb+ binary format, Parquet and Arrow. Parquet is a columnar file format widely used in data warehousing, while Arrow is an on-disk format which is supported by some of the most popular data science tools. Some benchmarking has already been performed by the developers of Apache Arrow. We opted to use Python for …

Michael TurkingtonComparing Arrow, Parquet and kdb+ File Formats

AquaQ Coding Challenge: Columnar Transposition Ciphers

Ryan McCarron Uncategorized Leave a Comment

Encryption at its best is nearly unbreakable. At its most fun, it’s flimsy and thousands of years old. Our latest challenge is to break an existing columnar transposition cipher, and the challenge details are located here as challenge 35, “Columns“, on our challenge hub. Once you’ve gotten the spec, send us a function or script to find any codewords which …

Ryan McCarronAquaQ Coding Challenge: Columnar Transposition Ciphers

AquaQuarantine Tech Talk Series 2021

Jonny Press data, datablog, kdb+ Leave a Comment

We’ve decided to bring back our data-oriented talks for another run, with some help from our friends. Please use the sign up form below if you would like to join us. The content for our previous webinars is available here. Vaex, Arrow, Parquet Thursday 11th March, 4pm GMT Vaex is an open source python library for analysing large on-disk datasets …

Jonny PressAquaQuarantine Tech Talk Series 2021

Level 2 Storage Formats

Andrew West data, kdb, kdb+ 2 Comments

Storing an order book can be tricky due to the large number of orders and prices that continually change throughout the day. Here we investigate and compare three common methods of storing an order book, assessing their benefits and limitations. To do that, the TorQ-CME pack was used to process CME MDP 3.0 FIX data on Soybean Futures for 10 …

Andrew WestLevel 2 Storage Formats

Chinese New Year 2021!

Calum McBurney Life At AquaQ Leave a Comment

At AquaQ we weren’t going to let lockdown number (I’ve lost count) stop us from celebrating Chinese New Year. This year is the ‘Year of the Ox’, said to bring stability, calmness, and great opportunities… I think we could all use a bit of that positivity! We’re counting on you, Ox. As we couldn’t celebrate Chinese New Year in person, …

Calum McBurneyChinese New Year 2021!

A C++ Utopia – Some comments on interfacing kdb+ and C++

Matt Hughes data, kdb+ Leave a Comment

Recently quite a few projects at AquaQ have involved the manipulation of large numeric vectors in kdb+, and passing this data to and from external applications or tools written in C or C++. A practical example would relate to the use of machine learning libraries written in C++ that require output to be written to a kdb+ tickerplant or database. …

Matt HughesA C++ Utopia – Some comments on interfacing kdb+ and C++

AquaQ Parquet kdb+ converter

William Lowe kdb+ Leave a Comment

Following on from the columnar and data formats blog post, we have decided to investigate creating a custom API for parquet. Traditionally, we would do this through python, however, we are interested to see what the performance benefits are. Kdb-Parquet is a library that can convert kdb+ tables to and from the Apache Parquet table format. The library provides a …

William LoweAquaQ Parquet kdb+ converter