kdb+ Feedhandler Development

Stephanie Woodings data capture, datablog, kdb, kdb+ Leave a Comment

Setting up a kdb+ database is relatively straight forward.  The TorQ Starter Pack is a good place to start.  Two of the trickier bits are: Estimating how much memory your dataset will require Writing applications (generally called “feedhandlers”) to insert data into the database For (1) we’ve written previously on some database set up utilities used to estimate the size of a dataset …

Stephanie Woodingskdb+ Feedhandler Development

Alteryx and kdb+ Integration

Michael Dougherty datablog, kdb, kdb+ Leave a Comment

Integrating kdb+ via TorQ with Alteryx Designer We decided to take a look at Alteryx (https://www.alteryx.com/), the company, their designer product and also see what is possible when rapidly integrated with TorQ, our open source kdb+ framework. Alteryx is based in Irvine, California, U.S. It offers a data science platform geared toward citizen data scientists. The platform’s self-service data preparation …

Michael DoughertyAlteryx and kdb+ Integration

TorQ – CME Data Processing Add-on

Aidan O'Gorman datablog, kdb, kdb+, TorQ 2 Comments

Our new addition to the TorQ framework presents a method for processing historical data in its native FIX format from the CME, building and maintaining an order book, and writing this data to disk in a variety of approaches suited to query efficiency or space efficiency. Our example data set is FX futures contracts of 11 major currency pairs, but …

Aidan O'GormanTorQ – CME Data Processing Add-on

Cloud kdb+

Jonathon McMurray data capture, datablog, kdb, kdb+ Leave a Comment

When setting up a kdb+ production system, you may not always have access to the hardware you need. In such instances, it can be helpful to turn to “the cloud”. There are a number of potential benefits to using a cloud system instead of a local server. For example, important server-related duties such as security, failover and data redundancy will …

Jonathon McMurrayCloud kdb+

kdb+ Database Setup Utilities

Aidan O'Gorman data capture, datablog, kdb, kdb+, TorQ 1 Comment

This blog post describes tools to help setup a new kdb+ database. The tools allow you to: calculate the expected memory requirements of the database check columns have been typed correctly in order to avoid sym file bloat Background The volume of data that a kdb+ process can store in memory is finite and must be estimated and considered when designing …

Aidan O'Gormankdb+ Database Setup Utilities

TorQ Permission Framework

Ryan McCarron datablog, kdb, kdb+, TorQ Leave a Comment

Our latest update to TorQ adds our new permissions system. Using this system any user connecting over IPC to a permissioned process can have selectively tiered access to any table or variable. This extends to limiting access to functions, through a fully customisable set of user privileges, while still allowing a super-user role with access to all functions, variables, and …

Ryan McCarronTorQ Permission Framework

Free of Charge kdb+ Architecture Workshop NYC

AquaQ Admin data capture, datablog, kdb, kdb+, News, TorQ, training Leave a Comment

We were discussing our launch party in NYC and were kicking around ideas of what to do before it.  Some of us wanted to go to the Metropolitan Museum of Art; some of us wanted to go for a long lunch and maybe a stroll through Central Park; some of us wanted to run a kdb+ Architecture Workshop. So here we are. …

AquaQ AdminFree of Charge kdb+ Architecture Workshop NYC

TorQ User Group

AquaQ Admin data capture, datablog, kdb, kdb+, TorQ Leave a Comment

We have recently created a TorQ user group, for questions and discussions on TorQ (our kdb+ framework).  We will also be using it for more general data capture architecture considerations, and we’ll be publishing coding challenges every two weeks. We hope it will become a useful resource!

AquaQ AdminTorQ User Group

End-of-Day Parallel Sorting in TorQ

AquaQ Admin data capture, datablog, kdb, kdb+, TorQ Leave a Comment

We’ve written some blog posts previously on TorQ End-of-Day and more efficient methods for structuring temporary data on disk.  Until now TorQ has handled the end-of-day sorting or merging of tables sequentially, processing one table at a time.  Additional functionality has been added in the latest release of TorQ (v2.6.2) which will enable the parallel sorting of tables. This will allow …

AquaQ AdminEnd-of-Day Parallel Sorting in TorQ