kdb+ 3.6: anymap

Jonny Press datablog, kdb+ 1 Comment

The recent release of kdb+3.6 brought a new data structure: anymap. As database nerds, this is very exciting news for us. We thought we’d have a play with it to see how it could be used. New Nested Lists The anymap features replaces all the previous structures used to create nested lists. It also supports storing non-uniform values within the …

Jonny Presskdb+ 3.6: anymap

kdb+ IoT Database Structure

Jonny Press data capture, datablog, kdb+ Leave a Comment

When implementing a kdb+ database the partitioning structure is a key decision point. kdb+ provides four in-built ways to partition database tables: by year, month, date or integer. Only one partitioning scheme may be used. The best option is usually based on two factors: how data arrives how it is accessed The partitioning scheme should facilitate easy addition of data …

Jonny Presskdb+ IoT Database Structure

As You Bike It: Mapping Belfast Bike Usage

Ryan McCarron data capture, datablog, kdb, kdb+, TorQ Leave a Comment

We recently noted the ability to use our TorQ package to collect data on our local Belfast Bikes transport network. So how does our cycle network look after collecting for a while? We recorded data on the status of each station in the network for 6 months, and we can now use this data to observe how the population of …

Ryan McCarronAs You Bike It: Mapping Belfast Bike Usage

kdb+ Feedhandler Development

Stephanie Woodings data capture, datablog, kdb, kdb+ Leave a Comment

Setting up a kdb+ database is relatively straight forward.  The TorQ Starter Pack is a good place to start.  Two of the trickier bits are: Estimating how much memory your dataset will require Writing applications (generally called “feedhandlers”) to insert data into the database For (1) we’ve written previously on some database set up utilities used to estimate the size of a dataset …

Stephanie Woodingskdb+ Feedhandler Development

Alteryx and kdb+ Integration

Michael Dougherty datablog, kdb, kdb+ Leave a Comment

Integrating kdb+ via TorQ with Alteryx Designer We decided to take a look at Alteryx (https://www.alteryx.com/), the company, their designer product and also see what is possible when rapidly integrated with TorQ, our open source kdb+ framework. Alteryx is based in Irvine, California, U.S. It offers a data science platform geared toward citizen data scientists. The platform’s self-service data preparation …

Michael DoughertyAlteryx and kdb+ Integration

TorQ – CME Data Processing Add-on

Aidan O'Gorman datablog, kdb, kdb+, TorQ 2 Comments

Our new addition to the TorQ framework presents a method for processing historical data in its native FIX format from the CME, building and maintaining an order book, and writing this data to disk in a variety of approaches suited to query efficiency or space efficiency. Our example data set is FX futures contracts of 11 major currency pairs, but …

Aidan O'GormanTorQ – CME Data Processing Add-on

Cloud kdb+

Jonathon McMurray data capture, datablog, kdb, kdb+ Leave a Comment

When setting up a kdb+ production system, you may not always have access to the hardware you need. In such instances, it can be helpful to turn to “the cloud”. There are a number of potential benefits to using a cloud system instead of a local server. For example, important server-related duties such as security, failover and data redundancy will …

Jonathon McMurrayCloud kdb+