kdb+ 3.6: anymap
The recent release of kdb+3.6 brought a new data structure: anymap. As database nerds, this is very exciting news for us. We thought we’d have a play with it to see how it could be used. New Nested Lists The anymap features replaces all the previous structures used to create nested lists. It also supports storing non-uniform values within the …
kdb+ IoT Database Structure
When implementing a kdb+ database the partitioning structure is a key decision point. kdb+ provides four in-built ways to partition database tables: by year, month, date or integer. Only one partitioning scheme may be used. The best option is usually based on two factors: how data arrives how it is accessed The partitioning scheme should facilitate easy addition of data …
As You Bike It: Mapping Belfast Bike Usage
We recently noted the ability to use our TorQ package to collect data on our local Belfast Bikes transport network. So how does our cycle network look after collecting for a while? We recorded data on the status of each station in the network for 6 months, and we can now use this data to observe how the population of …
kdb+ Feedhandler Development
Setting up a kdb+ database is relatively straight forward. The TorQ Starter Pack is a good place to start. Two of the trickier bits are: Estimating how much memory your dataset will require Writing applications (generally called “feedhandlers”) to insert data into the database For (1) we’ve written previously on some database set up utilities used to estimate the size of a dataset …
Alteryx and kdb+ Integration
Integrating kdb+ via TorQ with Alteryx Designer We decided to take a look at Alteryx (https://www.alteryx.com/), the company, their designer product and also see what is possible when rapidly integrated with TorQ, our open source kdb+ framework. Alteryx is based in Irvine, California, U.S. It offers a data science platform geared toward citizen data scientists. The platform’s self-service data preparation …
TorQ – CME Data Processing Add-on
Our new addition to the TorQ framework presents a method for processing historical data in its native FIX format from the CME, building and maintaining an order book, and writing this data to disk in a variety of approaches suited to query efficiency or space efficiency. Our example data set is FX futures contracts of 11 major currency pairs, but …
Cloud kdb+
When setting up a kdb+ production system, you may not always have access to the hardware you need. In such instances, it can be helpful to turn to “the cloud”. There are a number of potential benefits to using a cloud system instead of a local server. For example, important server-related duties such as security, failover and data redundancy will …