Setting up a kdb+ database is relatively straight forward. The TorQ Starter Pack is a good place to start. Two of the trickier bits are: Estimating how much memory your dataset will require Writing applications (generally called “feedhandlers”) to insert data into the database For (1) we’ve written previously on some database set up utilities used to estimate the size of a dataset …
Alteryx and kdb+ Integration
Integrating kdb+ via TorQ with Alteryx Designer We decided to take a look at Alteryx (https://www.alteryx.com/), the company, their designer product and also see what is possible when rapidly integrated with TorQ, our open source kdb+ framework. Alteryx is based in Irvine, California, U.S. It offers a data science platform geared toward citizen data scientists. The platform’s self-service data preparation …
TorQ – CME Data Processing Add-on
Our new addition to the TorQ framework presents a method for processing historical data in its native FIX format from the CME, building and maintaining an order book, and writing this data to disk in a variety of approaches suited to query efficiency or space efficiency. Our example data set is FX futures contracts of 11 major currency pairs, but …
Cloud kdb+
When setting up a kdb+ production system, you may not always have access to the hardware you need. In such instances, it can be helpful to turn to “the cloud”. There are a number of potential benefits to using a cloud system instead of a local server. For example, important server-related duties such as security, failover and data redundancy will …
kdb+ Database Setup Utilities
This blog post describes tools to help setup a new kdb+ database. The tools allow you to: calculate the expected memory requirements of the database check columns have been typed correctly in order to avoid sym file bloat Background The volume of data that a kdb+ process can store in memory is finite and must be estimated and considered when designing …
TorQ Permission Framework
Our latest update to TorQ adds our new permissions system. Using this system any user connecting over IPC to a permissioned process can have selectively tiered access to any table or variable. This extends to limiting access to functions, through a fully customisable set of user privileges, while still allowing a super-user role with access to all functions, variables, and …
Free of Charge kdb+ Architecture Workshop NYC
We were discussing our launch party in NYC and were kicking around ideas of what to do before it. Some of us wanted to go to the Metropolitan Museum of Art; some of us wanted to go for a long lunch and maybe a stroll through Central Park; some of us wanted to run a kdb+ Architecture Workshop. So here we are. …
TorQ User Group
We have recently created a TorQ user group, for questions and discussions on TorQ (our kdb+ framework). We will also be using it for more general data capture architecture considerations, and we’ll be publishing coding challenges every two weeks. We hope it will become a useful resource!
TorQ Data Capture in Two Minutes
How to set up a production-ready kdb+ based market data capture system in under two minutes using TorQ.
End-of-Day Parallel Sorting in TorQ
We’ve written some blog posts previously on TorQ End-of-Day and more efficient methods for structuring temporary data on disk. Until now TorQ has handled the end-of-day sorting or merging of tables sequentially, processing one table at a time. Additional functionality has been added in the latest release of TorQ (v2.6.2) which will enable the parallel sorting of tables. This will allow …
- Page 1 of 2
- 1
- 2