Integrating kdb+ via TorQ with Alteryx Designer We decided to take a look at Alteryx (https://www.alteryx.com/), the company, their designer product and also see what is possible when rapidly integrated with TorQ, our open source kdb+ framework. Alteryx is based in Irvine, California, U.S. It offers a data science platform geared toward citizen data scientists. The platform’s self-service data preparation …
TorQ Release v3.1 – Kafka, Data Replay and Subscriber Cutoff
We are delighted to announce release v3.1 of TorQ, the latest instalment of our kdb+ framework. The first of our great new additions is kafka.q, which provides q language bindings for the ‘distributed streaming platform’ Apache Kafka, a real time messaging system with persistent storage in message logs. An application architecture built around Kafka could dispense with a tickerplant component, …
Integrating kdb+ with Apache Kafka
We’ve noticed a few clients and listbox/google group members asking questions about Apache Kafka recently, so we decided to take a closer look and add it to TorQ. Apache Kafka bills itself as a ‘distributed streaming platform’ – what that seems to mean is that it is a real time messaging system with persistent storage in message logs. The distributed …
TorQ – CME Data Processing Add-on
Our new addition to the TorQ framework presents a method for processing historical data in its native FIX format from the CME, building and maintaining an order book, and writing this data to disk in a variety of approaches suited to query efficiency or space efficiency. Our example data set is FX futures contracts of 11 major currency pairs, but …
Data Replay and Backtest in TorQ
TorQ has a new utility for replaying historical data into real-time data processes, datareplay.q. Replaying data is usually the first step towards allowing you to backtest. datareplay.q builds a table of upd function calls like those generated by a tickerplant, but instead, using a historical database as the data source, making it simple to test new or existing real-time subscribers …
Cloud kdb+
When setting up a kdb+ production system, you may not always have access to the hardware you need. In such instances, it can be helpful to turn to “the cloud”. There are a number of potential benefits to using a cloud system instead of a local server. For example, important server-related duties such as security, failover and data redundancy will …
New TorQ-FX Released
For anyone who has used TorQ, you will most likely be familiar with the Finance Starter Pack. The starter pack is an example data capture system that is based on randomly generated equity data. It was created with the purpose of showing how to set up an example TorQ installation and how different applications can be built and deployed on …
kdb+ Database Setup Utilities
This blog post describes tools to help setup a new kdb+ database. The tools allow you to: calculate the expected memory requirements of the database check columns have been typed correctly in order to avoid sym file bloat Background The volume of data that a kdb+ process can store in memory is finite and must be estimated and considered when designing …
kdb+ Online Training Updated- Architecture
We recorded our recent kdb+ Architecture Workshop in London, and we’ve added the video and slide deck as a bonus module on our kdb+ Bootcamp Online Training Course. Above is a snapshot – a three hour in-depth workshop in 15 seconds. Topics covered include: the basics of data capture extending kdb+ tick and alternative strategies scaling throughput versus latency removing …
kdb+ Architecture Workshop
Thanks to everyone who attended our kdb+ Architecture Workshop in Code Node, London. We had a full house, and lots of great feedback. If you would like to get more information on future events, or have any topic suggestions for future workshops, please get in touch.