We have been investigating comparisons of performance and compression between the kdb+ binary format, Parquet and Arrow. Parquet is a columnar file format widely used in data warehousing, while Arrow is an on-disk format which is supported by some of the most popular data science tools. Some benchmarking has already been performed by the developers of Apache Arrow. We opted to use Python for …
AquaQ Analytics partners with Kinsetsu to bring combined offerings to public and private sector organisations
BELFAST, ANTRIM, NORTHERN IRELAND, March 4, 2021 /EINPresswire.com/ — Belfast-headquartered Big Data and Real-Time Analytics company AquaQ Analytics has formed a strategic partnership with Kinsetsu, a disrupter in Intelligent Tracking solutions, to enhance both company’s offerings to public and private sector organisations globally. Client needs around real time data capture and analytics continues to grow as data within the organisation …
AquaQuarantine Tech Talk Series 2021
We’ve decided to bring back our data-oriented talks for another run, with some help from our friends. Please use the sign up form below if you would like to join us. The content for our previous webinars is available here. Vaex, Arrow, Parquet Thursday 11th March, 4pm GMT Vaex is an open source python library for analysing large on-disk datasets …
Level 2 Storage Formats
Storing an order book can be tricky due to the large number of orders and prices that continually change throughout the day. Here we investigate and compare three common methods of storing an order book, assessing their benefits and limitations. To do that, the TorQ-CME pack was used to process CME MDP 3.0 FIX data on Soybean Futures for 10 …
A C++ Utopia – Some comments on interfacing kdb+ and C++
Recently quite a few projects at AquaQ have involved the manipulation of large numeric vectors in kdb+, and passing this data to and from external applications or tools written in C or C++. A practical example would relate to the use of machine learning libraries written in C++ that require output to be written to a kdb+ tickerplant or database. …
AquaQ Analytics and Pegasus Partner to provide Client Value
Capital markets relies on massive data sets that are exponentially growing and moving faster. The key to making use of the data to create information is through smart analytics, delivered in a cost effective and modular way. AquaQ Analytics and Pegasus Enterprise Solutions are teaming together to provide dynamic, cost effective access to your data that will yield tremendous insights …
Comparing columnar data formats: Arrow, Vaex and kdb+
Here at AquaQ we have years of experience using kdb+ in the financial sector, helping our clients capture, analyse and derive value from their data. We always take a keen interest in technologies that will help us solve new problems, or provide better solutions for existing problems. To this end Apache Arrow is very interesting. Arrow describes itself as a …