Gary Davies
For many of us, this month is about beginning the first delivery cycle of the year knowing that we have a lot of work to get through before the December change freeze hits us again. Requirements year on year usually include increasing data volume, throughput, availability and access while not wanting to impact on performance, availability and access. Often kdb+ systems, like all applications, include technical debt and with other high priority deliverables we don’t usually get the time to sit back and properly review our architectures.
This is where we would offer up the chance to board the AquaQ ARK (Architecture Review for kdb+). AquaQ are able to provide independent subject matter experts (SMEs) who can leverage their experience across many installations and to review system architectures, including the non-kdb+ components. In the ARK the SME will come in and provide an assessment for the system looking at opportunities for improvement across the board. This will include, but is not limited to the following:
Changes needed to meet current/increased demand on processes
This can be about investigating the number of users versus the process, looking into scalability or potentially changes to the architecture to meet those demands. Symptoms of current issues can include processes becoming increasingly unavailable or increased client complaints on access. Slow subscribers are also symptomatic of an issue in this area.
Design opportunities, leveraging latest developments in the kdb+ space
kdb+ is a continually changing technology and improvements, and releases, occur quite regularly. We can provide insight into these but also provide advice based on our studies and proof of concepts. We strive to investigate new technologies and their interoperability and opportunities with kdb+. An example of this is our Big Query API which was driven off the need to create an adaptable hybrid architecture for our clients.
Improved system throughput, horizontal and vertical scalability of systems
Scalability of kdb+ can be difficult to handle, it doesn’t scale horizontally out of the box and vertical scalability can lead to its own hardware, access and recovery issues. That being said this is a space we look at regularly for our clients and you will see some recent developments from AquaQ that were born out of solving such issues; the TorQ Segmented Tickerpant which improves recovery of large data sets (vertical) and leveraging kdb+ in containers (horizontal)
System simplifications
As mentioned above, kdb+ systems often inherit technical debt which can make the platform overly complicated which brings about its own problems for enhancing the system. We can take a look at the architecture and provide insight into potential simplifications for the platform with the goal of improving the supportability and ability to enhance the application.
Performance improvements to meet current of improved SLAs
Clients of kdb+ systems always want the queries returned faster, the analytics calculated faster. Our SMEs can come in and review the system with regards to performance and provide insight into potential improvements.
Security and access controls
In this area we can provide insight into improvements, developments in this area and also analyze the system for potential opportunities.
Cloud architecture readiness
Many of our clients are currently at some point on the cloud journey now and from a kdb+ perspective this can still be at the “what do we do” or the “trial implementation” phase. As you will have noted, AquaQ expanded to introduce a new Cloud Stream in 2021. This allows us to combine our Cloud and kdb+ SMEs to provide a review on applications. We can provide insight into which cloud and how a system might be migrated. Cloud can provide new opportunities to improve kdb+ applications but we are also aware that there can be pitfalls. We have already explored, and continue to do so, and therefore can provide this insight as part of the review.
kdb+ upgrades
Our clients and kdb+ applications in general tend to be at different stages along the kdb+ releases. Some like to be leading edge and are on the test versions immediately, others hold a few versions behind and some even lag back further. As each version is released there are opportunities for architecture improvements, but there are also opportunities for issues in upgrading systems. We have the expertise to be able to guide on this and have the experience of doing this for clients for many years.
Application build and deploy processes
Modern architectures now leverage build and deploy pipelines but like cloud kdb+ applications tend to be at different points on this journey. Often this is due to the fact that kdb+ crosses the boundaries between being database and application. We have experience in working in different pipelines and how to improve upon these; from a simple code repo to automated cycles with single click deployment. In 2021 we released our blog looking at CI/CD from the TorQ perspective.
System monitoring
In addtion to the base architecture we can also review the monitoring and data quality aspects of systems. We can provide inside into toolsets and frameworks. This is an area you will have seen a number of releases in the last few years from AquaQ; we had the Data Quality System, Interoperability with Datadog and our Grafana adapter (watch this space for new developments in this area).
On completion of the ARK, our SMEs will provide details on the deficiencies they have seen and potential opportunities.
If this is of interest to you, please reach out to have a chat and we can explore together.
Share this:
