TorQ Packaging

TorQ is our open source kdb+ framework.  We have been working with many clients over the last two years building bespoke systems on top of the core TorQ package.  Something that is not immediately clear is how a custom application built on TorQ should be structured and deployed.  We have made some changes to core TorQ over the last 6 months to make life easier.  This blog outlines those changes and how a custom application should be structured.

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Package Structure[/custom_headline]

In order to run a package with TorQ it is usually easiest to install the package on top of TorQ itself. For example, if we wish to install the Finance Starter Pack, it can be unzipped into a directory holding TorQ.  It can also be installed separately, though that may lead to additional configuration changes.

File Structure of unmerged TorQ Finance Starter Pack and TorQ

By copying the application, in this case the Finance Starter Pack, over base TorQ, folders with common names will merge, and files with common names will be overwritten. This, however, can be used to our advantage. For example, code that is available for all processes is stored in a folder named “common” within the code directory. In order to add further functionality to all processes, a “common” folder should be created within the application that stores the new code. This then merges with the original folder when copied over to TorQ. Conversely, if a file within the application’s common folder already exists in TorQ’s common folder, the file is overwritten. From the above image we can see that the setenv.sh file is common to both TorQ and the Finance Starter Pack and will be overwritten in TorQ during the merge. This can be beneficial if we wish to add functionality to the pre-existing code and make it an extension of the previous or to rewrite it from scratch. However, it is advisable that name clashes with files are avoided.

File structure of merged TorQ starter pack and TorQ, with similar files and folders highlighted in blue

It is worth highlighting that there aren’t any files or folders that are required for an application to work, but without them, TorQ will only be base TorQ and not actually “do” anything. It’s usual to create a start and stop script to make the processes run and exit appropriately.  Additionally, the creation of a directory holding the config specifically for the application can be incredibly useful, and is further explained in the Config Management section.

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Managing Deployments[/custom_headline]

For smaller installations it can make sense to use the version control system to help manage releases. This usually entails creating two local checkouts from version control – one for the base TorQ installation and another for the application specific package (at appropriate release tags). These are then merged into a common directory e.g.

cp –r TorQ/* deploy/
cp –r TorQ-Finance-Starter-Pack/* deploy/

Switching between releases for either TorQ or the application can be done by switching to the correct release tag in the version control system and recopying or recreating the deploy directory.

Ideally, the deploy directory should be easy to destroy and recreate. This means that it shouldn’t contain any data that should be preserved. As such, some of the standard environment variables may need repointed to locations outside the deployment directory e.g. $KDBLOG or $KDBHDB.

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Config Management[/custom_headline]

TorQ uses environment variables for managing configuration directories, meaning the configuration settings can be placed anywhere on the file system. These are defined by default in the setenv.sh script. $KDBCONFIG defines the main configuration directory, which defaults to “config” in the TorQ Finanace Starter Pack. $KDBAPPCONFIG defines the application specific configuration directory, which defaults to “appconfig” in the TorQ Finance Starter Pack, i.e.

export KDBCONFIG=./config
export KDBAPPCONFIG=./appconfig

By using environment variables the config and appconfig directories can exist anywhere and can be renamed as desired provided both $KDBCONFIG and $KDBAPPCONFIG are defined with the correct path in the setenv.sh file.

The purpose of $KDBCONFIG is to contain the base configuration settings for TorQ standard processes. For an application, however, it may be required to have additional or different configurations than the ones provided in the base package for TorQ standard or application specific processes. These details should be stored within a folder that is pointed to by the $KDBAPPCONFIG environment variable. TorQ then has the ability to make use of the function getconfigfile which allows it to read a configuration file from either $KDBCONFIG or $KDBAPPCONFIG, with preference towards $KDBAPPCONFIG if the file exists in both. In other words, if a file called example.q exists in $KDBCONFIG, which is defined as the config directory, and also in $KDBAPPCONFIG, defined as the appconfig directory, then it will be read from appconfig. Alternatively, if the file cannot be found in appconfig it will be read from config.

Contents of config & appconfig. The files highlighted in blue in config have less priority than their counterparts in appconfig. These files will be read from appconfig, not config.

The application configuration folder itself, should contain two directories, those being passwords and settings. The former will contain the username-password combinations required for access to each individual process, whilst the latter will hold specific settings for each process.

A custom settings file e.g. gateway.q, only needs to contain the variables that will change from their standard settings. This is because all of the settings files in the settings folder will be read from $KDBCONFIG and subsequently read from $KDBAPPCONFIG. In turn, if the same settings file exists in both of these folders, the settings in $KDBAPPCONFIG will override those in $KDBCONFIG. For example, if you wanted to add an additional process that connects to the gateway, such as the sandboxhdb, the CONNECTIONS variable in $KDBCONFIG/settings/gateway.q would remain as

/ extract from config/settings/gateway.q
\d .servers
CONNECTIONS:`rdb`hdb

but in $KDBAPPCONFIG/settings/gateway.q would change to:

/ example from appconfig/settings/gateway.q
\d .servers
CONNECTIONS:`rdb`hdb`sandboxhdb

In other words, the content of the $KDBAPPCONFIG settings will take priority over those in $KDBCONFIG. TorQ attempts to load settings files in the order:

$KDBCONFIG/settings/default.q
$KDBCONFIG/settings/<proctype>.q
$KDBCONFIG/settings/<procname>.q
$KDBAPPCONFIG/settings/default.q
$KDBAPPCONFIG/settings/<proctype>.q
$KDBAPPCONFIG/settings/<procname>.q

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Adding a New Process[/custom_headline]

Adding new code and processes to TorQ should be be straightforward. Using TorQ Finance Starter Pack as an example, to add a new process a new entry should be made in the start and stop scripts. The start line should look something like this:

nohup q torq.q -load <path to process code> ${KDBSTACKID} -proctype <process type> -procname <process name> <...Other options here...> </dev/null >$KDBLOG/<log file> 2>&1 &

Where the description in bold should be replaced by the relevant code. Within the end script the process could be successfully stopped using the following format.

q torq.q -load code/processes/kill.q -proctype kill -procname killtick -.servers.CONNECTIONS <process name>  </dev/null >$KDBLOG/torqkill.txt 2>&1 &

Secondly, within the $KDBAPPCONFIG folder, the process should be added to the process.csv file. This provides the details regarding the running of the process and should have the format:

host,port,proctype,procname

Typically the new process’ port number is equal to {KDBBASEPORT}+n+1, where n is the number of pre-existing processes. The proctype and procname should be identical to those used in the start script. This allows the process to work out which port to run on and its procname/proctype.

If the process configuration differs from the default configuration, files can be created in $KDBAPPCONFIG/settings. Configuration specific for that process type is stored in $KDBAPPCONFIG/settings/<proctype>.q and configuration overrides for specific instances in $KDBAPPCONFIG/settings/<procname>.q.

The process itself can be saved anywhere, provided the correct path is given on startup, however, for clarity it tends to be saved in the directory $KDBCODE/processes to be merged with TorQ’s processes folder.

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Making a Process Load a Specific File or Directory[/custom_headline]

A process can load specific code files or directories of code files in a number of different ways.  kdb+ itself only has the capability to load one script upon startup, therefore in order to load more than one file we can make use of the -load and -loaddir command line flags. -load allows a number of files to be loaded by specifying the path to each in the order that we wish to load them. Below we can see q loading the torq.q script and the additional rdb.q script using the load flag.

q torq.q -load code/processes/rdb.q

The -loaddir command line flag can be used to load directories of files. This is particularly useful if a number of processes need access to the same set of files. These should therefore be stored in a separate directory with an optional order.txt file specifying the load order. This text file should simply list the filenames on separate lines in order of preference. Again below, we can see q loading the torq.q process and all additional files within the hdb directory.

q torq.q -loaddir code/hdb

Sometimes it may be necessary to load files manually within the code itself. The functionality for this already exists within TorQ and can be achieved by using the  and .proc.loaddir functions with the path provided as a string. The advantage of using these above the standard \l is that the loading is logged and error trapped.

In addition, each process can be set to load a directories of code defined as $KDBCODE/<proctype>.q and $KDBCODE/<procname>.q. This must be enabled in the config by setting .proc.loadprocesscode and .proc.loadnamecode to appropriate Boolean values.

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Disabling Standard Loading[/custom_headline]

On standard start-up, TorQ loads the contents of the common folder and the handlers folder. These contain code that is common to all processes. In certain circumstances you may wish to change the order of which these files load. This can be done be switching off .proc.loadcommoncode and .proc.loadhandlers, that is, by setting them equal to 0b in a config file loaded by that process. The files can then be loaded using the command line argument -loaddir or manually using .proc.loaddir within the code, as previously described.

[custom_headline type=”left” level=”h4″ looks_like=”h4″]Conclusion[/custom_headline]
We’ve made quite a lot of improvements to TorQ over the last two years. If you would like to give us some feedback, suggest further improvements, or need help with a kdb+ installation (TorQ or otherwise), then please contact us.