JTools Table of Contents
Project Location: \JTools_StateFileGenerator
NxCore is a whole market-feed processing system. As such, it must process tapes from the start of the day in order to properly populate and maintain internal data structures through all corrections and exchange messages. However, NxCore capable of saving it's internal state at any time and restarting from that point using the saved state Understanding how to effectively generate and use NxCore State file can unlock many doors, specifically those relating to jumping into the tape at any given point in time. Some of the many possibilities include:
- Fast access to all tick data. Using multiple timed state files jumping to and from different times in any given tape becomes trivial.
- Backup for an application that may have crashed and needs to re-enter the real time buffers as quickly as possible at start-up.
In fact, the two most common question we hear from NxCore users both regard how to jump to specific points in time when processing from the NxCore API:
- "I want to jump into the real time buffers right away without running the tape from the beginning of the day."
- "I want to jump to any point in time quickly and begin the stream there."
Even advanced NxCore users may not be aware of the full potential State Files offer.
The State Writer Application is a revision of the JDStateWriter application, which I made available to users some time ago. Since the initial version however, I have modified the application for a numbers of various tasks as well as improve the code in general. I have recently been converted it to the JTools framework and I have included every option I have ever needed with regards to automatic NxCore state file generation.
|
Having state file generation done in a separate, stand-alone application such as this, as opposed to a state file generator imbedded into each NxCore application offers several benefits:
- State file generation is not free. There is a performance hit and if state
files are being generated frequently, this hit may an impact on the
application. By generating the state files in an application who's job is to do
nothing but create state file, this hit can be eliminated from all other NxCore
applications running on the machine.
- If the goal is to provide a recovery point in case an application crashes
and the application does indeed crash, the state file generator will keep
generating current state files. Regardless of how long the application is down
or takes to restart, it will always have a recent point in time to start NxCore
from a state file.
- Avoids the need for multiple state file locations if multiple applications are using state files on the same machine (as duplicate file write collisions will not be an issue).
All of the JTools applications have the ability to load and start from state files. Simply choose the state file as opposed to a full NxCore history file when choosing historical tapes. The state files the application generates will all end with the file extension ".nxs" as opposed to ".nxc" (the extension of NxCore historical tapes).
One Caveat
Because the State file write mechanism is a separate thread. there is one potential hitch when writing State Files; trying to generate a state file while another state file is still in the process of being created. This generates a -12 error (NxAPIERR_SAVE_STATE_IN_PROGRESS). There is no clean way to recover from the error and generate the state file you were intending to within the same NxCore thread.
The problem is rarely encountered when running the State Generator in real time (when it is in the real time buffers) and time is moving rather slow. To take this to an extreme however, consider loading from a historical weekend tape and running it as fast as possible. There is such little data that you could get to the end of the day in one or two minutes (if that). If you were trying to save a state file every 15 minutes (minutes on the tape) in this manner, you will generate -12 errors on multiple state file generation attempts. There has simply not been enough elapsed time for the last state file to be saved completely and the system will not generate another until the save state thread is complete.
If during these situations, it is critical for you to generate every state file as expected, there are 2 options. First, you could build more threads into the main application and relaunch multiple instances to regenerate the state files . Second, you could off-load the task to another application.
I chose the second of these for the state generator application, for a number of reasons:
- The method works *very* well.
- The seperate application could be spawned by multiple applications.
- To demonstrate off-loading NxCore tasks to another application or process entirely. This can open up other doors. While the information flow in the State File Generator is one way, if the spawned process had two way communication to the application that spawned it, a world of possibilities exist.
The code for the JTools_StateSpinner application is imbedded in the JTools_StateFileGenerator folder as \JTools_StateFileGenerator\JTools_StateSpinner\
JTools Table of Contents