JTOOLS
Hello and welcome. My name is Jeffrey Donovan and I am a senior software
engineer at Nanex, Inc.. I have been with Nanex since 2009 and previous to
that, I was a Nanex customer writing commercial applications based on NxCore,
most of which are still in use today.
During the course of developing with NxCore, I have written dozens of
stand-alone applications, both small and large (and am continually writing
more). Many of these are used on a daily basis by us at Nanex to quickly tackle
customer support issues, while others are applications more in line with
real-time use of NxCore (such as quote monitors, stock charting, etc).
Beginning in May/June of 2012 I put some final touches on a few of these apps
and designed each to work under an identical framework. I am making the
applications and the full source code available to all NxCore developers and if
you would like to receive one or all of them please email support@nanex.net and
ask for the 'JTools' library (yes JTools stands for Jeff Tools). Please include
your name and current Nanex account details. I will be making more of these
applications available as time permits.
The applications were designed to be robust, yet simple and easy to understand
by any intermediate level programmer (or beginning programmer with a little
extra effort). I am a believer in minimal abstraction and the code should
reflect this fact. As a Nanex customer you are free to modify and use the code
in your own applications as you see fit and should you make improvements to
these apps, I (we) would love to see them!
Volume I - Support Utilities, Data Extraction
 |
A Generic Shell Application for running NxCore. This application features a
completely detachable GUI, logging, real time simulation, delayed simulation
and a variety of other options. This is the application I usually start with to
build any new NxCore based application. It has served me very well. |
|
Along with the Generic Shell Application are included two
spin-off projects to demonstrate two core principles: |
 |
| JTools_ShellApp_Console |
Date Added: 07/04/2012 Last Date Modified: 07/04/2012 |
Demonstrates the detached GUI and how easy it is to un-attach it or pump
another type of GUI (such as a console application): |
 |
| JTools_ShellApp_Dual |
Date Added: 07/04/2012 Last Date Modified: 07/04/2012 |
Demonstrates how the shell application's interface uses/modifies only data
passed though the NxCore UserData pointer and as such, spinning multiple
instances of the NxCore within a single application becomes trivial. |
|
The Category Viewer provides easy viewing of all category messages as they
occur. Numerous filters and options make drilling down for a specific message
easy. We use the application on a daily basis to help in solving/debugging
client problems with category messages. |
|
The Message Tracker provides easy viewing of all messages as they occur, with
extreme detail on Trade, ExgQuote and MMQuote messages. Numerous filters and
options make drilling down for a specific message easy. As with the Category
Viewer, we use the application on a daily basis to help in solving/debugging
client problems with several message types. |
 |
The MSG Rate / Performance Monitor measures throughput of messages through
NxCore/end user application. It can be extremely valuable in determining what
historical throughput should be expected on any machine, as well as monitoring
real time performance. Performance is displayed in a chart similar to the
bandwidth chart in NxAccess. |
 |
The State Writer Application is a revision of the JDStateWriter application,
which I made available to users some time ago. Since the initial version
however, I have modified the application for a numbers of various tasks as well
as improve the code in general. I have recently been converted it to the JTools
framework and I have included every option I have ever needed with regards to
automatic NxCore state file generation. |
 |
The Time and Sales State Reader Application is a working time and sales viewer
that demonstrates the use of NxCore State Files and the
JTools_StateFileGenerator application. Instantly jump from time frame to time
frame over one or multiple dates in NxCore. |
 |
The JTools_TSFileGenerator is a command line console application driven by the
arguments passed into it. It allows you to dump time and sales reports for any
single symbol in almost any fashion you desire. Every field available for
NxCore Trade, ExgQuote and MMQuote messages is offered as possible output.
Fields generated and field order are determined by the usage strings passed
into the application arguments (see below for a complete list of possible
fields). It will dump the output to the console (if specified to) and to a text
file, the name of which is also passed in as an argument. |
|
|
Also included are 2 small projects that provide a GUI
interface to launch the TS File Generator with. |
 |
The JTools_EODFileGenerator allows you to dump summary trade information for
the entire universe of symbols in almost any fashion you desire. The system
gathers information from NxCore State Trades. As such it can be run from state
files generated just before or at the desired time. Every field available for
NxCoreStateTrade is offered as possible output. Fields generated and field
order, as well as symbol types (IE Equities, Futures, Indexes, etc) are
determined by the usage strings. It will dump the output to the console (if
specified to) and to a dated-timed text file(s).
The application comes in 2 flavors - a console application driven by command
line arguments passed into it and a GUI driven application with all setting
fields in the GUI. |
|
The Interval File Generator is very similar to the EOD File Generator but takes
the data generation one step further by also providing true interval data. That
is to say that if the timing interval is set to 1 then the values generated are
for 1 specific minute in time. Along with true interval data the application
can also generate timed EOD data, where data represents the composite values
for the day. |
 |
Just like the standard JTools_IntervalGenerator console application, the
JTools_IntervalFileGeneratorGUI allows you to dump trade information for the
entire universe of symbols in almost any fashion you desire. The system gathers
information from NxCore Trade messages for interval data, and State Trades for
EOD data. The application can be run from state files generated just before or
at the desired time. Fields generated and field order, as well as symbol types
(IE Equities, Futures, Indexes, etc) are determined by the usage strings (see
below for a complete list of possible fields).
Unlike the console based Interval Generator, the JTools Interval Generator GUI
application also combines the functionality of the JTools Batch Processor to
make processing data which spans days, months or years trivial. |
|
The Interval File Generator for Futures and Future Options is very similar to
the Standard Interval File Generator but contains some modifications and
special features.
The following interval data can be produced with the The
JTools_IntervalFileGenerator for Futures:
- One specific future.
- One specific future and all underlying future options.
- All futures belonging to a specific root symbol.
- All futures belonging to a specific root symbol plus all underlying future
options.
|
|
The Interval File Generator for Stocks and Stock Options is very similar to the
Standard Interval File Generator but contains some modifications and special
features.
The following interval data can be produced with the The
JTools_IntervalFileGenerator for Stocks/Stock Options:
- One specific stock, all stocks or a specific alpha series of stocks.
- One specific stock and all underlying stock options.
- Specific alpha series of stocks and all underlying stock options.
|
|
With the exception of the Generic Shell Application, the Table Viewer
application is the simplest of all JTools applications. The Table Viewer simply
retrieves the string definitions for each NxCore table. These tables include
(but are not limited to) exchange codes and names, trade conditions, quote
conditions, etc. By using the tables a developer has access to human readable
string definition for all NxCore tables without the need to hard-code them in.
More importantly, the table definitions are updated automatically each day so
there is never a need to update them manually. |
|
As NxCore is a whole market feed, many Nanex customers will build very
sophisticated applications using it. These may include Bar Data servers,
Streaming Data servers, Tick Data servers and/or a host of other applications
designed to maintain and deliver data to their own customers. The majority of
these applications will consist of proprietary databases that are fed with the
NxCore stream. In this regard, properly tracking new symbols, or changes to
existing symbols becomes critically important.
The Symbol Change Analyzer was developed to aid in this monitoring. It tracks
all relevant information regarding symbol additions and changes and has a
variety of mechanisms for flagging potential errors. |
|
The Halt Monitor is a simple application that monitors halts in all issues
throughout the trading day. The Halt Monitor listens to NxCore category 63
(halt) and will track the current status of any issue that has been halted,
display the reason(s) for the halt and track the number of times an issue has
been halted. |
|
Like equities, futures may also have options associated with them. Unlike
equities, futures may also have spreads associated with them. While NxCore has
built in mechanisms to identify a root symbol for a specific equity option (or
option chain), there is no such mechanism for future options and spreads.
Futhermore, future option symbols may be different than the root symbol prefix
of the option (much like equity options prior to 2010 and OSI, the Option
Symbol Initiative). As such one of the most common questions from users trading
futures is how to properly associate the future options and spreads to a
specific underlying future.
The FOS (Futures, future Options, future Speads) was developed to
demonstrate these concepts but goes much futher. It is one of the more complex
JTools components and the user/developer will be well served to study the code
closely. Once understood it will be possible to build complex applications
using the existing code. |
Volume II - Trade/Quote Monitors, Charting, Market
Analysis
 |
The Quote Montage HFT Application is one I initially wrote to test a new server
and at the same time, we wanted a new tool for viewing specific quoting stats
in any single stock (specifically when we would see high speed algos running in
particular issues). I've recently converted it over to the JTools/NxCore
framework and it is humming along better than ever. The application
demonstrates several key concepts related to performance oriented, real world
NxCore/GUI trading systems. |
 |
The Futures Depth Montage is another application I initially wrote to test a
new server and at the same time, provide a new tool to view futures depth and
depth stats. I have recently converted it over to the JTools/NxCore framework.
The Futures Depth Montage demonstrates several key concepts related to both
NxCore/GUI systems intended for real world use. |
|
Bar charting is perhaps one of the most common ways for humans to look at stock
data (or futures, etc). Bar Charts display price activity in discrete intervals
which can be anything from one day to several milliseconds.
The One Second Bar Charter is a simple application that plots data in one
second time increments. Both price and volume are plotted and the price may be
plotted as traditional bars or as candlesticks |
|
The Vwapper Application is an extension of the One Second Charter Application
that also charts VWAP (Volume Weighted Average Price) and VWAP vs Price
difference.
Because the most common VWAP calculation requires VWAP to be calculated from
the start of trading to the end of the trading day, The Vwapper Application
calculates and maintains VWAP numbers for every issue in the system and for
every trade received. As such, when you bring up new issues, the VWAP number is
current and the charting will reflect this. It is yet another example of the
'whole market analysis' one can perform with NxCore. This is not only
applicable to this charting app, but any application that uses VWAP. |
|
The Top Ten Blaster application calculates and maintains 5 top ten lists for
every exchange available. The lists are then separated by Equities, Futures and
Indexes, depending on what the exchange offers. It is a perfect example of the
'whole market analysis' one can perform with NxCore.
The application also calculates and maintains the number of advancing,
declining and unchanged issues for every exchange available. Like the top ten
Lists, they are separated by Equities, Futures and Indexes, depending on what
the exchange offers. |
|
The Pulsar Monitor demonstrates the concept of 'pulsed' quotes and trades (a
concept used in practice by many). In today's environment, even non liquid
stocks can see quote rates exceeding 5000+ quotes per second, which can place
an enormous load of both server and client software. As opposed to sending out
every quote and trade, pulsed systems send out quotes and trades on a regular
time interval, which usually ranges anywhere from 100 milliseconds to 1 second.
In doing this, considerable bandwidth in both delivery and processing of data
is saved.
The concept is demonstrated by pulsing quotes/trades to a GUI that would be the
type of system appropriate for pulsed data, which is a basic monitor containing
user defined symbols, updating on a real time basis with composite trade and
quote data. |
|
If you receive options though NxCore then you most likely have a keen interest
in building Option Chains. The following application shows you how to do just
that, plus a little more. Option Chains represent all the options available for
an underlying equity. Typically when used for displays the options are
separated by puts and calls and are sorted according to strike price and time
to expiration. Because some equities may have thousands of options, most
applications (including this one) will allow you to filter options on various
criteria. |
|
The FOS Live Chains application is very similar to the FOS Lookup Zombie
application, taking it one step further to demonstrate futures, future
option chains and future spread chains updating and populating with real
time data. If you are not familar with the FOS System please see the
documentation for the FOS Lookup Zombie application. |
Volume III - Market Stress, Market Phenomena,
Research
|
During our Flash Crash analysis, we were constantly looking for new ways to
visualize the information we were looking at. One of the focus's of the
analysis was on delays and saturation of CQS/UQDF due to extremely high quote
traffic. Because the feeds from each have Channels, or Lines, which represent
stocks by alphabet, we wanted a tool to show us how each channel was effected
to a given circumstance. I developed a charting system that would show
intensity of any given alpha channel by color.
|
|
The One Second Exchange plotter charts every exchange independently for trades,
bid quotes and ask quotes in one second intervals. It was one of the primary
tools we used in doing our initial Flash Crash analysis to visually confirm
data. This application has been refined and revised many times since we began
our research and many new items have been added.
|
|
The QTSequencer (Quote-Trade Sequencer) is an application that charts every
quote and trade (independently for each exchange) sequentially with no data
loss or compression. Originally developed for our Flash Crash Analysis it
remains a tool we use each and every day. This application has been refined and
revised many times since we began our research and many new items have been
added.
|
|
Using tools such as the QTSequencer, one can easily visualize stock activity on
a sub-second (tick) level. However, randomly poking about different times in
various stocks will yield little valuable results, what is needed is something
to screen through the universe of stocks in real time and display the results.
The HotPotato application is a conglomeration of three filters I continue to
use on a daily basis - High Quote Rates, Wide NBBO Spreads and Explode/Implode
(mini-crash) filters.
|
Volume IV - Nx3D - 3D Visualization
|
The 3D Depth Mapper application takes futures depth into another dimension.
Developed at Nanex to examine liquidity expansion and contraction in the
futures markets to see potential patterns that may not be obvious on a 2D
display. It works in one second intervals and is is one of our more advanced
(and unique) real time charting applications. The 3D Depth mapper does not
simply provide a static image for viewing but lets you navigate through the
image, allowing deeper investigation of individual components.
|
|
The 3D Depth Mapper application takes futures depth into another dimension.
Developed at Nanex to examine liquidity expansion and contraction in the
futures markets to see potential patterns that may not be obvious on a 2D
display. It works in one second intervals and is is one of our more advanced
(and unique) real time charting applications. The 3D Depth mapper does not
simply provide a static image for viewing but lets you navigate through the
image, allowing deeper investigation of individual components.
|
|
The 3D Quote/Trade Sequencer is an extension of the JTools_QTSequencer
application, bringing a new dimension to examining sequential quotes and
trades. Developed at Nanex to examine potential algorithmic patterns that may
not be obvious on a 2D display. Like the 2D version of the sequencer, the 3D
version plots every quote and trade by every reporting exchange sequencially as
they occur.
|
|
The 3D Quote/Trade Sequencer is an extension of the JTools_QTSequencer
application, bringing a new dimension to examining sequential quotes and
trades. Developed at Nanex to examine potential algorithmic patterns that may
not be obvious on a 2D display. Like the 2D version of the sequencer, the 3D
version plots every quote and trade by every reporting exchange sequencially as
they occur.
|
Volume V - Odds & Ends
|
The ASCII compression utility is a very small application that does not use
NxCore and is not based on the JTools_ShellApplication framework. Rather, it is
a simple application to demonstrate simple compression (up to 50%) and
decompression of ASCII data (which can be implemented on any platform) with
minimal performance impact. This can be very useful to clients who redistribute
financial data in ASCII format.
|
|
Because of the in-depth historical files NxCore provides, there are several
times when the developer may want to run an application over many tapes.
Possibly for analysis, possibly to populate historical databases.
The Batch Processor application is a very simple modification of the Generic
Shell App. It takes a start date, stop date (and a few other parameters) and
processes through the dates, stopping only when the last historical file has
been processed. With very little effort, any one of the appropriate JTools
applications could be modified to batch process historical files using the code
provided with the Batch Processor application.
|
|
