Developing a Trading Framework from Scratch

First pass is going to be compiled in. Second pass will likely be DLLs. I have some ideas for this. I can add support for things like COM, Python, .NET, etc.

The application is written as an ATL .EXE, so it can register itself as a COM server. COM has performance issues though with data-marshalling, so I haven't fully decided what ways I'm going to implement strategy management.

I believe NeoTicker supports COM-based strategies. E-Signal supports DLL function calls from its EFS. I think AMIBroker might also offer this feature.

I'll think about it over the weekend. One thing is for sure -- I want a well-defined interface that gets exported.

 

To implement and backtest ideas and work with time in an easier fashion [and to avoid E-Signal APIs], I wrote a DLL that helps me pull information from E-Signal and populate a database with it.

The intention of this code, however crappy it may be, is to help me implement Don's Openings and have a strategy to backtest the performance of his openings, assuming envelope-based fills get taken care of on the first opening print where the exchange = the NYSE.

I am not posting DLL source code in public yet. But since it's a utility, I will likely formalize it and release it to the public. I am releasing the script interface to the DLL. You could probably implement your own DLL with relative ease, even if I never posted such a thing.

The license on the following code here is GPL, not BSD. If you are unfamiliar with the terms of the GPL, stop dead in your tracks in proceed to gnu.org and read, read, read. Please note that while I am not using my name, I am asserting copyright via this alias which is ultimately connected to my real name by the e-mail address registered with ET. I am using this license because if you lose all your money because of this script, I am not liable. Usage of this script means that you acknowledge that I am not liable in any way shape or form for your future misery as a result of using this script.

I want you to be rich, but I make mistakes and write bad software on occasion too.

-- Code Begin --


// Script to pull content from E-Signal
// Copyright (C) 2006, authored by FatRat. (alias fatrat on www.elitetrader.com)

// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.

// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
// See the GNU General Public License for more details.

// You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.


var g_DEBUG = true;
var g_EnableDataExport = false;
var g_DLL = new DLL("Chart2DB.dll");

function preMain()
{
if( g_DEBUG ) debugPrint( "[DEBUG] Entered preMain()\n" );

setStudyTitle( "Char2DB Transfer" );
setCursorLabelName( "C2DB" );
setDefaultBarFgColor( Color.green );

if( g_DEBUG ) debugPrint( "[DEBUG] Binding Function...\n" );
g_DLL.addFunction( "EFS_Export1mOHLC", DLL.INT, DLL.CDECL, "Export1mOHLC", DLL.STRING, DLL.INT, DLL.INT, DLL.INT, DLL.INT, DLL.INT, DLL.INT, DLL.DOUBLE, DLL.DOUBLE, DLL.DOUBLE, DLL.DOUBLE );
if( g_DEBUG ) debugPrint( "[DEBUG] Function bound.\n" );

var interval = getInterval();
if( interval == "1" )
{
debugPrint( "Enabling Data Export\n" );
g_EnableDataExport = true;
}
else
{
debugPrint( "Disabling Data Export [non-1m chart]\n" );
g_EnableDataExport = false
}

if( g_DEBUG ) debugPrint( "[DEBUG] Exited preMain()\n" );
}


function main()
{
if( g_DEBUG == true ) debugPrint( "[DEBUG] Entered main()\n" );
if( g_EnableDataExport == true )
{
var _month, _day, _year, _hour, _minute, _second, _open, _high, _low, _close, _symbol;
_year = getYear(0);
_month = getMonth(0);
_day = getDay(0);
_hour = getHour(0);
_minute = getMinute(0);
_second = getSecond(0);
_open = open( 0 );
_high = high( 0 );
_low = low( 0 );
_close = close( 0 );
_symbol = getSymbol();
if( g_DEBUG == true )
{
debugPrint( "[DEBUG] Parameters: " + _symbol + "," + _month + "," + _day + "," + _year + "," + _hour + "," +
_minute + "," + _second + "," + _open + "," + _high + "," + _low + "," +
_close + "\n" );
}

var result =
g_DLL.call( "EFS_Export1mOHLC", _symbol, _month, _day, _year, _hour, _minute, _second, _open, _high, _low, _close );
return result;
}
else
{
if( g_DEBUG ) debugPrint( "Data Export Unavailable" );
return 0;
}
}

//
// Unload Cleanup
//

function postMain()
{
if( g_DEBUG )
debugPrint( "[DEBUG] postMain()\n" );
}

-- Code End--

 

After some thoughts on the matter, I've decided to isolate strategies and models from the core application of the Incisor Trading Engine.

The primary reason? Because a crash in one model should not affect another model. While there is some performance gain to be had from having code execute within the same process space, the performance gain does not offset the cost of a crash affecting multiple other strategies. If there are 2 strategies running on the market, for example, and one is tried and true and can maintain 5000 shares at a time, a less tested, less effective strategy doing odd-lots and playing games should not be even in the same process space.

A crash is exceptionally dangerous, and if the Incisor Trading Engine fails, side models and strategies can text-message me or route information to me alerting me that something is wrong. Redundancy can be built in when multiple processes are checking on each other. With DCOM, I can even push the strategies to other boxes and have them work seamlessly within the framework exported by my application.

If performance becomes an issue, I'll allow the code to run internally; however, this ability will only be allowed for the holiest of holy algorithms and trading systems.

Finally, if there are shares on the book and a strategy doesn't give a keep-alive or stops responding, I have more recourse with the processes split out. I can check the books and liquidate positions or bring up a manual interface to figure out how to manage shares.

Additionally, if I export an IBroker interface, an IRiskManager, and an IDataFeed interface, I can scrap the trading engine and write alternative interfaces to other feeds and other software packages should another package come into existence that puts my work to shame and offers such compelling features that I have no choice but to switch.

Comments, thoughts, and feelings on the matter of strategy placement into different binaries would be appreciated.

 

A moment aside from seriousness: I *fucking love* CTime and CTimeSpan classes from Microsoft's ATL library.

They make my life so incredibly easy.

 

I'm just curious about this - If you are relying on ATL and WTL to speed up development, using COM/DCOM and implementing IThisInterface and IThatInterface, using ADO for DB access, etc., how much of the final code would really be portable to other operating systems? It seems that you might be unintentionally locking yourself down to Windows.

Also, if you built all this to run as 32-bits, it is not that trivial to later get this ATL/COM stuff to recompile and run natively under 64-bit Windows (where there is no more 4GB virtual address space limitation!). Had you used C# and .NET 2.0 you wouldn't even need to recompile. In the immediate future, I can easily see desktops loaded with 16GB of memory or more - something which your tick-cache should be able to take advantage of if your system were a native 64-bit process.

 

How do you interface with e-Signal without using their API? Are you retrieving real-time tick data or are you retrieving historical data as well?

 

There's a misconception that COM is a windows specific thing. COM is used more frequently on Windows, but there are other implementations.

http://www.mozilla.org/projects/xpcom/

It's more popular in the windows world, but the Mozilla guys have needed this functionality for quite some time as well. As long as the interfaces are cleanly designed, there will never be a significant problem porting.

Secondly, who's to say I won't define interfaces such that the trading engine figures out a way to enumerate a bunch of ITradingStrategy interfaces, where ITradingStrategy is implemented in .NET? Again, when interfaces are cleanly defined, any transition to future architectures will not be very difficult at all.

Any sophisticated GUI will likely have significant tie-ins to windows. That would be reimplemented in a port to another platform, but it's not as dire when the core functionality of trading components is encapsulated properly.

With regards to ADO, I will likely not move away from SQL Server. If a cross-platform switch is required, I'd have to rewrite only the layer that accesses a database and then switch to another implementation. Because the software relies on stored procs, not much would change provided that I just swap out the components that access the database.

There will always be some work required to move from one platform to another when software is written in C++. I just don't think it's as dramatic a change when the pieces of the application are several different components.

 

I'm using some of E-Signal's functionality as a crutch right now until I have my own systems in place. This is part of the reason I am breaking my application apart into several different pieces and components. E-Signal, for example, has a back-testing component that I don't have and generates back-test reports. I want to use this for the time being.

The way I am doing things right now is I have a DLL that gets loaded by EFS. I then load the product that has the history I'm interested in, go all the way back to the time from which I want to start analyses, and then I load my EFS "formula." (I posted source in this thread.) The formula then in turn calls my DLL for each piece of historical information, and my DLL puts that information into a database.

Then, another application that is considered a "strategy" implementation makes use of this historical information. E-Signal has a much, much larger store of information than I have, so taking the pieces I'm interested in and putting them into my database in a form that I like makes sense. It's much easier than fighting with their API, and it allows me to access information in a random fashion, without having to be tied down to their per-bar function call implentation.

The current goal right now is to get -a- strategy running as soon as possible so the trading system will start to pay for my development time. So if it's a hodge podge of strategies and binaries, that's fine.

 

One more comment:

I have the book "ATL Internals" -- So I've been familiarizing myself with the inside of ATL. Getting rid of the ATL wrappers for a cross-platform push would be easy, because ATL has wrapped the windows-specific functionality away from the code that does the work I need to do. Taking the core classes that do the work and disconnecting them from ATL will be a very, very simple job.

 

"After some thoughts on the matter, I've decided to isolate strategies and models from the core application of the Incisor Trading Engine."


Smart move.

Richard