Search
× Search
Tuesday, November 19, 2024

Archived Discussions

Recent member discussions

The Algorithmic Traders' Association prides itself on providing a forum for the publication and dissemination of its members' white papers, research, reflections, works in progress, and other contributions. Please Note that archive searches and some of our members' publications are reserved for members only, so please log in or sign up to gain the most from our members' contributions.

Complex Event Processing - System Benchmarks

photo

 Nick Wilton, Managing Director at Alpha Integralis

 Wednesday, July 31, 2013

Dear All, A client of mine has asked me to develop a system that will make decisions in real time on a number of streams/feeds of trade data. As part of the feasibility study I have set up a number of prototypes that make use of Microsoft StreamInsight (formerly Microsoft CEP) and have been measuring the latency between a matching pattern entering the system and an event being raised. For example, if I have a price feed where I wish to measure the mean price every second I would want to raise an event once that price goes above some arbitrary threshold. I'm finding that using C# and StreamInsight that the simple query above seems to respond in between 4 and 10 milliseconds. This seems good enough for my client and suggests that I should probably concentrate on identifying bottlenecks elsewhere. However, I wonder how this compares to other major systems in use today. Before I look deeper into this myself, does anyone else have any experience benchmarking CEP systems? If so, what did you find? Google doesn't appear to have the answer. Thanks in advance, Nick


Print

5 comments on article "Complex Event Processing - System Benchmarks"

photo

 Steve Summa, Senior Relationship Manager at Interactive Data

 Friday, August 9, 2013



I represent a company called OptionsCity. We offer an algo engine that is low latency and easy to program (JAVA). You can code both algos and CEP. We also offer an algo store so any algos you create can be resold to others. If you care to learn more please call me at 646-755-7914 or email me at ssumma@optionscity.com


photo

 Michael Wynne, Director Technical Development at ADS Securities

 Friday, August 9, 2013



Jean-Noel. I agree, from market event to order we were looking at 10's of usecs. And yes Ed, cheap way to speed things up is to overclock to over 5Ghz. People tend to overlook this, but its quick and cheap unless you need 100 servers running.


photo

 sumit S., Senior software developer / Architect - financial services.

 Friday, August 9, 2013



One of my clients had a need for processing 1Million events per second while doing aggregation on a dynamic window( requiring a functionality to update the window sizes).


We did evaluation of esper and few other CEP tools but all came to about 350K events per second( for our use case), Esper was the best some other commercial tools also did good ( coral8/apama).


At the same time we came accross a research on dbtoaster which promised a better throughput :http://www.dbtoaster.org/


Using dbtoaster like paradigm we created our own sql based engine which would turn continous query statement into c/cc or java code and would run it.


Our design was:

Overall we used: ZQL[http://zql.sourceforge.net/] to parse and generate the code.


For java based system design we could easily turn the sql Like queries with our window functions in to java code.

For C/C system:


1) we generated the code

2) then using swig built java interfaces

3) compiled code on fly for both c and java

4) loaded c/c library via JNA.

This solution processed streaming data much faster while allowing us to use LEDA/sparse hashmap and other c libraries (improving over all throughput).


photo

 Boris Litvin, CEO at Ortess Inc

 Monday, August 12, 2013



We use Esper for low-latency algo trading for 3 years. It is the most optimum CEP that gives you the optimum combination of speed, flexibility, time-to-market and costs. However we had to build sophisticated framework around it to manage HA, data persistence and variety of application level features that none of off-the-shelves CEP engines can provide right out of the box.


photo

 Jim Giovinazzo, Sales Director at Azul Systems

 Wednesday, August 14, 2013



We built own own CEP and have an algorithm kit built around it. This decreases time to market and costs dramatically. If you would like to learn more let me know or view www.embium.com

Please login or register to post comments.

TRADING FUTURES AND OPTIONS INVOLVES SUBSTANTIAL RISK OF LOSS AND IS NOT SUITABLE FOR ALL INVESTORS
Terms Of UsePrivacy StatementCopyright 2018 Algorithmic Traders Association