Yes, you can use EC2 to dynamically change the amount of resource available to your calculation. They make it pretty easy, and it sounds like you've already written the code in a parallel way.

Let me state it differently - a 32 bit app has inherent limitations so ultimately it dictated our quasi manual approach to parallel computing using the divide & conquer principle. Now I am curious to know it is possible to create a single giant instance (c4.8.large) on the Amazon cloud launch our application and get results quicker… to me this is not obvious and maybe our approach to run it manually on several own Xeons still works better…

NB: the app itself relies heavily on multithreading and actually each calculation inside is a thread and is independent of the others. However each such thread takes up to 20 minutes to calculate and we have 50 threads so you count …

Hi, yes we do. You either can create a master-slaves architecture for example using tcp/ip sockets for the communications within nodes running on top of an aws cluster or you can use natively parallel applications dependin on your specific needs such as hadoop, graphlab, h2o, etc. Best regards! Andres

If there is no communication between threads then its embarrassingly parallel and you should be able to complete all 50 threads in 20 mins.

I would rewrite the app to be 64 bit anyway, its likely that just using a modern compiler would get you some extra speed up.

The you are down to micro optimizations and fine tuning, or picking better data structures, minimizing threading contention, IO etc.

You could also look at GPGPU but not algos are easy to convert to a GPGPU approach.

Lastly using a bigger machine with a more powerful processor will of course give the easiest improvement.

David thanks for your answer. However If it was that simple I would not be asking. The app is TradeSignal and I can not "rewrite it" it is a programming envrionment and for the time being the vendor has decided that it has to be 32 bit period. The business application has been designed on purpose to be that "embarrassingly parallel". The limitation is that you can not have as many simutaneous instances of Tradesignal as you like because for that you need to have as many bloombergs and as many TS licenses... ( BBG = 2k, Ts=0.5k per month). My initial question is: can one create a unique instance so powerful that even despite all the abovementioned limitation one gets the full benefit of parallel computing bearing in mind the TS will create 50 threads ....thanks !

NB: we can not and are not willing to change the environment because we have tens of thousands of lines of well tested & working production code running on it

Hi people thanks for allowing me into the group. I hope I don't break any rules but we would like to ask for some help and pointers. I run the biggest shares chat website in Australia. It's called Hotcopper.com.au it is mostly speculative stocks. Out BIG data is amazing but we need some help creating algorithms for the sentiment data. We don't know where to start.

Amazon has a number of different instance types including memory and I/O optimized. You should be able to easily test different configs to see which gives you the best balance of performance and cost. Best of all, you can simple test by the hour with no commitment. If your workload permits, Amazon also has spot instances.

Hi Kirill, you are talking a 'big iron', approach of vertical rather than horizontal scaling.

The most powerful Xeons have 18 cores with hyper threading, so that is 36 active threads on one processor, it possible a 2-CPU server would meet your needs, you could always buy a 4-socket server and leave two sockets empty until you determine your needs.

Hi Kirill, it sounds like you have outgrown TradeSignal. If you are technically proficient, or have access to developers, I would suggest developing your own solution. If your strategy is purely technical it will probably not be too difficult to write it in another language without the restrictions of TradeStation.

make no mistake : "tradeSignal" has nothing to do with tradeStation. Our system has several components one of them is executed in tradeSignal and unfortunately it really out of question to rewrite it because it works so well and because it has literally 10s of thousands of lines of code accross a dozen modules...tradeSignal is a fantastic piece of software..One way we found how to deal with the situation is divide the task in 2 parts and run them separately on two servers