[JBookTrader] Re: Parallel Processing for Running Optimizations

Discussion:

Klaus

2014-10-08 10:55:03 UTC

Hi,

I was just wondering: was there any further efforts to do CUDA-based
backtesting for JBT?

Cheers
Klaus

I am looking into this, but it is still far from ready for prime time. I
have a good speedup, but for some reason the results are diverging from the
expected values in seemingly random ways. I will update the group when I
have something ready to share... it is still 'pre-alpha' at this point.
I asked Eugene to remove some info I shared with him on this topic as it
is premature at this point, and he was kind enough to do so.

Hi all,
Just wondering if anyone has looked at use parallel processing for
optimizing strategies yet. I've been using the NVIDIA's CUDA library for
other heavy lifting jobs with good results. There's a project called JCUDA
which supplies a Runtime API with Java bindings for CUDA. If no one has
written anything for this yet, I may take it on as a side project.
http://www.jcuda.org/
regards,
Michael

--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jbooktrader+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at http://groups.google.com/group/jbooktrader.
For more options, visit https://groups.google.com/d/optout.

Judson Wilson

2014-10-08 20:14:43 UTC

Permalink

I think it would be more worthwhile to do a cloud based multi-computer
approach.

Post by Klaus
Hi,
I was just wondering: was there any further efforts to do CUDA-based
backtesting for JBT?
Cheers
Klaus

You received this message because you are subscribed to the Google Groups
"JBookTrader" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/jbooktrader.
For more options, visit https://groups.google.com/d/optout.

Klaus

2014-10-09 08:18:40 UTC

Permalink

Your mileage may vary.
For me, such a solution might mean a speedup of about 30 (given the number
of graphics cores, the differences in performance, non-optimal
parallelization, etc. - but it is a rather wild, but conservative guess).
If you look at pricing on the cloud, using this might cost like 5-10 USD /
hour (were also partial hours count).
Given the number of runs one has to do to reach a reasonable result, this
can easily go to dozens/hundreds of hours. This might be ok from your
perspective, it seems like a bad deal from my point of view. (this is also
why modern high-performance computers often take a GPU heavy approach)
So, I think, going GPU first is probably a good idea, then one might want
to cascade it later with a multicomputer approach (AWS offers specific GPU
instances for the cloud).

But all this is mute, the question is whether any one with the necessary
expertise is pursuing this approach.
Or perhaps someone has better data for estimating the kinds of results to
be expected.

Klaus

Post by Judson Wilson
I think it would be more worthwhile to do a cloud based multi-computer
approach.

Post by Klaus
Hi,
I was just wondering: was there any further efforts to do CUDA-based
backtesting for JBT?
Cheers
Klaus

I am looking into this, but it is still far from ready for prime time.
I have a good speedup, but for some reason the results are diverging from
the expected values in seemingly random ways. I will update the group when
I have something ready to share... it is still 'pre-alpha' at this point.
I asked Eugene to remove some info I shared with him on this topic as it
is premature at this point, and he was kind enough to do so.

You received this message because you are subscribed to the Google Groups
"JBookTrader" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/jbooktrader.
For more options, visit https://groups.google.com/d/optout.

Judson Wilson

2014-10-09 09:33:26 UTC

Permalink

I didn't notice this was in the original thread. A factor of 1000x
improvement? REALLY? I have a hard time believing it.

Post by Klaus
Your mileage may vary.
For me, such a solution might mean a speedup of about 30 (given the number
of graphics cores, the differences in performance, non-optimal
parallelization, etc. - but it is a rather wild, but conservative guess).
If you look at pricing on the cloud, using this might cost like 5-10 USD /
hour (were also partial hours count).
Given the number of runs one has to do to reach a reasonable result, this
can easily go to dozens/hundreds of hours. This might be ok from your
perspective, it seems like a bad deal from my point of view. (this is also
why modern high-performance computers often take a GPU heavy approach)
So, I think, going GPU first is probably a good idea, then one might want
to cascade it later with a multicomputer approach (AWS offers specific GPU
instances for the cloud).
But all this is mute, the question is whether any one with the necessary
expertise is pursuing this approach.
Or perhaps someone has better data for estimating the kinds of results to
be expected.
Klaus

Post by Judson Wilson
I think it would be more worthwhile to do a cloud based multi-computer
approach.

Post by Klaus
Hi,
I was just wondering: was there any further efforts to do CUDA-based
backtesting for JBT?
Cheers
Klaus

I am looking into this, but it is still far from ready for prime time.
I have a good speedup, but for some reason the results are diverging from
the expected values in seemingly random ways. I will update the group when
I have something ready to share... it is still 'pre-alpha' at this point.
I asked Eugene to remove some info I shared with him on this topic as
it is premature at this point, and he was kind enough to do so.

You received this message because you are subscribed to the Google
Groups "JBookTrader" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/jbooktrader.
For more options, visit https://groups.google.com/d/optout.

Klaus

2014-10-09 12:51:58 UTC

Permalink

Not sure where you take the 1000x improvement from? I was talking about a
30x - and this was an ad hoc guess.

If you look around, there are comparisons which make this seem realistic
(e.g., http://www.hs.uni-hamburg.de/DE/Ins/Per/Abhranil/summer_report.pdf),
but this also shows it dependents very much on the task specifics. This
would be the interesting question here, whether such a speed-up would be
realistic.
(there the border is at a factor of 400, but this obviously depends on the
specifics of the CPU and GPU compared)

Post by Judson Wilson
I didn't notice this was in the original thread. A factor of 1000x
improvement? REALLY? I have a hard time believing it.

Post by Klaus
Your mileage may vary.
For me, such a solution might mean a speedup of about 30 (given the
number of graphics cores, the differences in performance, non-optimal
parallelization, etc. - but it is a rather wild, but conservative guess).
If you look at pricing on the cloud, using this might cost like 5-10 USD
/ hour (were also partial hours count).
Given the number of runs one has to do to reach a reasonable result, this
can easily go to dozens/hundreds of hours. This might be ok from your
perspective, it seems like a bad deal from my point of view. (this is also
why modern high-performance computers often take a GPU heavy approach)
So, I think, going GPU first is probably a good idea, then one might want
to cascade it later with a multicomputer approach (AWS offers specific GPU
instances for the cloud).
But all this is mute, the question is whether any one with the necessary
expertise is pursuing this approach.
Or perhaps someone has better data for estimating the kinds of results to
be expected.
Klaus

Post by Judson Wilson
I think it would be more worthwhile to do a cloud based multi-computer
approach.

Post by Klaus
Hi,
I was just wondering: was there any further efforts to do CUDA-based
backtesting for JBT?
Cheers
Klaus

I am looking into this, but it is still far from ready for prime
time. I have a good speedup, but for some reason the results are diverging
from the expected values in seemingly random ways. I will update the group
when I have something ready to share... it is still 'pre-alpha' at this
point.
I asked Eugene to remove some info I shared with him on this topic as
it is premature at this point, and he was kind enough to do so.

Judson Wilson

2014-10-10 03:03:29 UTC

Permalink

Riunning the JBT optimization on he high end GPU (AMD Radeon 7900 series)

indicates a speedup in the order of 1000 times faster. The big drawback is
that it required a complete re-write of the multiple JBT classes,
essentially flattening the object-oriented model into array processing...

Not sure where you take the 1000x improvement from? I was talking about a
30x - and this was an ad hoc guess.
If you look around, there are comparisons which make this seem realistic
(e.g., http://www.hs.uni-hamburg.de/DE/Ins/Per/Abhranil/summer_report.pdf),
but this also shows it dependents very much on the task specifics. This
would be the interesting question here, whether such a speed-up would be
realistic.
(there the border is at a factor of 400, but this obviously depends on the
specifics of the CPU and GPU compared)

Post by Judson Wilson
I didn't notice this was in the original thread. A factor of 1000x
improvement? REALLY? I have a hard time believing it.

Post by Klaus
Your mileage may vary.
For me, such a solution might mean a speedup of about 30 (given the
number of graphics cores, the differences in performance, non-optimal
parallelization, etc. - but it is a rather wild, but conservative guess).
If you look at pricing on the cloud, using this might cost like 5-10 USD
/ hour (were also partial hours count).
Given the number of runs one has to do to reach a reasonable result,
this can easily go to dozens/hundreds of hours. This might be ok from your
perspective, it seems like a bad deal from my point of view. (this is also
why modern high-performance computers often take a GPU heavy approach)
So, I think, going GPU first is probably a good idea, then one might
want to cascade it later with a multicomputer approach (AWS offers specific
GPU instances for the cloud).
But all this is mute, the question is whether any one with the necessary
expertise is pursuing this approach.
Or perhaps someone has better data for estimating the kinds of results
to be expected.
Klaus

Post by Judson Wilson
I think it would be more worthwhile to do a cloud based multi-computer
approach.

Post by Klaus
Hi,
I was just wondering: was there any further efforts to do CUDA-based
backtesting for JBT?
Cheers
Klaus

I am looking into this, but it is still far from ready for prime
time. I have a good speedup, but for some reason the results are diverging
from the expected values in seemingly random ways. I will update the group
when I have something ready to share... it is still 'pre-alpha' at this
point.
I asked Eugene to remove some info I shared with him on this topic as
it is premature at this point, and he was kind enough to do so.