Automated optimization of a cBot

PanagiotisCharalampous
04 May 2020, 10:25

Hi Christian,

Unfortunately there is no feature for automatic optimization at the moment neither we have plans for this in the near future. If you need this, it is something you will need to develop yourself.

Best Regards,

Panagiotis 

Join us on Telegram

Shares4us
15 May 2020, 18:55

RE:

I wrote 2 different types of learning bots (single thread & threaded).
Also tried ML but the problems with core & framework killed that development line.
Maybe brainstorm a little about it?

PanagiotisCharalampous
18 May 2020, 08:35

RE: RE:

ctid956028 said:

I wrote 2 different types of learning bots (single thread & threaded).
Also tried ML but the problems with core & framework killed that development line.
Maybe brainstorm a little about it?

 

Hi there,

What do you want to us to brainstorm about?

Best Regards,

Panagiotis 

Join us on Telegram

genappsforex
18 May 2020, 16:48

RE: RE: RE:

PanagiotisCharalampous said:

ctid956028 said:

I wrote 2 different types of learning bots (single thread & threaded).
Also tried ML but the problems with core & framework killed that development line.
Maybe brainstorm a little about it?

 

Hi there,

What do you want to us to brainstorm about?

Best Regards,

Panagiotis 

Join us on Telegram

 

remark was for Christian

ctid2032775
26 Jun 2020, 10:25

RE:

PanagiotisCharalampous said:

Hi Christian,

Unfortunately there is no feature for automatic optimization at the moment neither we have plans for this in the near future. If you need this, it is something you will need to develop yourself.

Best Regards,

Panagiotis 

Join us on Telegram

 

Hi,

I made it to develop a module for "automatic" optimization but have a quick question to finish it...

Right now this module reads historic data from a text file. This file is created by running a (small) cBot in backtesting mode that just gets the data (OnTick/OnBar) and writes it into the file. This requires some manual steps and to not loose any data (i. e. missing some ticks/bars) this can only be done during weekends!

I would like to fetch the data (OHLC, Ask, Bid) for a specific time frame (e. g. last 2 years) every time the cBot is started (OnStart) and run the optimization with this historic data - is there a way to realize this (maybe with FIX API or Open API)? Can you provide some code snippets and/or examples where this was already done?

Many thanks for your support.

BR,
Christian

genappsforex
26 Jun 2020, 16:08

RE: RE:

This requires some manual steps and to not loose any data (i. e. missing some ticks/bars) this can only be done during weekends!

Maybe use threading, that way you will not miss ticks and then you can run it anytime

prosteel1
27 Jun 2020, 12:51

RE: Brainstorming - Genetic Algorithms & Agile Development

It looks like the built in optimization might use something like the Quasispecies model of Genetic evolution. A simple wiki is: https://en.wikipedia.org/wiki/Quasispecies_model  and a much more advanced ieeexplore paper based on Quantum Physics is: https://ieeexplore.ieee.org/ielx7/6287639/8600701/08684946.pdf?tp=&arnumber=8684946&isnumber=8600701&ref=aHR0cHM6Ly9pZWVleHBsb3JlLmllZWUub3JnL2RvY3VtZW50Lzg2ODQ5NDY=

I haven't got to implimenting it yet, however I'm hoping Eigenvalues could be calculated using a matrix determinant. 

The reason I am looking at this Eigenvalue concept is that charts are created by short term and longer term supports and resistance that have a probability of hitting a take profit. The most fit species would be the shorter term that would have it's probability of reaching it's take profit increased when it's resistance or support are broken and when it hits it's take profit it would die out due to 'error catastrophe' (see the bottom of the wiki link), and allow longer term species to eventually take over. Ofcourse new species would be added as they are identified.

So rather than running optimizations to find the parameters, the matrix of species would be the parameters.

Agile software development looks like it has potential as it starts out knowing that it does not know the solution and so it is based on finding the solution.

If we could get the optimization code that spotware uses so we could impliment it ourselves that would likely be quite useful. The GetFittnessArgs implimentation should allow some access but perhaps during manual optimization only (haven't got that far yet).

prosteel1
27 Jun 2020, 13:21

RE: RE:

ctid2032775 said:

PanagiotisCharalampous said:

Hi Christian,

Unfortunately there is no feature for automatic optimization at the moment neither we have plans for this in the near future. If you need this, it is something you will need to develop yourself.

Best Regards,

Panagiotis 

Join us on Telegram

 

Hi,

I made it to develop a module for "automatic" optimization but have a quick question to finish it...

Right now this module reads historic data from a text file. This file is created by running a (small) cBot in backtesting mode that just gets the data (OnTick/OnBar) and writes it into the file. This requires some manual steps and to not loose any data (i. e. missing some ticks/bars) this can only be done during weekends!

I would like to fetch the data (OHLC, Ask, Bid) for a specific time frame (e. g. last 2 years) every time the cBot is started (OnStart) and run the optimization with this historic data - is there a way to realize this (maybe with FIX API or Open API)? Can you provide some code snippets and/or examples where this was already done?

Many thanks for your support.

BR,
Christian

I use a for loop OnStart to cycle through each bar from the current bar count back to count = 1. There is also a newish  'Bars.LoadMoreHistory' command to get a specified number of more bars. OnTick I check for a new Bar and I run it again but only go back 2 bars.  I think this is what you are looking for as the data available to Backtesting lags a day, whereas this accesses the live data :) Current bar data can be accessed using Bars.LastBar.Low etc and current bid/ask using Symbol.Bid and Symbol.Ask.

I run OnTick so I can check for the price reaching take profit 1 price to close half the position and to move the stoploss to entry and checking that the Symbol.Ask is not above the short stoploss when placing an order (as that prevents the stoploss from being set), etc etc.

OnStart()

Bars series = MarketData.GetBars(frame);
int i = series.Count - 1;

BarLast = i - 2;
for (int ii = i - 2; ii >= 1; ii--)
{

      // Write OHLC Data

}

OnTick()

// Check for a new bar on specific timeframe

Bars series = MarketData.GetBars(frame);
if (series.Count - 1 > BarLast)
{

     for (int ii = i - 2; ii >= x; ii--)
     {

           // Write OHLC Data

     }

}

In regards to your question above "Part of the bot is a machine learning algorithm that is already existing in C# - I guess it's not a problem to include this in a cBot, right?", I am quite a noob but when I investigated this I think the answer was C# was not suitable so it would need to use another language suited to machine learning and so going through the FIX API would be required. Possibly making an API to connect the ML code to the C# code?

ctid2032775
10 Jul 2020, 11:29

RE: RE: RE:

prosteel1 said:

ctid2032775 said:

PanagiotisCharalampous said:

Hi Christian,

Unfortunately there is no feature for automatic optimization at the moment neither we have plans for this in the near future. If you need this, it is something you will need to develop yourself.

Best Regards,

Panagiotis 

Join us on Telegram

 

Hi,

I made it to develop a module for "automatic" optimization but have a quick question to finish it...

Right now this module reads historic data from a text file. This file is created by running a (small) cBot in backtesting mode that just gets the data (OnTick/OnBar) and writes it into the file. This requires some manual steps and to not loose any data (i. e. missing some ticks/bars) this can only be done during weekends!

I would like to fetch the data (OHLC, Ask, Bid) for a specific time frame (e. g. last 2 years) every time the cBot is started (OnStart) and run the optimization with this historic data - is there a way to realize this (maybe with FIX API or Open API)? Can you provide some code snippets and/or examples where this was already done?

Many thanks for your support.

BR,
Christian

I use a for loop OnStart to cycle through each bar from the current bar count back to count = 1. There is also a newish  'Bars.LoadMoreHistory' command to get a specified number of more bars. OnTick I check for a new Bar and I run it again but only go back 2 bars.  I think this is what you are looking for as the data available to Backtesting lags a day, whereas this accesses the live data :) Current bar data can be accessed using Bars.LastBar.Low etc and current bid/ask using Symbol.Bid and Symbol.Ask.

I run OnTick so I can check for the price reaching take profit 1 price to close half the position and to move the stoploss to entry and checking that the Symbol.Ask is not above the short stoploss when placing an order (as that prevents the stoploss from being set), etc etc.

OnStart()

Bars series = MarketData.GetBars(frame);
int i = series.Count - 1;

BarLast = i - 2;
for (int ii = i - 2; ii >= 1; ii--)
{

      // Write OHLC Data

}

OnTick()

// Check for a new bar on specific timeframe

Bars series = MarketData.GetBars(frame);
if (series.Count - 1 > BarLast)
{

     for (int ii = i - 2; ii >= x; ii--)
     {

           // Write OHLC Data

     }

}

 

In regards to your question above "Part of the bot is a machine learning algorithm that is already existing in C# - I guess it's not a problem to include this in a cBot, right?", I am quite a noob but when I investigated this I think the answer was C# was not suitable so it would need to use another language suited to machine learning and so going through the FIX API would be required. Possibly making an API to connect the ML code to the C# code?

First of all many thanks for your reply!

As there are no problems but a lot of challenges ;-) - my challenge is to get the historical ask and bid prices for about one year for my automatic optimization algorithm (please keep in mind that I cannot use the built-in optimization for this part of the cBot; see above). Unfortunately the bars do not contain this information...

But, I found a way to get the information while cBot starts - by fetching the ticks (Ticks ticks = MarketData.GetTicks(); ... i = ticks.LoadMoreHistory();) I get the required information but this process is extremely memory consuming. This leads to slowing down the system even if the garbage collector removes the array(s) from the memory!

Any idea to get the data without such a high memory consumption is welcome! :-)

The machine learning part is finished - I am using the C4.5 algorithm in the accord.net framework to create a decision tree based on some (calculated) attributes. The prediction of the tree with current attributes is then used to calculate the size of the opened orders (i. e. money management)...

prosteel1
10 Jul 2020, 22:29

RE: RE: RE: RE:

I’ve been testing this and seems good to manage memory.

I’ve setup a loadmorehistory routine that gets the date of the first bar on the highest timeframe and then loads more bars on lower timeframes.

I haven’t had to use the Ticks as you mentioned as yet, rather, I use bars of Tick10 etc. using a workaround I posed in this linked post below.

When I set the higher timeframe as 4 hour it went back almost a year, then the other 6 timeframes down to Tick5 loaded history back to there which resulted in 1,700,000+ bars on Tick 5 and 1 minute and yes it took a very long time to load and analyse on i7 with 32GB ram haha

ctid2032775
16 Jul 2020, 11:54

RE: RE: RE: RE: RE:

prosteel1 said:

cTrader cBot Memory Manager

I’ve been testing this and seems good to manage memory.

I’ve setup a loadmorehistory routine that gets the date of the first bar on the highest timeframe and then loads more bars on lower timeframes.

I haven’t had to use the Ticks as you mentioned as yet, rather, I use bars of Tick10 etc. using a workaround I posed in this linked post below.

When I set the higher timeframe as 4 hour it went back almost a year, then the other 6 timeframes down to Tick5 loaded history back to there which resulted in 1,700,000+ bars on Tick 5 and 1 minute and yes it took a very long time to load and analyse on i7 with 32GB ram haha

cTDN Forum - Tick timeframes not part of TimeFrame Class

 

Many thanks for your suggestion!

Based on this - and as I need ask and bid prices to calculate mean prices for the automatic optimization and for the machine learning algorithm - I changed the logic to "reading bars" and calculating the prices (bid = bar.Open, ask = bid + (fix) spread) as it's also done by the internal cTrader optimization. This runs (a lot) faster and takes (extremely) less memory.