Backtester/Optimiser not matching with same parameters

PanagiotisCharalampous
09 Oct 2024, 12:19

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

davidrob
10 Oct 2024, 03:56 ( Updated at: 10 Oct 2024, 06:52 )

RE: Backtester/Optimiser not matching with same parameters

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

davidrob
10 Oct 2024, 03:56 ( Updated at: 10 Oct 2024, 06:52 )

RE: Backtester/Optimiser not matching with same parameters

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

davidrob
10 Oct 2024, 10:27

RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. 

1.

davidrob
10 Oct 2024, 10:27

RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. 

1.

PanagiotisCharalampous
10 Oct 2024, 10:41

RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. 

1.

Hi there,

There is no information in your message.

Best regards,

Panagiotis

davidrob
10 Oct 2024, 11:36 ( Updated at: 10 Oct 2024, 11:41 )

RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Sorry somehow I stuffed up the forum formatting… Arghhh… Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. I am running BlackBull v 5.0.38.32105. (JUMP to the conclusion at the bottom if the whole process (most likely) bores you.

All tests were on the EURUSD.

1.    I tried a new bot, RSI Sample, and carefully made my tests. All worked perfectly.
2.    I went back to my original bot, "Macd & Rsi Strategy", deleted it and reinstalled.  Changed no parms.
3.    Set the BackTest (BT) date parms… 01/09/2024 - 09/10/2024.
4.    Ran BT, checked to make sure no open positions. Profit(P) = 201.36
5.    Setup Optimisation (OPT) set dates again, unticked ALL parms, set TimeFrame(TF) to h1;h4, 
6.    Ran OPT, Ph1=201.36, Ph4=-204.52.
7.    Set OPT parms again, to m30;m45;h1. Checked my dates again, and ran. Chose the m45 OPT line (P=-894.90) clicked APPLY.
8.    Noted correct change in left panel instance. 
9.    Ran BT :  Pm45=-786.98.  Aha…. DIFFERENT. 
10.   Compared the BT/OPT trade lists, and the number of trades varied, one was 17, one was 18 (both no open pos's), and the last two trades opentimes, etc were all different. Nearl;y all the parms did not match. It's like that run used a different time window, or different source parms.  I also noted several of the openTimes in the OPT list were all set to 00:00:00.
11.  From the OPT lines, I APPLIED and BT'd the h1 timeframe… same as always…. 201.36.
12.   I tried OPT on several values.. m30;m45;h1;h2;h3;h4.  The m45 reported the correct matching BT value: Pm45=-786.98. Selected it - APPLY - BT - P=-786.98. Perfect.
13.  I thought WTF?  This is a prick of a bug.  Must be the combination of values… M30;m45;h1. OPT'd them and all was perfect.
14.  So I went back to the beginning, deleted the bot, closed CTrader, reopened it, reinstalled.
15.   Setup the same tests… BT- set dates.  (I noticed the remembered FROM date was 09/09/2024, not 01/09/2024. The remembered end date was correct at 09/10/2024.
16.   Reset the BT start date, ran the BT - Ph1=201.36 same so far.
17.  OPT - Noted same date issue, fixed that, set OPT values to h1;h4, and ran.  Ph1 = 201.36, Ph4= -204.52. matches for all my tests.
18.  OPT - set TF values to m30;m45,h1.  Ran again, m30 and h1 P values same as along. m45 value was the bum value… -894.90.
19.  Subsequent OPT's and the bum value remains.
20.  I'm tired but I just had a thought…. Set the left panel timeframe to h1 and run an OPT with the TF parms set to m30;m45,h1. 
21.   Now, Set the left panel timeframe to m45 and run the same OPT… and you get the correct value! 
22.   One more test… set the left panel timeframe to m45, run, set it to m20  and run again… the same OPT…  all P values are correct.
23. Set the left panel to h1, OPT again, and the Pm45 value of -894.90 is wrong. 
24. CONCLUSION: the bug has to do with the OPT TimeFrame combo of m30;m45,h1, and the left panel setting of h1.
25. Hope that helps!

PanagiotisCharalampous
10 Oct 2024, 12:51

RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Sorry somehow I stuffed up the forum formatting… Arghhh… Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. I am running BlackBull v 5.0.38.32105. (JUMP to the conclusion at the bottom if the whole process (most likely) bores you.

All tests were on the EURUSD.

1.    I tried a new bot, RSI Sample, and carefully made my tests. All worked perfectly.
2.    I went back to my original bot, "Macd & Rsi Strategy", deleted it and reinstalled.  Changed no parms.
3.    Set the BackTest (BT) date parms… 01/09/2024 - 09/10/2024.
4.    Ran BT, checked to make sure no open positions. Profit(P) = 201.36
5.    Setup Optimisation (OPT) set dates again, unticked ALL parms, set TimeFrame(TF) to h1;h4, 
6.    Ran OPT, Ph1=201.36, Ph4=-204.52.
7.    Set OPT parms again, to m30;m45;h1. Checked my dates again, and ran. Chose the m45 OPT line (P=-894.90) clicked APPLY.
8.    Noted correct change in left panel instance. 
9.    Ran BT :  Pm45=-786.98.  Aha…. DIFFERENT. 
10.   Compared the BT/OPT trade lists, and the number of trades varied, one was 17, one was 18 (both no open pos's), and the last two trades opentimes, etc were all different. Nearl;y all the parms did not match. It's like that run used a different time window, or different source parms.  I also noted several of the openTimes in the OPT list were all set to 00:00:00.
11.  From the OPT lines, I APPLIED and BT'd the h1 timeframe… same as always…. 201.36.
12.   I tried OPT on several values.. m30;m45;h1;h2;h3;h4.  The m45 reported the correct matching BT value: Pm45=-786.98. Selected it - APPLY - BT - P=-786.98. Perfect.
13.  I thought WTF?  This is a prick of a bug.  Must be the combination of values… M30;m45;h1. OPT'd them and all was perfect.
14.  So I went back to the beginning, deleted the bot, closed CTrader, reopened it, reinstalled.
15.   Setup the same tests… BT- set dates.  (I noticed the remembered FROM date was 09/09/2024, not 01/09/2024. The remembered end date was correct at 09/10/2024.
16.   Reset the BT start date, ran the BT - Ph1=201.36 same so far.
17.  OPT - Noted same date issue, fixed that, set OPT values to h1;h4, and ran.  Ph1 = 201.36, Ph4= -204.52. matches for all my tests.
18.  OPT - set TF values to m30;m45,h1.  Ran again, m30 and h1 P values same as along. m45 value was the bum value… -894.90.
19.  Subsequent OPT's and the bum value remains.
20.  I'm tired but I just had a thought…. Set the left panel timeframe to h1 and run an OPT with the TF parms set to m30;m45,h1. 
21.   Now, Set the left panel timeframe to m45 and run the same OPT… and you get the correct value! 
22.   One more test… set the left panel timeframe to m45, run, set it to m20  and run again… the same OPT…  all P values are correct.
23. Set the left panel to h1, OPT again, and the Pm45 value of -894.90 is wrong. 
24. CONCLUSION: the bug has to do with the OPT TimeFrame combo of m30;m45,h1, and the left panel setting of h1.
25. Hope that helps!

 

Hi,

I still need the information I asked for to reproduce the problem.

Best regards,

Panagiotis

davidrob
10 Oct 2024, 22:36

RE: RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

PanagiotisCharalampous said: 

davidrob said: 

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Sorry somehow I stuffed up the forum formatting… Arghhh… Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. I am running BlackBull v 5.0.38.32105. (JUMP to the conclusion at the bottom if the whole process (most likely) bores you.

All tests were on the EURUSD.

1.    I tried a new bot, RSI Sample, and carefully made my tests. All worked perfectly.
2.    I went back to my original bot, "Macd & Rsi Strategy", deleted it and reinstalled.  Changed no parms.
3.    Set the BackTest (BT) date parms… 01/09/2024 - 09/10/2024.
4.    Ran BT, checked to make sure no open positions. Profit(P) = 201.36
5.    Setup Optimisation (OPT) set dates again, unticked ALL parms, set TimeFrame(TF) to h1;h4, 
6.    Ran OPT, Ph1=201.36, Ph4=-204.52.
7.    Set OPT parms again, to m30;m45;h1. Checked my dates again, and ran. Chose the m45 OPT line (P=-894.90) clicked APPLY.
8.    Noted correct change in left panel instance. 
9.    Ran BT :  Pm45=-786.98.  Aha…. DIFFERENT. 
10.   Compared the BT/OPT trade lists, and the number of trades varied, one was 17, one was 18 (both no open pos's), and the last two trades opentimes, etc were all different. Nearl;y all the parms did not match. It's like that run used a different time window, or different source parms.  I also noted several of the openTimes in the OPT list were all set to 00:00:00.
11.  From the OPT lines, I APPLIED and BT'd the h1 timeframe… same as always…. 201.36.
12.   I tried OPT on several values.. m30;m45;h1;h2;h3;h4.  The m45 reported the correct matching BT value: Pm45=-786.98. Selected it - APPLY - BT - P=-786.98. Perfect.
13.  I thought WTF?  This is a prick of a bug.  Must be the combination of values… M30;m45;h1. OPT'd them and all was perfect.
14.  So I went back to the beginning, deleted the bot, closed CTrader, reopened it, reinstalled.
15.   Setup the same tests… BT- set dates.  (I noticed the remembered FROM date was 09/09/2024, not 01/09/2024. The remembered end date was correct at 09/10/2024.
16.   Reset the BT start date, ran the BT - Ph1=201.36 same so far.
17.  OPT - Noted same date issue, fixed that, set OPT values to h1;h4, and ran.  Ph1 = 201.36, Ph4= -204.52. matches for all my tests.
18.  OPT - set TF values to m30;m45,h1.  Ran again, m30 and h1 P values same as along. m45 value was the bum value… -894.90.
19.  Subsequent OPT's and the bum value remains.
20.  I'm tired but I just had a thought…. Set the left panel timeframe to h1 and run an OPT with the TF parms set to m30;m45,h1. 
21.   Now, Set the left panel timeframe to m45 and run the same OPT… and you get the correct value! 
22.   One more test… set the left panel timeframe to m45, run, set it to m20  and run again… the same OPT…  all P values are correct.
23. Set the left panel to h1, OPT again, and the Pm45 value of -894.90 is wrong. 
24. CONCLUSION: the bug has to do with the OPT TimeFrame combo of m30;m45,h1, and the left panel setting of h1.
25. Hope that helps!

 

Hi,

I still need the information I asked for to reproduce the problem.

Best regards,

Panagiotis

Info attached - gees I was hoping I could attach a file containing all the info, but find that option not available above. Can I email it somehow?

 

davidrob
10 Oct 2024, 22:36

RE: RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

PanagiotisCharalampous said: 

davidrob said: 

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Sorry somehow I stuffed up the forum formatting… Arghhh… Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. I am running BlackBull v 5.0.38.32105. (JUMP to the conclusion at the bottom if the whole process (most likely) bores you.

All tests were on the EURUSD.

1.    I tried a new bot, RSI Sample, and carefully made my tests. All worked perfectly.
2.    I went back to my original bot, "Macd & Rsi Strategy", deleted it and reinstalled.  Changed no parms.
3.    Set the BackTest (BT) date parms… 01/09/2024 - 09/10/2024.
4.    Ran BT, checked to make sure no open positions. Profit(P) = 201.36
5.    Setup Optimisation (OPT) set dates again, unticked ALL parms, set TimeFrame(TF) to h1;h4, 
6.    Ran OPT, Ph1=201.36, Ph4=-204.52.
7.    Set OPT parms again, to m30;m45;h1. Checked my dates again, and ran. Chose the m45 OPT line (P=-894.90) clicked APPLY.
8.    Noted correct change in left panel instance. 
9.    Ran BT :  Pm45=-786.98.  Aha…. DIFFERENT. 
10.   Compared the BT/OPT trade lists, and the number of trades varied, one was 17, one was 18 (both no open pos's), and the last two trades opentimes, etc were all different. Nearl;y all the parms did not match. It's like that run used a different time window, or different source parms.  I also noted several of the openTimes in the OPT list were all set to 00:00:00.
11.  From the OPT lines, I APPLIED and BT'd the h1 timeframe… same as always…. 201.36.
12.   I tried OPT on several values.. m30;m45;h1;h2;h3;h4.  The m45 reported the correct matching BT value: Pm45=-786.98. Selected it - APPLY - BT - P=-786.98. Perfect.
13.  I thought WTF?  This is a prick of a bug.  Must be the combination of values… M30;m45;h1. OPT'd them and all was perfect.
14.  So I went back to the beginning, deleted the bot, closed CTrader, reopened it, reinstalled.
15.   Setup the same tests… BT- set dates.  (I noticed the remembered FROM date was 09/09/2024, not 01/09/2024. The remembered end date was correct at 09/10/2024.
16.   Reset the BT start date, ran the BT - Ph1=201.36 same so far.
17.  OPT - Noted same date issue, fixed that, set OPT values to h1;h4, and ran.  Ph1 = 201.36, Ph4= -204.52. matches for all my tests.
18.  OPT - set TF values to m30;m45,h1.  Ran again, m30 and h1 P values same as along. m45 value was the bum value… -894.90.
19.  Subsequent OPT's and the bum value remains.
20.  I'm tired but I just had a thought…. Set the left panel timeframe to h1 and run an OPT with the TF parms set to m30;m45,h1. 
21.   Now, Set the left panel timeframe to m45 and run the same OPT… and you get the correct value! 
22.   One more test… set the left panel timeframe to m45, run, set it to m20  and run again… the same OPT…  all P values are correct.
23. Set the left panel to h1, OPT again, and the Pm45 value of -894.90 is wrong. 
24. CONCLUSION: the bug has to do with the OPT TimeFrame combo of m30;m45,h1, and the left panel setting of h1.
25. Hope that helps!

 

Hi,

I still need the information I asked for to reproduce the problem.

Best regards,

Panagiotis

Info attached - gees I was hoping I could attach a file containing all the info, but find that option not available above. Can I email it somehow?

 

PanagiotisCharalampous
11 Oct 2024, 05:18

RE: RE: RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

PanagiotisCharalampous said: 

davidrob said: 

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Sorry somehow I stuffed up the forum formatting… Arghhh… Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. I am running BlackBull v 5.0.38.32105. (JUMP to the conclusion at the bottom if the whole process (most likely) bores you.

All tests were on the EURUSD.

1.    I tried a new bot, RSI Sample, and carefully made my tests. All worked perfectly.
2.    I went back to my original bot, "Macd & Rsi Strategy", deleted it and reinstalled.  Changed no parms.
3.    Set the BackTest (BT) date parms… 01/09/2024 - 09/10/2024.
4.    Ran BT, checked to make sure no open positions. Profit(P) = 201.36
5.    Setup Optimisation (OPT) set dates again, unticked ALL parms, set TimeFrame(TF) to h1;h4, 
6.    Ran OPT, Ph1=201.36, Ph4=-204.52.
7.    Set OPT parms again, to m30;m45;h1. Checked my dates again, and ran. Chose the m45 OPT line (P=-894.90) clicked APPLY.
8.    Noted correct change in left panel instance. 
9.    Ran BT :  Pm45=-786.98.  Aha…. DIFFERENT. 
10.   Compared the BT/OPT trade lists, and the number of trades varied, one was 17, one was 18 (both no open pos's), and the last two trades opentimes, etc were all different. Nearl;y all the parms did not match. It's like that run used a different time window, or different source parms.  I also noted several of the openTimes in the OPT list were all set to 00:00:00.
11.  From the OPT lines, I APPLIED and BT'd the h1 timeframe… same as always…. 201.36.
12.   I tried OPT on several values.. m30;m45;h1;h2;h3;h4.  The m45 reported the correct matching BT value: Pm45=-786.98. Selected it - APPLY - BT - P=-786.98. Perfect.
13.  I thought WTF?  This is a prick of a bug.  Must be the combination of values… M30;m45;h1. OPT'd them and all was perfect.
14.  So I went back to the beginning, deleted the bot, closed CTrader, reopened it, reinstalled.
15.   Setup the same tests… BT- set dates.  (I noticed the remembered FROM date was 09/09/2024, not 01/09/2024. The remembered end date was correct at 09/10/2024.
16.   Reset the BT start date, ran the BT - Ph1=201.36 same so far.
17.  OPT - Noted same date issue, fixed that, set OPT values to h1;h4, and ran.  Ph1 = 201.36, Ph4= -204.52. matches for all my tests.
18.  OPT - set TF values to m30;m45,h1.  Ran again, m30 and h1 P values same as along. m45 value was the bum value… -894.90.
19.  Subsequent OPT's and the bum value remains.
20.  I'm tired but I just had a thought…. Set the left panel timeframe to h1 and run an OPT with the TF parms set to m30;m45,h1. 
21.   Now, Set the left panel timeframe to m45 and run the same OPT… and you get the correct value! 
22.   One more test… set the left panel timeframe to m45, run, set it to m20  and run again… the same OPT…  all P values are correct.
23. Set the left panel to h1, OPT again, and the Pm45 value of -894.90 is wrong. 
24. CONCLUSION: the bug has to do with the OPT TimeFrame combo of m30;m45,h1, and the left panel setting of h1.
25. Hope that helps!

 

Hi,

I still need the information I asked for to reproduce the problem.

Best regards,

Panagiotis

Info attached - gees I was hoping I could attach a file containing all the info, but find that option not available above. Can I email it somehow?

 

 

You can send it to community@ctrader.com

davidrob
17 Oct 2024, 21:45

RE: RE: RE: RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob
17 Oct 2024, 21:46

RE: RE: RE: RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob
17 Oct 2024, 21:48

RE: RE: RE: RE: RE: RE: RE: RE: Backtester/Optimiser not matching with same parameters

davidrob said: 

PanagiotisCharalampous said: 

davidrob said: 

PanagiotisCharalampous said: 

davidrob said: 

davidrob said: 

davidrob said: 

PanagiotisCharalampous said: 

Hi there,

Make sure that the backtesting settings are the same. 99% of the times it happened this to me it was caused by some oversight in my backtesing settings. If there is still a problem, then we will need to following information in order to reproduce the problem

• cBot code
• Optimization parameters file, symbol, timeframe and dates
• The exact set of parameters for which the discrepancy is observed

Best regards,

Panagiotis

Hi, Thanks your response, I will try a third evening to figure this issue out, starting from scratch on a different cBot, and report back.

Sorry somehow I stuffed up the forum formatting… Arghhh… Ok, another couple of hours and I managed to reproduce the issue. I am sure it is a bug (I am a programmer myself). Two solid pages of notes in fine print… I will try to best explain it, maybe you can reproduce it. I am old to debugging, but new to forums, so please advise if I should do something differently. I am running BlackBull v 5.0.38.32105. (JUMP to the conclusion at the bottom if the whole process (most likely) bores you.

All tests were on the EURUSD.

1.    I tried a new bot, RSI Sample, and carefully made my tests. All worked perfectly.
2.    I went back to my original bot, "Macd & Rsi Strategy", deleted it and reinstalled.  Changed no parms.
3.    Set the BackTest (BT) date parms… 01/09/2024 - 09/10/2024.
4.    Ran BT, checked to make sure no open positions. Profit(P) = 201.36
5.    Setup Optimisation (OPT) set dates again, unticked ALL parms, set TimeFrame(TF) to h1;h4, 
6.    Ran OPT, Ph1=201.36, Ph4=-204.52.
7.    Set OPT parms again, to m30;m45;h1. Checked my dates again, and ran. Chose the m45 OPT line (P=-894.90) clicked APPLY.
8.    Noted correct change in left panel instance. 
9.    Ran BT :  Pm45=-786.98.  Aha…. DIFFERENT. 
10.   Compared the BT/OPT trade lists, and the number of trades varied, one was 17, one was 18 (both no open pos's), and the last two trades opentimes, etc were all different. Nearl;y all the parms did not match. It's like that run used a different time window, or different source parms.  I also noted several of the openTimes in the OPT list were all set to 00:00:00.
11.  From the OPT lines, I APPLIED and BT'd the h1 timeframe… same as always…. 201.36.
12.   I tried OPT on several values.. m30;m45;h1;h2;h3;h4.  The m45 reported the correct matching BT value: Pm45=-786.98. Selected it - APPLY - BT - P=-786.98. Perfect.
13.  I thought WTF?  This is a prick of a bug.  Must be the combination of values… M30;m45;h1. OPT'd them and all was perfect.
14.  So I went back to the beginning, deleted the bot, closed CTrader, reopened it, reinstalled.
15.   Setup the same tests… BT- set dates.  (I noticed the remembered FROM date was 09/09/2024, not 01/09/2024. The remembered end date was correct at 09/10/2024.
16.   Reset the BT start date, ran the BT - Ph1=201.36 same so far.
17.  OPT - Noted same date issue, fixed that, set OPT values to h1;h4, and ran.  Ph1 = 201.36, Ph4= -204.52. matches for all my tests.
18.  OPT - set TF values to m30;m45,h1.  Ran again, m30 and h1 P values same as along. m45 value was the bum value… -894.90.
19.  Subsequent OPT's and the bum value remains.
20.  I'm tired but I just had a thought…. Set the left panel timeframe to h1 and run an OPT with the TF parms set to m30;m45,h1. 
21.   Now, Set the left panel timeframe to m45 and run the same OPT… and you get the correct value! 
22.   One more test… set the left panel timeframe to m45, run, set it to m20  and run again… the same OPT…  all P values are correct.
23. Set the left panel to h1, OPT again, and the Pm45 value of -894.90 is wrong. 
24. CONCLUSION: the bug has to do with the OPT TimeFrame combo of m30;m45,h1, and the left panel setting of h1.
25. Hope that helps!

 

Hi,

I still need the information I asked for to reproduce the problem.

Best regards,

Panagiotis

Info attached - gees I was hoping I could attach a file containing all the info, but find that option not available above. Can I email it somehow?

 

 

You can send it to community@ctrader.com

Hi Panagiotis, the information was sent a few days ago. I will be keen to hear the fix to the issue. Thanks.

PanagiotisCharalampous
18 Oct 2024, 06:04

Hi there,

We have received the information and the issue will be investigated soon.

Best regards,

Panagiotis

martins
11 Nov 2024, 22:38 ( Updated at: 11 Nov 2024, 22:54 )

RE: Backtester/Optimiser not matching with same parameters

PanagiotisCharalampous said: 

Hi there,

We have received the information and the issue will be investigated soon.

Best regards,

Panagiotis

I have found that it can be ‘correct’ for there to be differences between backtests on v4 vs v5 because v5 handes swap differently - it deducts it from the Account.Equity around 22:00 on days it is due, whereas v4 didn't seem to ever deduct swap (it just reported an accumulated amount at the end) - as far as I can see by comparing detailed logs of same test on both versions. So if decisions are based on equity, they will be different.

Whilst having more accurate swap treatment is commendable, it is highly inconvenient to not be able to exactly recreate old saved results in order to know whether a change to the bot has improved it or not. An option to again NOT deduct swap would also act as a sanity checker / confidence booster for our migration to v5.

To address the above, please can there be consideration of adding an option to disable the newly imposed mandatory deduction of swap from equity in backtest & optimise, and an option to specify a fixed custom rate (so old tests can still be recreated even if the automatic rate changes), in much the same way as there is already an option to set commission & spread.  Thanks for reading, Martin