Different Backtesting and Optimization Results

andi21
14 Dec 2016, 20:27

RE:

andi21
15 Dec 2016, 12:39

First of all, i am sorry for my second post in this thread - i wanted to correct the first post, but then i posted a second one (i added the info "it uses multiple timeframes / marketseries data").

I think probably i have found something, that could have to do with the problem.

So when i run backtesting MarketData.GetSeries returns valid (Open.Count > 0) results for all the different timeframes i am using simultaneously in one bot.

But when i run optimization MarketData.GetSeries returns ONLY valid data for the current timeframe, but for all others it returns zero data (Open.count == 0).

@Spotware:

I think this is a bug - could you help me with this, please? Otherwise i (and other users using multiple timeframes in their bots) cannot go further with optimization.

Thanks in advance.

andi21
15 Dec 2016, 15:43 ( Updated at: 21 Dec 2023, 09:20 )

I have found one more problem (in addition to the problem of my last post) - look at the screenshot, please:

 So it seems, that if i have retrieved multiple marketdata of different timeframes only the current marketseries (the one of the robot, e.g. here in this example it is the hour-frame) seems to deliver correct data. 

How can that be (also please notice my previous post)?

andi21
15 Dec 2016, 17:05

Ok, i have investigated this further and found out more info:

1. series = MarketData.GetSeries(Symbol, TimeFrame.Hour) -> series.Open.Count = 125 // Hour is current Robot TimeFrame

2. series2 = MarketData.GetSeries(Symbol, TimeFrame.Hour12)

-> series.Open.Count = 125 // remains at 125

-> series2.Open.Count = 75

3. series3 = MarketData.GetSeries(Symbol, TimeFrame.Daily) 

-> series.Open.Count = 125 // remains at 125

-> series2.Open.Count = 148 // is now old value 75 + 73 from Daily ???

-> series3.Open.Count = 73

4. series4 = MarketData.GetSeries(Symbol, TimeFrame.Day2) 

-> series.Open.Count = 125 // remains at 125

-> series2.Open.Count = 235 // is now old wrong value 148 + 87 from Day2 ???

-> series3.Open.Count = 160 // is now old value 73 + 87 from Day2 ???

-> series4.Open.Count = 87

....

I think internally there is some "static"-problem - can that be?

Jiri
15 Dec 2016, 17:12

Hi Andi. There isn't anything wrong with different count values. I advice you to write an indicator ploting multiple timeframe Moving Averages into one chart so you will understand the logic. /forum/whats-new/1463

Jiri
15 Dec 2016, 17:33

I forgot to mention. When you are backtesting the market series loaded go more into past, the zero index isn't equal to the starting time of the backtest. The optimization is not loading any additional data.

andi21
15 Dec 2016, 18:15

Hi tmc.,

thanks for your reply.

I advice you to write an indicator ploting multiple timeframe Moving Averages into one chart so you will understand the logic

Ok, i will check that.

 When you are backtesting the market series loaded go more into past, the zero index isn't equal to the starting time of the backtest. The optimization is not loading any additional data.

It is loading data - but only for the current timeframe. All others stay empty. Ok, but am i able to get past data from another timeframe, because i need to make calculations based on that?

Jiri
15 Dec 2016, 18:26

RE:

andi21 said:

It is loading data - but only for the current timeframe. All others stay empty. Ok, but am i able to get past data from another timeframe, because i need to make calculations based on that?

No, the optimization starts to load the data from exact same time as you set the optimization.

You can contact me via email, pass the code and I will help you out with it.

Jiri
15 Dec 2016, 18:32

I suppose you are trying to calculate a value from past data that are not there so it returns you NaN value instead. You could pause the robot logic until it has loaded enough data for the calculation.

andi21
15 Dec 2016, 18:59

No, the optimization starts to load the data from exact same time as you set the optimization.

But i debugged the code and when i call MarketData.GetSeries for the current bot timeframe (i know the current data is also in .MarketSeries, but the bot is still under construction, so far away from release :) ) it returns the same marketdata like in backtesting, that is in this example, 125 hour-bars before the starttime-period. But when i call MarketData.GetSeries for any other timeframe than the current one i get zero data which is not really good because i need also past data from other timeframes.

You can contact me via email, pass the code and I will help you out with it.

Thank you very much, that is really nice of you. Unfortunately i am developing this bot already a long time and it consists of 4 different projects, so i hope you understand that i do not want to give my "baby away" ;). But like i said, that is very very nice of you. Thanks.

I suppose you are trying to calculate a value from past data that are not there so it returns you NaN value instead. You could pause the robot logic until it has loaded enough data for the calculation.

Yes, that is correct - i need past data and if there is not any past data available - which is always the case in real-time - the bot, of course cannot work like expected like in backtesting or real-environment.

Jiri
15 Dec 2016, 19:41

RE:

But i debugged the code and when i call MarketData.GetSeries for the current bot timeframe (i know the current data is also in .MarketSeries, but the bot is still under construction, so far away from release :) ) it returns the same marketdata like in backtesting, that is in this example, 125 hour-bars before the starttime-period. But when i call MarketData.GetSeries for any other timeframe than the current one i get zero data which is not really good because i need also past data from other timeframes.

Yes, I was talking about series obtained via GetSeries() method. The series of current chart timeframe are always loaded with historical data.

Thank you very much, that is really nice of you. Unfortunately i am developing this bot already a long time and it consists of 4 different projects, so i hope you understand that i do not want to give my "baby away" ;). But like i said, that is very very nice of you. Thanks.

I understand. I am not here to steal your robot or anything similar to this. It would just make helping you much easier. 

andi21
15 Dec 2016, 19:59

The series of current chart timeframe are always loaded with historical data.

Is it possible to load also historical data for another timeframe in optimization - like it is working in backtesting etc.?

It would just make helping you much easier.  

You are absolutely right.

Probably i have an idea (because the simple example which you posted did work without the "data-mess"):

1. Do you or anybody know if Indicators.GetIndicator<IndicatorXY>() always returns the same instance (so singleton-instance)?
2. Are Robot.MarketData and Indicator.MarketData always equal? So when i run a bot and the bot has one or multiple instances of IndicatorXY, is Indicator.MarketData the same as Robot.MarketData, so does the indicators get same MarketData?

andi21
16 Dec 2016, 13:10

Hello everyone back again,

i could finally create a simple test which can reproduce the problem:

using System;
using cAlgo.API;
using cAlgo.API.Internals;
using cAlgo.API.Indicators;
using cAlgo.Indicators;

namespace cAlgo
{
    [Indicator(IsOverlay = true, TimeZone = TimeZones.UTC, AccessRights = AccessRights.None)]
    public class MultiTF_MA : Indicator
    {
        [Parameter(DefaultValue = 50)]
        public int Period { get; set; }

        [Output("MA", Color = Colors.Yellow)]
        public IndicatorDataSeries MA { get; set; }

        [Output("MA5", Color = Colors.Orange)]
        public IndicatorDataSeries MA5 { get; set; }

        [Output("MA10", Color = Colors.Red)]
        public IndicatorDataSeries MA10 { get; set; }

        private MarketSeries series1;
        private MarketSeries series2;
        private MarketSeries series3;
        private MarketSeries series4;

        private MovingAverage ma;
        private MovingAverage ma5;
        private MovingAverage ma10;

        protected override void Initialize()
        {
            series1 = MarketData.GetSeries(TimeFrame.Hour);
            Print("1. Hour: " + series1.Open.Count);
            series2 = MarketData.GetSeries(TimeFrame.Hour12);
            Print("2. Hour: " + series1.Open.Count);
            Print("2. Hour12: " + series2.Open.Count);
            series3 = MarketData.GetSeries(TimeFrame.Daily);
            Print("3. Hour: " + series1.Open.Count);
            Print("3. Hour12: " + series2.Open.Count);
            Print("3. Daily: " + series3.Open.Count);
            series4 = MarketData.GetSeries(TimeFrame.Day2);
            Print("4. Hour: " + series1.Open.Count);
            Print("4. Hour12: " + series2.Open.Count);
            Print("4. Daily: " + series3.Open.Count);
            Print("4. Day2: " + series4.Open.Count);

            ma = Indicators.MovingAverage(MarketSeries.Close, Period, MovingAverageType.Triangular);
            ma5 = Indicators.MovingAverage(series1.Close, Period, MovingAverageType.Triangular);
            ma10 = Indicators.MovingAverage(series2.Close, Period, MovingAverageType.Triangular);
        }

        public override void Calculate(int index)
        {
            MA[index] = ma.Result[index];

            var index5 = GetIndexByDate(series1, MarketSeries.OpenTime[index]);
            if (index5 != -1)
                MA5[index] = ma5.Result[index5];

            var index10 = GetIndexByDate(series2, MarketSeries.OpenTime[index]);
            if (index10 != -1)
                MA10[index] = ma10.Result[index10];
        }


        private int GetIndexByDate(MarketSeries series, DateTime time)
        {
            for (int i = series.Close.Count - 1; i > 0; i--)
            {
                if (time == series.OpenTime[i])
                    return i;
            }
            return -1;
        }
    }
}

using System;
using System.Linq;
using cAlgo.API;
using cAlgo.API.Indicators;
using cAlgo.API.Internals;
using cAlgo.Indicators;

namespace cAlgo
{
    [Robot(TimeZone = TimeZones.UTC, AccessRights = AccessRights.None)]
    public class Test123Bot : Robot
    {
        private MultiTF_MA indicatorX;

        protected override void OnStart()
        {
            // Put your initialization logic here
            indicatorX = Indicators.GetIndicator<MultiTF_MA>(50);
        }

        protected override void OnTick()
        {
            // Put your core logic here
        }

        protected override void OnStop()
        {
            // Put your deinitialization logic here
        }
    }
}

Please start the bot (one day period is enough) and see how the counts of the different timeframes are increasing, although only other timeseries are created.

In my opinion it is a bug - or have i made something wrong in this simple example?

andi21
16 Dec 2016, 13:11

One last thing: so this problem does NOT exist if only the indicator is used. But i happens if the indicator is used in a bot.

andi21
16 Dec 2016, 13:19

And one more thing:

and it does NOT affect the current timeframe. But it only affects COMBINATIONS of different Timeframes other than the current. So you can see 2. is ok, but at 3. (so combination of Hour12 and Daily) it happens and 4.

By the way: i used hour as current timeframe in the bot, so you can see the same results.

Jiri
16 Dec 2016, 16:03 ( Updated at: 21 Dec 2023, 09:20 )

Yes, it seems like cAlgo platform is loading additional data for every used series everytime you load the chart. Same behaviour happens when you are forcing to reload the chart.

andi21
16 Dec 2016, 16:19

You are right. Really good catch tmc.

Hopefully this can be fixed as soon as possible. Perhaps also with an improvement (same topic "data") for this /forum/suggestions/11031.

That would be like christmas is coming earlier this year!

andi21
28 Dec 2016, 20:30

Dear Spotware,

can you give us an answer to this bug, please?

I have also already contacted you via feedback@spotware.com.

Best Regards,

Andi21

MartSmart
14 Jan 2017, 11:45

Dear Spotware

could you share your view on this issue and confirm if you plan to fix it and possibly when? I'm also struggling with getting right data during optimization of multi-timeframe algos

Thank you!

andi21
25 Jul 2017, 15:52

Dear Spotware,

have you reviewed this bug-case?

Thanks in advance.

Best regards,

andi21

Spotware
25 Jul 2017, 16:09

Hi andi21,

Thanks for bringing this up. We will check the status of this issue with the product team and update the thread soon.

Best Regards,

cTrader Team

Spotware
25 Jul 2017, 17:41

Dear Traders,

This issue is known to the cAlgo team and it is planned to be resolved in the next major release of cTrader. We will update you about this release soon.

Best Regards,

cTrader Team

der.timosch
24 Nov 2017, 00:03

Hallo Spotware,

what is the current status here? Will it still be resolved in the next major release? And what do you exactly mean by "resolved"? Will an optimisation run have the same history for different timeframes like in a backtest run ? Will it even be configurable how much history is needed ? When using an indicator like KAMA which is using result[index - 1] as part of the calculation for result[index] a longer history before the start may be needed as the settle phase might take a lot of time. With a "daily" timeframe it may need a year or even longer.

Best regards,

der.timosch