Backtesting really, really slow when using nested custom indicator

PanagiotisCharalampous
18 Feb 2021, 14:02

Hi Christian,

Can you please post the complete cBot and Indicator source codes and steps to reproduce the problem?

Best Regards,

Panagiotis 

Join us on Telegram

amch
18 Feb 2021, 14:10 ( Updated at: 18 Feb 2021, 14:26 )

RE:

PanagiotisCharalampous said:

Hi Christian,

Can you please post the complete cBot and Indicator source codes and steps to reproduce the problem?

Best Regards,

Panagiotis 

Join us on Telegram

Please find below the indicator as well as the cBot:

using System;
using cAlgo.API;
using cAlgo.API.Internals;
using cAlgo.API.Indicators;
using cAlgo.Indicators;

using UnscentedKalmanFilter;
using primeAlgoCore.Enums;

namespace cAlgo
{
    [Indicator(IsOverlay = true, TimeZone = TimeZones.UTC, AccessRights = AccessRights.None)]
    public class iUnscentedKalmanFilterRenko : Indicator
    {
        [Parameter("Source")]
        public DataSeries Source { get; set; }

        [Parameter("MA Periods", DefaultValue = 14, MinValue = 1, MaxValue = 55)]
        public int Periods { get; set; }

        [Parameter("Threshold", DefaultValue = 0.00015)]
        public double Threshold { get; set; }

        [Parameter("Brick size", DefaultValue = 0.0005)]
        public double BrickSize { get; set; }



        [Output("Neutral", LineColor = "Gray")]
        public IndicatorDataSeries Neutral { get; set; }

        [Output("Buy", LineColor = "LimeGreen", Thickness = 2, PlotType = PlotType.DiscontinuousLine)]
        public IndicatorDataSeries Buy { get; set; }

        [Output("Sell", LineColor = "Red", Thickness = 2, PlotType = PlotType.DiscontinuousLine)]
        public IndicatorDataSeries Sell { get; set; }

        [Output("Renko", LineColor = "Blue", Thickness = 2, PlotType = PlotType.DiscontinuousLine)]
        public IndicatorDataSeries Renko { get; set; }


        private UKF filter;

        private IndicatorDataSeries filterIndi;
        public MovingAverage filterMA;

        public TrendType CurrentTrend;

        private int CountBars;
        private double renkoValue = 0;
        public TrendType renkoTrendType = TrendType.None;
        public double prevUKF;
        public double prevUKF2;

        private int CurrentIndex = -1;

        private int ukfIndex = -1;


        protected override void Initialize()
        {
            filter = new UKF();

            filterIndi = CreateDataSeries();

            filterMA = Indicators.MovingAverage(filterIndi, Periods, MovingAverageType.Hull);

            CountBars = 0;

            
        }

        public override void Calculate(int index)
        {
            CurrentIndex = index;

            if (CurrentIndex == 0)
            {
                filter.Update(new[] 
                {
                    Source[0]
                });

                ukfIndex++;

                filterIndi[ukfIndex] = filter.getState()[0];


                return;
            }

            if (CurrentIndex == 1)
            {
                ukfIndex++;

                filter.Update(new[] 
                {
                    Source[0]
                });

                filterIndi[ukfIndex] = filter.getState()[0];

                Neutral[CurrentIndex] = filterMA.Result[CurrentIndex];

                renkoValue = Bars.ClosePrices[CurrentIndex];
                //Renko[CurrentIndex] = renkoValue;
                if (Bars.ClosePrices[CurrentIndex] - Bars.ClosePrices[CurrentIndex - 1] > 0)
                    renkoTrendType = TrendType.Up;
                else if (Bars.ClosePrices[CurrentIndex] - Bars.ClosePrices[CurrentIndex - 1] < 0)
                    renkoTrendType = TrendType.Down;


                return;

            }


            double renkoUpperLimit = RenkoUpperLimit();
            double renkoLowerLimit = RenkoLowerLimit();

            if (CurrentIndex > CountBars)
            {
                double lastPrice = Bars.ClosePrices[CurrentIndex];

                if (lastPrice > renkoUpperLimit && renkoTrendType == TrendType.Up)
                {
                    renkoValue += Math.Floor((lastPrice - renkoValue) / BrickSize) * BrickSize;
                    Renko[CurrentIndex - 1] = renkoValue;

                    SetKalmanFilter();

                }
                else if (lastPrice < renkoLowerLimit && renkoTrendType == TrendType.Up)
                {
                    renkoValue -= Math.Floor((renkoValue - lastPrice) / BrickSize) * BrickSize;
                    renkoTrendType = TrendType.Down;

                    Renko[CurrentIndex - 1] = renkoValue;
                    // LinesSeries[3].SetMarker(1, new IndicatorLineMarker(Color.Red, upperIcon: IndicatorLineMarkerIconType.FillCircle));

                    SetKalmanFilter();

                }
                else if (lastPrice > renkoUpperLimit && renkoTrendType == TrendType.Down)
                {
                    renkoValue += Math.Floor((lastPrice - renkoValue) / BrickSize) * BrickSize;
                    renkoTrendType = TrendType.Up;

                    Renko[CurrentIndex - 1] = renkoValue;
                    // LinesSeries[3].SetMarker(1, new IndicatorLineMarker(Color.Green, bottomIcon: IndicatorLineMarkerIconType.FillCircle));

                    SetKalmanFilter();

                }
                else if (lastPrice < renkoLowerLimit && renkoTrendType == TrendType.Down)
                {
                    renkoValue -= Math.Floor((renkoValue - lastPrice) / BrickSize) * BrickSize;

                    SetKalmanFilter();
                    Renko[CurrentIndex - 1] = renkoValue;

                }
                else
                {
                    Renko[CurrentIndex - 1] = renkoValue;
                    Neutral[CurrentIndex - 1] = Neutral[CurrentIndex - 2];
                    Buy[CurrentIndex - 1] = Buy[CurrentIndex - 2];
                    Sell[CurrentIndex - 1] = Sell[CurrentIndex - 2];
                }

                CountBars = CurrentIndex;
            }

        }

        private void SetKalmanFilter()
        {
            filter.Update(new[] 
            {
                Source[CurrentIndex]
            });

            ukfIndex++;

            filterIndi[ukfIndex] = filter.getState()[0];

            Neutral[CurrentIndex - 1] = filterMA.Result[ukfIndex];

            prevUKF2 = prevUKF;
            prevUKF = Neutral[CurrentIndex - 2];



            Buy[CurrentIndex - 1] = double.NaN;
            Sell[CurrentIndex - 1] = double.NaN;


            double delta1 = filterMA.Result[ukfIndex] - filterMA.Result[ukfIndex - 1];
            double delta2 = filterMA.Result[ukfIndex] - filterMA.Result[ukfIndex - 2];

            //Core.Loggers.Log($"KF Value: {Settings.FirstOrDefault(i => i.Name.Equals("Sharpness")).Value}", LoggingLevel.System);

            if ((Math.Abs(delta1) > Threshold) || (Math.Abs(delta2) > Threshold))
            {
                if (delta1 > Threshold || delta2 > Threshold)
                {
                    Buy[CurrentIndex - 1] = Neutral[CurrentIndex - 1];
                    CurrentTrend = TrendType.Up;
                }

                else if (delta1 < -Threshold || delta2 < -Threshold)
                {
                    Sell[CurrentIndex - 1] = Neutral[CurrentIndex - 1];
                    CurrentTrend = TrendType.Down;
                }
            }

        }

        private double RenkoUpperLimit()
        {
            return renkoTrendType == TrendType.Up ? renkoValue + BrickSize : renkoValue + 2 * BrickSize;
        }

        private double RenkoLowerLimit()
        {
            return renkoTrendType == TrendType.Up ? renkoValue - 2 * BrickSize : renkoValue - BrickSize;
        }


    }
}

using System;
using System.Linq;
using cAlgo.API;
using cAlgo.API.Indicators;
using cAlgo.API.Internals;
using cAlgo.Indicators;

using cAlgo;

using primeAlgoCore.Enums;

namespace cAlgo.Robots
{
    [Robot(TimeZone = TimeZones.UTC, AccessRights = AccessRights.None)]
    public class primeAlgoUFK : Robot
    {
        [Parameter("Source")]
        public DataSeries Source { get; set; }

        [Parameter("MA Periods", DefaultValue = 14, MinValue = 1, MaxValue = 55)]
        public int Periods { get; set; }

        [Parameter("Threshold", DefaultValue = 0.00015)]
        public double Threshold { get; set; }

        [Parameter("Brick size", DefaultValue = 0.0005)]
        public double BrickSize { get; set; }

        [Parameter("Quantity (Lots)", DefaultValue = 1, MinValue = 0.01, Step = 0.01)]
        public double Quantity { get; set; }

        private const string label = "Sample Trend cBot";

        //private TrendType CurrentFastTrend;

        private const double TrailingStopTrigger = 0;
        private const double TrailingStopStep = 0;

        private const double StopLossPips = double.NaN;
        private const double TakeProfitPips = double.NaN;

        public iUnscentedKalmanFilterRenko UnscentedKalman;



        protected override void OnStart()
        {
            UnscentedKalman = Indicators.GetIndicator<iUnscentedKalmanFilterRenko>(Source, Periods, Threshold, BrickSize);
        }

        protected override void OnTick()
        {
            // Put your core logic here
        }

        protected override void OnBar()
        {

            // Print("{0:0.00000}   {1:0.00000}", SlowKalmanRenko.Neutral.LastOrDefault(), FastKalmanRenko.Neutral.LastOrDefault());
            // Print(Bars.TickVolumes.Last(1));
            if (double.IsNaN(UnscentedKalman.Renko.Last(0)) || double.IsNaN(UnscentedKalman.Renko.Last(1)))
                return;

            if (UnscentedKalman.Renko.Last(0) == UnscentedKalman.Renko.Last(1))
                return;

            //Print("Bar: {0}   {1}   Renko: {2:0.0000(0)}   {3:0.0000(0)} Renko Trend: {4}", Bars.OpenTimes.Last(), Bars.OpenPrices.Last(), UnscentedKalman.prevUKF2, UnscentedKalman.prevUKF, UnscentedKalman.renkoTrendType.ToString());

            // SetFastTrend();

            var longPosition = Positions.Find(label, SymbolName, TradeType.Buy);
            var shortPosition = Positions.Find(label, SymbolName, TradeType.Sell);

            //var currentSlowMa = SlowKalmanRenko.Neutral.Last(1);
            //var currentFastMa = FastKalmanRenko.Neutral.Last(1);
            //var previousSlowMa = SlowKalmanRenko.Neutral.Last(2);
            //var previousFastMa = FastKalmanRenko.Neutral.Last(2);

            //Print("{0:0.00000}   {1:0.00000}   cs: {2:0.00000}   cf: {3:0.00000}   ps: {4:0.00000}   pf: {5:0.00000}", previousSlowMa > previousFastMa && currentSlowMa < currentFastMa, previousSlowMa < previousFastMa && currentSlowMa > currentFastMa, currentSlowMa, currentFastMa, previousFastMa, previousSlowMa);

            // if (previousSlowMa > previousFastMa && currentSlowMa < currentFastMa && longPosition == null)
            // && CurrentFastTrend == TrendType.Up)



            if (UnscentedKalman.CurrentTrend == TrendType.Up && longPosition == null)
            {
                if (shortPosition != null)
                    ClosePosition(shortPosition);
                var result = ExecuteMarketOrder(TradeType.Buy, SymbolName, VolumeInUnits, label, null, TakeProfitPips);
                if (result.IsSuccessful)
                {
                    var position = result.Position;
                    // Print("Position SL price is {0}", position.StopLoss);

                    if (!double.IsNaN(StopLossPips))
                    {
                        var stopLoss = position.EntryPrice - StopLossPips * Symbol.PipSize;
                        ModifyPosition(position, stopLoss, position.TakeProfit);
                    }

                    // SetTrailingStop();


                    // Print("New Position SL price is {0}", position.StopLoss);

                }
            }
            // else if (previousSlowMa < previousFastMa && currentSlowMa >= currentFastMa && shortPosition == null)
            // && CurrentFastTrend == TrendType.Down)
            else if (UnscentedKalman.CurrentTrend == TrendType.Down && shortPosition == null)
            {
                if (longPosition != null)
                    ClosePosition(longPosition);
                var result = ExecuteMarketOrder(TradeType.Sell, SymbolName, VolumeInUnits, label, null, TakeProfitPips);
                if (result.IsSuccessful)
                {
                    var position = result.Position;
                    // Print("Position SL price is {0}", position.StopLoss);

                    if (!double.IsNaN(StopLossPips))
                    {
                        var stopLoss = position.EntryPrice + StopLossPips * Symbol.PipSize;
                        ModifyPosition(position, stopLoss, position.TakeProfit);
                    }

                    // SetTrailingStop();

                    // Print("New Position SL price is {0}", position.StopLoss);

                }
            }
        }
        protected override void OnStop()
        {
            // Put your deinitialization logic here
        }

        private double VolumeInUnits
        {
            get { return Symbol.QuantityToVolumeInUnits(Quantity); }
        }

        //private TrendType GetSlowTrend()
        //{
        //    var trend = TrendType.None;

        //    double delta = UnscentedKalman.Neutral.Last(0) - UnscentedKalman.prevUKF;
        //    double delta2 = UnscentedKalman.Neutral.Last(0) - UnscentedKalman.prevUKF2;

        //    if (delta > Threshold || delta2 > Threshold)
        //        trend = TrendType.Up;
        //    else if (delta < -Threshold || delta2 < -Threshold)
        //        trend = TrendType.Down;

        //    return trend;
        //}



        private void SetTrailingStop()
        {
            var sellPositions = Positions.FindAll(label, SymbolName, TradeType.Sell);

            foreach (Position position in sellPositions)
            {
                double distance = position.EntryPrice - Symbol.Ask;

                if (distance < TrailingStopTrigger * Symbol.PipSize)
                    continue;

                double newStopLossPrice = Symbol.Ask + TrailingStopStep * Symbol.PipSize;

                if (position.StopLoss == null || newStopLossPrice < position.StopLoss)
                {
                    ModifyPosition(position, newStopLossPrice, position.TakeProfit);
                }
            }

            var buyPositions = Positions.FindAll(label, SymbolName, TradeType.Buy);

            foreach (Position position in buyPositions)
            {
                double distance = Symbol.Bid - position.EntryPrice;

                if (distance < TrailingStopTrigger * Symbol.PipSize)
                    continue;

                double newStopLossPrice = Symbol.Bid - TrailingStopStep * Symbol.PipSize;
                if (position.StopLoss == null || newStopLossPrice > position.StopLoss)
                {
                    ModifyPosition(position, newStopLossPrice, position.TakeProfit);
                }
            }
        }

        protected void CloseAllPosition()
        {
            //foreach (var position in Positions.Find("MyLabel"))
            //{
            //    // Modify/Close positions
            //}
        }
    }
}

Hier the test parameter:

[ChartParameters]
Symbol = EURUSD
Timeframe = m1

[cBotParameters]
Source = Close
Periods = 30
Threshold = 5E-05
BrickSize = 0.0011
Quantity = 0.1

PanagiotisCharalampous
18 Feb 2021, 14:28

Hi Christian,

I cannot seem to be able to build the indicator. There are a lot of references missing.

In any case a slower backtesting does not necessarily mean that there is a problem. Some calculations are more complicated and take more time than others.

Best Regards,

Panagiotis 

Join us on Telegram

amch
18 Feb 2021, 15:01 ( Updated at: 21 Dec 2023, 09:22 )

RE:

PanagiotisCharalampous said:

Hi Christian,

I cannot seem to be able to build the indicator. There are a lot of references missing.

In any case a slower backtesting does not necessarily mean that there is a problem. Some calculations are more complicated and take more time than others.

Best Regards,

Panagiotis 

Join us on Telegram

Hello Panagiotis

The problem starts when smooth the UKF() results.

protected override void Initialize()
        {
            filter = new UKF();

            filterIndi = CreateDataSeries();

            filterMA = Indicators.MovingAverage(filterIndi, Periods, MovingAverageType.Hull);

            CountBars = 0;

            
        }

Without smoothing there is no performance issue.

Br

Christian