Tick LoadMoreHistory() problem

firemyst
16 Jan 2023, 08:23

Since you haven't posted all your code, will the while loop ever end? That is, what is "firstDt" set to and will that ever evaluate to true? Or is it just going to continuously load tick data back from the beginning of time because it'll never evaluate to true?

That would explain why it never stops loading and your CPU is at full capacity.

GridSurfer
16 Jan 2023, 08:36

Thanks firemyst for your thoughts, but yes it is a valid date. Usually 1 week before, because I update my tick data on the weekend. I have been using the script for years now.

I watched the execution with a debugger as well. The first few times it executes LoadMoreHistory() - which is always a 3 hour chunk of tickdata - works almost fine, then it gets slower and slower until it stops working.

Spotware
16 Jan 2023, 09:13

Hi GridSurfer,

Please share the complete cBot code alongside with the necessary cBot parameters that will allow us to reproduce the problem so that we can investigate.

Best regards,

cTrader Team

GridSurfer
16 Jan 2023, 09:38

Hi Support,

thanks, here is the code. No parameters.

-------------------------------------------------------------------------------------------------------------

using System;
using System.Linq;
using System.Collections.Generic;
using System.Diagnostics;
using cAlgo.API;
using cAlgo.API.Indicators;
using cAlgo.API.Internals;
using cAlgo.Indicators;
using System.IO;


namespace cAlgo.Robots
{
    [Robot(TimeZone = TimeZones.GMTStandardTime, AccessRights = AccessRights.FullAccess)]
    public class TickdataExportMultiSymbol : Robot
    {
        private string _dataDirectory = "O:\\_TradingData\\_Data_ICMarkets\\_Source";
        private string[] _symbols = {  "EURAUD", "EURGBP", "EURJPY", "EURUSD", "EURCHF", "EURCAD", "EURNZD", 
                                    "GBPUSD", "GBPJPY", "GBPCHF", "GBPCAD", "GBPAUD","GBPNZD"};

        protected override void OnStart()
        {
            // System.Diagnostics.Debugger.Launch();
        
            foreach (string ticker in _symbols)
            {
                // Read last Tick in file (Load Ticks after that)
                //
                DateTime firstDt = GetLastTickInFiles(ticker);

                // For the initial files use one month more back
                //
                DateTime firstMaxDt = new DateTime(DateTime.Now.Year, DateTime.Now.Month, 1);
                if (firstDt.Year < DateTime.Now.Year - 1)
                    firstDt = firstMaxDt.AddMonths(-1);

                // Load Ticks
                //
                API.Ticks symbolTicks;
                symbolTicks = MarketData.GetTicks(ticker);
                while (symbolTicks[0].Time.CompareTo(firstDt.AddHours(-4)) > 0)
                {
                    symbolTicks.LoadMoreHistory();
                }

                List<TickRec> tickList = new List<TickRec>();
                DateTime lastDt = new DateTime(1970, 1, 1);

                for (int idx = 0; idx < symbolTicks.Count; idx++)
                {
                    // Adjust by 2 hours - forex trade week starts 00:00 (needs addl. 1 hour [e.g. 08.03.2020 - 29.03.2020])
                    //
                    DateTime dt = symbolTicks[idx].Time.AddHours(2);

                    // If by any chance there are two ticks with same time, add 1ms to the second tick
                    //
                    while ((dt - lastDt).TotalMilliseconds <= 0)
                        dt = dt.AddMilliseconds(1);
                    lastDt = dt;

                    if (dt.CompareTo(firstDt) > 0)
                    {
                        TickRec newTick = new TickRec();
                        newTick.Bid = symbolTicks[idx].Bid;
                        newTick.Ask = symbolTicks[idx].Ask;
                        newTick.DateTime = dt;
                        tickList.Add(newTick);
                    }
                }

                SaveTicks(ticker, tickList);
            }

            Notifications.PlaySound("C:/Windows/Media/Ring05.wav");
            
        }

        protected override void OnTick()
        {
        }

        protected override void OnStop()
        {
        }

        private DateTime GetLastTickInFiles(string Ticker)
        {
            DateTime lastTickDt = new DateTime(1970, 1, 1);
            var dirInfo = new DirectoryInfo(_dataDirectory);
            FileInfo[] files = dirInfo.GetFiles(Ticker + "_*.csv");            
            if (files.Length > 0)
            {
                // Locate the last tick file of this symbol
                // Read to the end and return DateTime of last Tick
                //
                FileInfo fInfo = files.OrderByDescending(x => x.Name).First();
                string line = "", lastLine = "";
                DateTime readDt = new DateTime(1970,1,1);
                using (StreamReader file = new StreamReader(fInfo.FullName))
                {
                    while ((line = file.ReadLine()) != null)
                        if (line.Length > 10)
                            lastLine = line;
                }
                string[] parts = lastLine.Split('\t');
                DateTime.TryParse(parts[0], out lastTickDt);
            }
            return lastTickDt;
        }

        private void SaveTicks(string Ticker, List<TickRec>TickList)
        {
            if (TickList.Count == 0)
                return;

            int loopMonth = TickList[0].DateTime.Month;
            int loopYear = TickList[0].DateTime.Year;
            bool finished = false;
            while (!finished)
            {
                string fileName = _dataDirectory + "\\" + Ticker + "_" + loopYear + "-" + loopMonth.ToString("00") + ".csv";

                StreamWriter sw = new StreamWriter(fileName, true);
                foreach (TickRec tick in TickList.Where(x => x.DateTime.Year == loopYear && x.DateTime.Month == loopMonth))
                {
                    sw.WriteLine(tick.DateTime.ToString("dd.MM.yyyy HH:mm:ss.fff") + "\t" + tick.Bid + "\t" + tick.Ask);
                }
                sw.Close();
                loopMonth++;
                if (loopMonth == 13)
                {
                    loopMonth = 1;
                    loopYear++;
                }
                finished = (TickList[TickList.Count - 1].DateTime.CompareTo(new DateTime(loopYear, loopMonth, 1)) < 0);
            }
        }
    }

    public class TickRec
    {
        public DateTime DateTime;
        public double Bid = 0;
        public double Ask = 0;
    }
}

Spotware
16 Jan 2023, 13:07

Hi GridSurfer,

Thank you, we were able to reproduce the issue and we will fix it in one of the upcoming updates.

Best regards,

cTrader Team