Export bar data to DSV file.

Description:

• Execute in backtesting mode, to save historical bar data to a DSV file.
• Various features of the DSV file can be customized, including delimiter, date and time formatting, price formatting, etc.
• Market sessions can also be included, with their own “sub-formatting”, so that they can be parsed independently if needed. 

Parameters:

• DSV File
  • File Path
    • Path where the DSV file will be saved.
    • The path should end with a back or forward slash, as the current version of the code does not add a back or forward slash automatically.
      • e.g. x:/data/test/. When a file name, for example, data.csv is appended to the path, the full path will be x:/data/test/data.csv.
      • If you specify a path like x:/data/test for example, then when a file name is appended you will get an invalid full path, e.g. x:/data/testdata.csv. 
  • File Name
    • The file name to save bar data to. 
    • It is up to the user to align the file name extension with the delimiter.
      • For example, if one sets the DSV delimiter to a "," character, then the most appropriate file extension might be ".csv".
      • If one sets the DSV delimiter to a "\t" character, then the most appropriate file extension might be ".tsv".
      • …etc.
  • Delimiter
    • The DSV delimiter you want to use to separate values within the DSV file. e.g. ",", "\t", ";", etc. 
    • The delimiter does not need to be a single character. You can use a string of any length.
      • For example, if you want a space after the delimiter, you could define the delimiter as ", ".
  • Date Format
    • A C# format string that defines the date format for the date field in the DSV file.
    • For example, ISO 8601 date standard would be defined as "yyyy-MM-dd". 
  • Time Format
    • A C# format string that defines the time format for the time field in the DSV file. 
    • For example, ISO 8601 time standard including milliseconds would be defined as "HH:mm:ss.fff".
  • Price Format
    • Format the floating point price value. 
    • For example, ".00000" for most currency pairs. Or ".00" for gold and other commodities, etc. 
• Market Sessions
  • Market Sessions Enabled
    • Enabled inclusion of the list of active market sessions for the bar. 
    • All market sessions are treated as a single value.
    • The list of market sessions may be formatted to specify its own delimiter and optional brackets.  
  • Delimiter
    • The delimiter used to separate the market sessions.
    • Because the market sessions are treated as a single value in the DSV file, the market sessions delimiter should be different from the DSV file delimiter. If the DSV file delimiter and the market sessions delimiters are the same, then the column structure of the DSV file will not align with the DSV file header row. 
    • For example, if one sets the DSV file delimiter to ",", then the market sessions delimiter can be anything “except” ",". So you could chose say "," for the DSV file delimiter, and then say ";" for the market session delimiter.
      • 2023-12-10,22:50:00,1.07614,1.07616,1.07608,1.07615,25,[Wellington;Tokyo;Sydney] 
  • Brackets
    • The market session data may be optionally enclosed with brackets.
    • The bracket options are, None, Round, Square, Curly, Angle. 
      • In the case of “None”, i.e. no brackets, a row of data might look like this, 2023-12-10,22:50:00,1.07614,1.07616,1.07608,1.07615,25,Wellington;Tokyo;Sydney.
      • The no-bracket example above highlights the value of ensuring that the DSV file delimiter and the market session delimiter are different, "," and ";" in the example above. 
• Date & Time Range
  • Date and Time Range Enabled
    • Enable user-defined date and time filtering of bar data. 
    • When using the cTrader Automate back testing interface, you are only able to select a start and end date, but not the start and end time.
    • With date and time range enabled, the “from” (start) and “to” (end) times may also be specified. 
    • It is important to ensure that when enabling date and time range, that the date range falls within the boundaries of the currently selected back testing start and end date settings. 
  • From: Date
    • The date from which to start including bars in the DSV file. 
    • The from-date is inclusive. For example, starting with a from-date of “2024-01-08”, will include data from “2024-01-08” onwards.
  • From: Time
    • The time from which to start including bars in the DSV file.
    • The from-time is inclusive. For example, starting with a from-time of “09:00:00”, will include data from 9am onwards. 
  • To: Date
    • The date from which to stop including bars in the DSV file. 
    • The to-date is inclusive. For example, ending with a to-date of “2024-01-09”, will include data up to and including “2024-01-09”.
  • To: Time
    • The time from which to stop including bars in the DSV file.
    • The to-time is inclusive. For example, ending with a to-time of “09:50:00”, will include data up to and including from 9:50 am. 
• Debug
  • Debugger Enabled
    • Set to Yes to attach a debugger for real-time debugging in Visual Studio. 

Example Usage:

• Backtesting configuration.
  • Note the date and time range vs. the backtesting environment date range. 

• DSV (CSV) File Generated

date,time,open,high,low,close,tick_volume,active_sessions
2023-12-08,20:00:00,1.07585,1.07614,1.07570,1.07606,902,[NewYork]
2023-12-08,20:10:00,1.07607,1.07607,1.07582,1.07607,822,[NewYork]
2023-12-08,20:20:00,1.07606,1.07653,1.07606,1.07640,852,[NewYork]
2023-12-08,20:30:00,1.07636,1.07656,1.07629,1.07647,592,[NewYork]
2023-12-08,20:40:00,1.07646,1.07660,1.07638,1.07646,644,[NewYork]
2023-12-08,20:50:00,1.07648,1.07648,1.07628,1.07632,646,[None]
2023-12-08,21:00:00,1.07633,1.07638,1.07623,1.07626,224,[None]
2023-12-08,21:10:00,1.07627,1.07627,1.07614,1.07614,148,[None]
2023-12-08,21:20:00,1.07615,1.07632,1.07605,1.07630,292,[None]
2023-12-08,21:30:00,1.07631,1.07633,1.07616,1.07621,288,[None]
2023-12-08,21:40:00,1.07622,1.07633,1.07622,1.07627,168,[None]
2023-12-08,21:50:00,1.07628,1.07635,1.07574,1.07629,542,[Wellington;Sydney]
2023-12-10,22:00:00,1.07555,1.07589,1.07555,1.07579,22,[Wellington;Sydney]
2023-12-10,22:10:00,1.07582,1.07595,1.07563,1.07574,76,[Wellington;Sydney]
2023-12-10,22:20:00,1.07573,1.07580,1.07571,1.07578,30,[Wellington;Sydney]
2023-12-10,22:30:00,1.07579,1.07613,1.07572,1.07611,359,[Wellington;Sydney]
2023-12-10,22:40:00,1.07611,1.07616,1.07608,1.07615,58,[Wellington;Sydney]
2023-12-10,22:50:00,1.07614,1.07616,1.07608,1.07615,25,[Wellington;Tokyo;Sydney]
2023-12-10,23:00:00,1.07617,1.07674,1.07617,1.07658,376,[Wellington;Tokyo;Sydney]
2023-12-10,23:10:00,1.07659,1.07664,1.07643,1.07646,214,[Wellington;Tokyo;Sydney]
2023-12-10,23:20:00,1.07649,1.07665,1.07639,1.07645,302,[Wellington;Tokyo;Sydney]
2023-12-10,23:30:00,1.07644,1.07654,1.07620,1.07639,540,[Wellington;Tokyo;Sydney]
2023-12-10,23:40:00,1.07635,1.07646,1.07629,1.07645,264,[Wellington;Tokyo;Sydney]

Source Code:

//------------------------------------------------------------------------------------------------------------------------------------------------------------
// Name:    Export Bar Data
// Version: 1.0
// Author:  Rohin Gosling
//
// Description:
//
// - Use this cBot in back testing mode to export historical bar data to DSV file.
//
// - You can either use the normal back testing start and stop dates in the cTrader Automate back testing interface. Or, you can enable and specify user
//   defined from and to dates and times.
//
// - If you use the user defined from and to dates and times, make sure that your user defined from and to dates are with in the range of dates that you
//   setup in the cTrader Automate back tester.
//
// cBot Parameters:
//
//   -------------------- -------------------------- ----------------------------------------------------------------------------------------------------
//   Group                Paramter                   Description
//   -------------------- -------------------------- ----------------------------------------------------------------------------------------------------
//
//   DSV File             File Path                  Path where the DSV file will be saved to. Note: The path should end with a back or forward slash.
//   DSV File             File Name                  The file name of the DSV file.
//   DSV File             Delimiter                  The delimiter for the DSV file. E.g. "," for CSV (Default), "\t" for TSV, ";", "XYZ", etc. 
//   DSV File             Date Format                Format of the date values. E.g. "yyyy-MM-dd" for ISO 8601.
//   DSV File             Time Format                Format of the time values. 
//   DSV File             Price Format               Price format. E.g. ".00000" for most currency pairs, ".00" for XAUUSD and other commodities, etc. 
//
//   Market Sessions      Market Sessions Enabled    Enable market sessions to be added to the DSV file. They will be included in a single column. 
//   Market Sessions      Delimiter                  Market session delimiter. In general this should be different to the DSV file delimiter. 
//   Market Sessions      Brackets                   Bracket style used to enclose the session names. 
//
//   Date & Time Range    Date & Time Range Enabled  Enable date and time range. Normal cTrader Automate back testing only allows for date, but not time. 
//   Date & Time Range    From: Date                 Date from which to start including bars in the DSV file. Should be >= backtesting start date. 
//   Date & Time Range    From: Time                 Time from which to start including bars in the DSV file.
//   Date & Time Range    To: Date                   Date at which to stop including bars in the DSV file. Should be <= backtesting end date. 
//   Date & Time Range    To: Time                   Time at which to stop including bars in the DSV file.
//
//   Debug                Debugger Enabled           Attach a debugger for debuging code in Visual Studio. 
//
//   -------------------- -------------------------- ----------------------------------------------------------------------------------------------------
//
//------------------------------------------------------------------------------------------------------------------------------------------------------------

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Globalization;
using cAlgo.API;
using cAlgo.API.Collections;
using cAlgo.API.Indicators;
using cAlgo.API.Internals;

namespace cAlgo.Robots
{
    public enum BracketType
    {
        None,
        Round,
        Square,
        Curly,
        Angle
    }

    [Robot(AccessRights = AccessRights.FullAccess)]
    public class ExportBarData : Robot
    {
        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Constants. 
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        // Parameter groups. 

        private const string PARAMETER_GROUP_DSV_FILE            = "DSV File";
        private const string PARAMETER_GROUP_MARKET_SESSIONS     = "Market Sessions";
        private const string PARAMETER_GROUP_DATE_AND_TIME_RANGE = "Date & Time Range";
        private const string PARAMETER_GROUP_DEBUG               = "Debug";

        // Default user parameter values. 

        private const string      DEFAULT_FILE_PATH                   = "F:/data/financial_data/forex/test/";
        private const string      DEFAULT_FILE_NAME                   = "forex_data.csv";
        private const string      DEFAULT_DELIMITER                   = ",";
        private const bool        DEFAULT_MARKET_SESSION_ENABLED      = true;
        private const string      DEFAULT_MARKET_SESSION_DELIMITER    = ";";
        private const BracketType DEFAULT_MARKET_SESSION_BRACKETS     = BracketType.Square;
        private const bool        DEFAULT_DATE_AND_TIME_RANGE_ENABLED = true;
        private const string      DEFAULT_FORMAT_DATE                 = "yyyy-MM-dd";
        private const string      DEFAULT_FORMAT_TIME                 = "HH:mm:ss";
        private const string      DEFAULT_FORMAT_PRICE                = ".00000";
        private const string      DEFAULT_FROM_DATE                   = "2023-12-08";
        private const string      DEFAULT_FROM_TIME                   = "20:00:00";
        private const string      DEFAULT_TO_DATE                     = "2023-12-10";
        private const string      DEFAULT_TO_TIME                     = "23:40:00";
        private const bool        DEFAULT_DEBUGGER_ENABLED            = false;

        // Log messaging 

        private const string LOG_MESSAGE_APPLICATION = "[APPLICATION]";
        private const string LOG_MESSAGE_DEBUG       = "[DEBUG]";

        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Public user paramter properties. 
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        // DSV File Parameters.

        [Parameter ( "File Path", Group = PARAMETER_GROUP_DSV_FILE, DefaultValue = DEFAULT_FILE_PATH )]
        public string FilePath { get; set; }
                
        [Parameter ( "File Name", Group = PARAMETER_GROUP_DSV_FILE, DefaultValue = DEFAULT_FILE_NAME )]
        public string FileName { get; set; }
        
        [Parameter ( "Delimiter", Group = PARAMETER_GROUP_DSV_FILE, DefaultValue = DEFAULT_DELIMITER )]
        public string Delimiter
        {
            get { return this.delimiter; }

            // For cases where the user wants to generate tab seperated data, we have to convert the property's verbatum representation of "\t" to a regular
            // string representation, so that escape sequences are interpreted by the compiler. 

            set { this.delimiter = value.Replace ( "\\t", "\t" ); }
        }    

        [Parameter ( "Date Format", Group = PARAMETER_GROUP_DSV_FILE, DefaultValue = DEFAULT_FORMAT_DATE )]
        public string FormatDate { get; set; }
        
        [Parameter ( "Time Format", Group = PARAMETER_GROUP_DSV_FILE, DefaultValue = DEFAULT_FORMAT_TIME )]
        public string FormatTime { get; set; }

        [Parameter ( "Price Format", Group = PARAMETER_GROUP_DSV_FILE, DefaultValue = DEFAULT_FORMAT_PRICE )]
        public string FormatPrice { get; set; }

        // Market Sessions.

        [Parameter ( "Market Sessions Enabled", Group = PARAMETER_GROUP_MARKET_SESSIONS, DefaultValue = DEFAULT_MARKET_SESSION_ENABLED )]
        public bool MarketSessionsEnabled { get; set; }

        [Parameter ( "Delimiter", Group = PARAMETER_GROUP_MARKET_SESSIONS, DefaultValue = DEFAULT_MARKET_SESSION_DELIMITER )]
        public string MarketSessionsDelimiter { get; set; }

        [Parameter ( "Brackets", Group = PARAMETER_GROUP_MARKET_SESSIONS, DefaultValue = DEFAULT_MARKET_SESSION_BRACKETS )]
        public BracketType MarketSessionsBrackets { get; set; }

        // Date and Time Parameters.

        [Parameter ( "Date & Time Range Enabled", Group = PARAMETER_GROUP_DATE_AND_TIME_RANGE, DefaultValue = DEFAULT_DATE_AND_TIME_RANGE_ENABLED )]
        public bool DateTimeRangeEnabled { get; set; }
        
        [Parameter ( "From: Date", Group = PARAMETER_GROUP_DATE_AND_TIME_RANGE, DefaultValue = DEFAULT_FROM_DATE )]
        public string FromDate { get; set; }
        
        [Parameter ( "From: Time", Group = PARAMETER_GROUP_DATE_AND_TIME_RANGE, DefaultValue = DEFAULT_FROM_TIME )]
        public string FromTime { get; set; }

        [Parameter ( "To: Date", Group = PARAMETER_GROUP_DATE_AND_TIME_RANGE, DefaultValue = DEFAULT_TO_DATE )]
        public string ToDate { get; set; }
        
        [Parameter ( "To: Time", Group = PARAMETER_GROUP_DATE_AND_TIME_RANGE, DefaultValue = DEFAULT_TO_TIME )]
        public string ToTime { get; set; }

        // Debug paramters. 

        [Parameter ( "Debugger Enabled", Group = PARAMETER_GROUP_DEBUG, DefaultValue = DEFAULT_DEBUGGER_ENABLED )]
        public bool DebuggerEnabled { get; set; }

        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Private internal properties. 
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        // Internal date and time properties. These are used to allow filtering by time as well as date during backtesting. 

        public DateTime FromDateTime
        {
            get { return this.fromDateTime; }

            set
            {
                DateTime tempDateTime;
                DateTime.TryParseExact ( $"{FromDate} {FromTime}", $"{FormatDate} {FormatTime}", CultureInfo.InvariantCulture, DateTimeStyles.None, out tempDateTime );
                this.fromDateTime = tempDateTime;
            }
        }

        public DateTime ToDateTime
        {
            get { return this.toDateTime; }

            set
            {
                DateTime tempDateTime;
                DateTime.TryParseExact ( $"{ToDate} {ToTime}", $"{FormatDate} {FormatTime}", CultureInfo.InvariantCulture, DateTimeStyles.None, out tempDateTime );
                this.toDateTime = tempDateTime;
            }
        }

        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Local class member variables.
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        private DateTime fromDateTime;
        private DateTime toDateTime;
        private long     dsvDataRowCount;
        private string   delimiter;
        
        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Event: OnStart
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        protected override void OnStart ()
        {
            // Attach a debugger. 

            if ( DebuggerEnabled )
            {
                bool debugger_launched = System.Diagnostics.Debugger.Launch ();
                if ( debugger_launched )
                {
                    Print ( $"{LOG_MESSAGE_DEBUG} Debugger launched." );
                }
                else
                {
                    Print ( $"{LOG_MESSAGE_DEBUG} Debugger failed to launched." );
                }
            }

            // Initialise date and time range.
            // Note:
            // - Calling `DateTime.Now` does not initialise the date and time to the current date and time.
            // - Rather, the value returned by `DateTime.Now` is ignored in this case, and is simply used to trigger the mutator code for `FromDateTime` and `ToDateTime`.
            // - You could inface assign eany valie `DateTime` value here, and it will be ingnored, simply resulting in the mutator code being triggered. 

            FromDateTime = DateTime.Now;
            ToDateTime   = DateTime.Now;

            // Initialise DSV file header row. 

            string dsvHeaderRow = "";
            dsvHeaderRow += $"date{Delimiter}";
            dsvHeaderRow += $"time{Delimiter}";
            dsvHeaderRow += $"open{Delimiter}";
            dsvHeaderRow += $"high{Delimiter}";
            dsvHeaderRow += $"low{Delimiter}";
            dsvHeaderRow += $"close{Delimiter}";
            dsvHeaderRow += $"tick_volume";            
            dsvHeaderRow += ( MarketSessionsEnabled ) ? $"{Delimiter}active_sessions\n" : "\n";

            // Write DSV header row to file. 

            File.WriteAllText ( FilePath + FileName, dsvHeaderRow );

            // Initialise DSV data row count.

            this.dsvDataRowCount = 0;

            Print ( $"{LOG_MESSAGE_APPLICATION} Application initialised." );
        }

        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Event: OnStop
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        protected override void OnStop ()
        {
            Print ( $"{LOG_MESSAGE_APPLICATION} {this.dsvDataRowCount} lines of data writen to DSV file \"{FileName}.\"" );
        }

        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Event: OnBar
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        protected override void OnBar ()
        {
            // Local constants.

            const int OPEN  = 0;    // Open bracket string index. e.g. "()"[OPEN] = "(".
            const int CLOSE = 1;    // Close bracket string index. e.g. "()"[CLOSE] = ")".

            // Initialise market sessions data brackets. 
            // - We will get a 2 character string from `GetBracketString`, e.g. "()", "[]", "{}", etc.
            // - We will use the constants `OPEN` and `CLOSE` as indiceis to access the indevidual open and close brackets, e.g. "()"[OPEN] = "(" and "()"[CLOSE] = ")"

            string   brackets       = GetBracketString ( MarketSessionsBrackets );
            string   bracket_open   = ( brackets.Length == 2 ) ? $"{brackets [ OPEN  ]}" : "";
            string   bracket_close  = ( brackets.Length == 2 ) ? $"{brackets [ CLOSE ]}" : "";

            // We will either write all bar date specified in the normal back testing from and to date.
            // Or if the user specified date and time is enabled, then we will use the user specified date and time range. 

            int      barIndex       = 1;
            DateTime dateCurrentBar = Bars.OpenTimes.Last ( barIndex );

            if ( ( ( dateCurrentBar >= FromDateTime) && ( dateCurrentBar <= this.toDateTime ) && DateTimeRangeEnabled || !DateTimeRangeEnabled ) )
            {
                // Compile DSV data row to write to file.
                
                string dsvDataRow = "";
            
                dsvDataRow += $"{dateCurrentBar.ToString ( FormatDate )}{Delimiter}";
                dsvDataRow += $"{dateCurrentBar.ToString ( FormatTime )}{Delimiter}";                
                dsvDataRow += $"{Bars.OpenPrices.Last ( barIndex ).ToString ( FormatPrice )}{Delimiter}";
                dsvDataRow += $"{Bars.HighPrices.Last ( barIndex ).ToString ( FormatPrice )}{Delimiter}";
                dsvDataRow += $"{Bars.LowPrices.Last ( barIndex ).ToString ( FormatPrice )}{Delimiter}";
                dsvDataRow += $"{Bars.ClosePrices.Last ( barIndex ).ToString ( FormatPrice )}{Delimiter}";
                dsvDataRow += $"{(long)Bars.TickVolumes.Last ( barIndex )}";                
                dsvDataRow += ( MarketSessionsEnabled ) ? $"{Delimiter}{bracket_open}{MarketSessions.ToString ().Replace ( ", ", MarketSessionsDelimiter )}{bracket_close}\n" : "\n";

                // Write DSV data row to file. TickVolumes

                File.AppendAllText ( FilePath + FileName, dsvDataRow );

                // Update DSV data row line count.

                this.dsvDataRowCount++;
            }
        }

        //----------------------------------------------------------------------------------------------------------------------------------------------------
        // Name:
        //
        // Description:
        //
        // Paramters:
        //
        // Return Value:
        //
        //----------------------------------------------------------------------------------------------------------------------------------------------------

        public string GetBracketString ( BracketType bracketType )
        {
            string brackets = "";

            switch ( bracketType )
            {
                case BracketType.None:   brackets = "";   break;
                case BracketType.Round:  brackets = "()"; break;
                case BracketType.Square: brackets = "[]"; break;
                case BracketType.Curly:  brackets = "{}"; break;
                case BracketType.Angle:  brackets = "<>"; break;                
            }

            return brackets;
        }
    }
}