Format of the tdbc backtesting cache files
Format of the tdbc backtesting cache files
18 Nov 2017, 10:48
Can you please disclose the format of the tdbc files cAlgo is using for cachicng backtesting tick data?
If this format would be disclosed, own backtesting tick data files could be generated replacing the original cAlgo files with much more time span then the original files from cAlgo. There are many free sources to get those data from like tickstory, Dukascopy, etc.
i know that this feature is voted for already since 3 years, but there are many cAlgo users like me which need this feature urgently. Beeing able to replace the tdbc files would enable a sufficient work araound until cAlgo will release the possibility to import 3rd party tick data. And btw: If this feature ever will come, please do not realize it with importing CSV files only. Iy one wants to backtest several years with tick data, the import would take an endless amout of time. Or import CSV files only once and convert it immediately into the binary tdbc format.
With the help of these ressources
//https://stackoverflow.com/questions/3052202/how-to-analyse-contents-of-binary-serialization-stream
//https://docs.microsoft.com/en-us/dotnet/standard/serialization/basic-serialization
I have found out so far, that the tdbc files (after un-zip-ing) are holding a serialized binary stream of this class type:
public class TickDataResult
{
public bool[] IsBids { get; set; }
public double[] Bids { get; set; }
public double[] Asks { get; set; }
public DateTime[] DateTimes { get; set; }
public bool Count { get; set; }
}
However, even with the knowledge of this structure, I could not manage it to deserialize a original cAlgo file (just for testing) nor serializing my own set of tick data (main goal).
And btw: can you explain, what the menaing of the boolean "IsBids" and "Count" is?
Thanks in advance
Hanky
hmunz@icloud.com
19 Nov 2017, 23:30
RE:
hmunz@icloud.com said:
Problem solved!
Here is a little test program to read in an original tdbc file, generate an own tdbc file and change some Bids just for testing. So it is now possible to generate self made history files on tick level.
Here are my mistakes/answers from previous post:
1. the class TickDataResult contains Coint as a int, not as a bool
2. The IsBids booleans indicatey if the Bid is different from the prvious Bid
Good Luck
Hanky
using System;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.IO;
using System.IO.Compression;
namespace Frontend
{
namespace CrossDomain
{
namespace TicksGenerating
{
//https://stackoverflow.com/questions/3052202/how-to-analyse-contents-of-binary-serialization-stream
//https://docs.microsoft.com/en-us/dotnet/standard/serialization/basic-serialization
[Serializable]
public class TickDataResult
{
public bool[] IsBids { get; set; }
public double[] Bids { get; set; }
public double[] Asks { get; set; }
public DateTime[] DateTimes { get; set; }
public int Count { get; set; }
}
public class Program
{
static void generate(string filename, TickDataResult obj)
{
// Delete old file, if exists
string monitorFilename = filename + ".bin";
if (File.Exists(filename)) File.Delete(filename);
if (File.Exists(filename)) File.Delete(monitorFilename);
// 1st serialize it
MemoryStream sourceStream = new MemoryStream();
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(sourceStream, obj);
sourceStream.Position = 0;
byte[] targetBa = sourceStream.ToArray();
using (FileStream targetFs = File.Create(filename)) // 2nd create output file
using (GZipStream compressor = new GZipStream(targetFs, CompressionMode.Compress)) //3rd compress it
compressor.Write(targetBa, 0, targetBa.Length);
#if true
// create file for monitoring
File.WriteAllBytes(monitorFilename, targetBa);
#endif
}
static void Main(string[] args)
{
string filename = "2017.11.12.org.tdbc";
TickDataResult readObj;
using (FileStream fs = new FileStream(filename, FileMode.Open)) // open file
using (GZipStream decompressor = new GZipStream(fs, CompressionMode.Decompress)) // decompress
{
BinaryFormatter bf = new BinaryFormatter();
readObj = (TickDataResult)bf.Deserialize(decompressor); // deserialize
}
TickDataResult writeObj = new TickDataResult();
writeObj.Bids = new double[readObj.Count];
writeObj.Asks = new double[readObj.Count];
writeObj.IsBids = new bool[readObj.Count];
writeObj.DateTimes = new DateTime[readObj.Count];
writeObj.Count = readObj.Count;
double prevBids = default(double);
for (int i = 0; i < readObj.Count; i++)
{
writeObj.Bids[i] = readObj.Bids[i];
writeObj.Asks[i] = readObj.Asks[i];
writeObj.DateTimes[i] = readObj.DateTimes[i];
writeObj.IsBids[i] = prevBids != readObj.Bids[i];
if (writeObj.IsBids[i] != readObj.IsBids[i]) Console.WriteLine("IsBids Error");
prevBids = readObj.Bids[i];
}
writeObj.Bids[0] = 1.11111;
writeObj.Bids[1] = 1.15555;
writeObj.Bids[2] = 1.16666;
string newFile = filename.Replace(".org", "");
generate(newFile, writeObj);
}
}
}
}
}
@hmunz@icloud.com