Backtesting with data from CSV file

Hyperloop
25 Nov 2013, 23:03

Do you have an ETA on tick data testing?

oktrader
27 Nov 2013, 15:33

RE:

Will this allow the use of more accurate historical data (tick-data)?

Spotware
27 Nov 2013, 16:50

RE:

Hyperloop said:

Do you have an ETA on tick data testing?

We cannot provide an ETA but it is planned.

kyro747
30 Nov 2013, 09:41

RE: RE:

Will it be possible to load 1m data in blocks rather than have to run it off one file as it is unlikely you have all the data in 1 file? E.g. I have it in 1 year files.

Spotware
02 Dec 2013, 16:19

It is not possible to load more than one file. You can merge the files manually into one.

AwarenessForex
07 Dec 2013, 23:58

RE: RE:

Spotware said:

Hyperloop said:

Do you have an ETA on tick data testing?

We cannot provide an ETA but it is planned.

can you go into more details as to why tick data backtesting wasn't considered from the start of cAlgo development?  I find it hard to believe that a modern trading package post-metatrader 4 is limited to 1 minute data bars for testing strategies.

Spotware
10 Dec 2013, 09:42

RE: RE: RE:

AwarenessForex said:

can you go into more details as to why tick data backtesting wasn't considered from the start of cAlgo development?  I find it hard to believe that a modern trading package post-metatrader 4 is limited to 1 minute data bars for testing strategies.

You are right, tick-based backtesting can be very useful. The reason is that we have of lot of things to do. Tick-based backtesting will be implemented release of tick-charts.

lec0456
13 Dec 2013, 00:47

Can you download a csv from the server?

Spotware
13 Dec 2013, 14:50

RE:

lec0456 said:

Can you download a csv from the server?

No, this is not possible.

lec0456
13 Dec 2013, 16:39

I've always wanted to do back testing without having to be connected to the server.  That way I don't even have to be connected to the internet.   But I don't have my own data.  Any suggestions?

cogs
23 Dec 2013, 07:48 ( Updated at: 21 Dec 2023, 09:20 )

I have this:

but need 2013.07.19,00:00, etc. as stated in previous post.

The problem lies with the 'period' or decimal place when using excel for csv files.

Any ideas?

daemon
27 Dec 2013, 14:51 ( Updated at: 21 Dec 2023, 09:20 )

You can right click -> Format Cells (highlight the column) the choose the format with the period.

jeex
28 Dec 2013, 01:21

Suggestion

@ lec0456

You can export data from MT4 quite easy. But you have to generate them first with the Hostiry function. Then export to CSV and you have all the data you can dream of in all possible timeframes and pairs.

bendyarm
28 Dec 2013, 19:56

RE: Suggestion

jeex said:

@ lec0456

You can export data from MT4 quite easy. But you have to generate them first with the Hostiry function. Then export to CSV and you have all the data you can dream of in all possible timeframes and pairs.

You can also download tick data from http://www.dukascopy.com/swiss/english/marketwatch/historical/

bendyarm
29 Dec 2013, 02:40

RE:

Spotware said:

A new feature will be available in the next release of cAlgo: Backtesting with data from a CSV file.

x

Any update on a release date yet?

Thanks

Spotware
30 Dec 2013, 11:55 ( Updated at: 23 Jan 2024, 13:16 )

RE: RE:

bendyarm said:

Spotware said:

A new feature will be available in the next release of cAlgo: Backtesting with data from a CSV file.

x

Any update on a release date yet?

Thanks

Backtesting with data from a CSV file has already been released.

[/forum/whats-new/1931]

cogs
31 Dec 2013, 05:08 ( Updated at: 21 Dec 2023, 09:20 )

I get this error, any suggestions?

daemon
31 Dec 2013, 09:24 ( Updated at: 21 Dec 2023, 09:20 )

RE:

cogs said:

I get this error, any suggestions?

Remove the first column and probably the first row.

The example in the first page is:

2003.06.18,16:01,1.11423,1.11428,1.11332,1.11374,19
2003.06.18,16:02,1.11364,1.11436,1.11361,1.11405,7
2003.06.18,16:03,1.11402,1.11455,1.11400,1.11440,5
2003.06.18,16:04,1.11446,1.11461,1.11401,1.11447,14

cogs
31 Dec 2013, 13:27

I removed the first column and worked perfectly.

Thanks.

cogs
04 Jan 2014, 05:13

Anyone know why repeated back testing without changing any settings still yields different results each time?

Everything should be constant, the only variation (which never effected MT4 backtesting) is CPU usage.

I simply don't know how one could ever get consistent positive results with moving goal posts???

Any ideas anyone? Could it be my robot coding?

Researcher
04 Jan 2014, 18:59

Which Spread settings do you use for backtesting? It must be "Fixed Spread" the same results.

cogs
05 Jan 2014, 00:40

I use fixed spread.

I have removed all variables I can as to obtain a repeatable test.

cogs
05 Jan 2014, 00:52

RE:

cogs said:

I use fixed spread.

I have removed all variables I can as to obtain a repeatable test.

It looks like it's the robot coding, as testing using some of the free robots here yields consistent repeatable results.

madopter
16 Mar 2014, 03:01

I would like to make the move from MT4 to cTrader/cAlgo but tick-testing is a must-have. It's a deal-breaker for me as long as it's not available -- is there any estimate regarding when tick-testing would be implemented in cAlgo?

Spotware
17 Mar 2014, 10:22

RE:

madopter said:

I would like to make the move from MT4 to cTrader/cAlgo but tick-testing is a must-have. It's a deal-breaker for me as long as it's not available -- is there any estimate regarding when tick-testing would be implemented in cAlgo?

This feature is in our short term roadmap. We are going to release it in couple of months.

Spotware
27 Aug 2014, 17:03 ( Updated at: 21 Dec 2023, 09:20 )

Tick data backtesting has been released:

chanet
18 Sep 2014, 12:15 ( Updated at: 21 Dec 2023, 09:20 )

RE:

Spotware said:

Tick data backtesting has been released:

Hello, From which date I can use this new feature tick data backtesting?

Spotware
19 Sep 2014, 09:25 ( Updated at: 21 Dec 2023, 09:20 )

RE: RE:

chanet said:

Spotware said:

Tick data backtesting has been released:

Hello, From which date I can use this new feature tick data backtesting?

The available data range depends on specific Symbol and Broker. You can check the available data range by using data range slider

AlexanderRC
19 Sep 2014, 11:07

RE: RE: RE:

Very encouraging screenshot. Any predictions when optimizer will be released?

Spotware
19 Sep 2014, 11:24

RE: RE: RE: RE:

AlexanderRC said:

Very encouraging screenshot. Any predictions when optimizer will be released?

Optimization is currently in beta stage. You can check it in Spotware cAlgo. We plan to release Optimization for all our brokers in 2 weeks.

emeeder
06 Oct 2014, 22:20

Any idea if backtesting multiple symbols and timeframes will be available soon?

Are there any other options for me available anywhere where i can backtest multiple currency/symbol cbots?

Spotware
07 Oct 2014, 10:01

RE:

emeeder said:

Any idea if backtesting multiple symbols and timeframes will be available soon?

Are there any other options for me available anywhere where i can backtest multiple currency/symbol cbots?

 

 

multi-timeframe backtesting is already available. We cannot provide an ETA but multi-symbol backtesting is planned.

asdk
28 Oct 2014, 15:34 ( Updated at: 21 Dec 2023, 09:20 )

RE: RE: RE:

Spotware said:

chanet said:

Spotware said:

Tick data backtesting has been released:

Hello, From which date I can use this new feature tick data backtesting?

The available data range depends on specific Symbol and Broker. You can check the available data range by using data range slider

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

and it also seems like users need to be connected to the internet to run a backtest, why?

Spotware
29 Oct 2014, 09:30

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

AlexanderRC
29 Oct 2014, 15:45

RE:

Spotware said:

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

For every date there is a separate file in folder named after the symbol. The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

Please add the ability to use user specified tick data from a CSV file at the same time.

NonciAzzecco
26 Jan 2015, 17:20 ( Updated at: 21 Dec 2023, 09:20 )

RE: Formatting proper CSV File for Backtesting and optimisation

Hi

Using Tickhistory Lite, and triking a little the output format I can easily export data for backtesting and optimisation. This is a short examples of data for EURUSD. Unfortunately It does not work The parse did not like it. The only thing I can see is that volume is not an integer but a float number,

2014.12.24,00:01,1.21747,1.21772,1.21747,1.21768,46.9399999380112
2014.12.24,00:02,1.21768,1.21768,1.21768,1.21768,11.2000000476837
2014.12.24,00:03,1.21768,1.21774,1.21768,1.21774,21.9799998998642
2014.12.24,00:04,1.21775,1.21776,1.21774,1.21776,39.1200000047684
2014.12.24,00:05,1.21776,1.21777,1.21776,1.21777,8.96000003814697
2014.12.24,00:06,1.21777,1.21778,1.21775,1.21775,10.2300000190735

Any Idea how to fix it?

Thank you

Spotware said:

A new feature will be available in the next release of cAlgo: Backtesting with data from a CSV file.

Before backtesting, you will have the option to choose data source - Either one minute trend bars from a broker server or one minute trend bars from a CSV file on your local disk. If you choose to use a CSV file you will need to specify its location, after which cAlgo will analyze the file and adjust minimum and maximum available dates for backtesting.

CSV file format

In order to use trend bars from the file you need to have comma-separated CSV file with following columns:

• Date
• Time
• Open Price
• High Price
• Low Price
• Close Price
• Volume

Example:

2003.06.18,16:01,1.11423,1.11428,1.11332,1.11374,19
2003.06.18,16:02,1.11364,1.11436,1.11361,1.11405,7
2003.06.18,16:03,1.11402,1.11455,1.11400,1.11440,5
2003.06.18,16:04,1.11446,1.11461,1.11401,1.11447,14

SwapBridgeCapital
01 Feb 2015, 15:22

tick

 The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

Will the tick data be available for multipurpose in the future ( without  encryption). 

AlexanderRC
02 Feb 2015, 18:46

RE: tick

RedRhinoFX said:

 The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

Will the tick data be available for multipurpose in the future ( without  encryption). 

As has been suggested elsewhere on this forum you can write a simple cBot which would log the ticks in any format you like. Maybe even someone has already written it.

Nobody
14 Jan 2016, 18:53

RE: RE: Formatting proper CSV File for Backtesting and optimisation

This post was removed by moderator because it duplicates another post. /forum/calgo-support/7782

aharonzbaida
14 Jan 2016, 23:16

RE: RE:

AlexanderRC said:

Spotware said:

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

For every date there is a separate file in folder named after the symbol. The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

Please add the ability to use user specified tick data from a CSV file at the same time.

Hi Alexander,

Spotware, and the community, does anyone have news about importing tick data into back tester? The files in the cache directory are not human readable format with .tdbc extension. Any suggestion on converting tick data to this format, or other method available to import historical tick data?

- Roni

Spotware
15 Jan 2016, 13:38

Dear aharonzbaida,

It's still not possible to backtest using custom tick data. We cannot provide you with an ETA of this feature. Altering cTrader/cAlgo files is prohibited as it is a violation of the EULA.

aharonzbaida
15 Jan 2016, 14:15

RE:

Dinger
24 Jan 2016, 10:32

RE: RE: RE:

Tcl DataBase Connectivity (TDBC)

good luck figuring out how to use that.

Also if you use wireshark you can see that calgo is downloading tick data from their website, but that tick data also happens to be in a weird format.

Example:

http://spotware-ticks.s3.amazonaws.com/cs/pepperstone-live/1/eurjpy/b/15/06/11.csv

If their data went far enough back it may actually be worth the time deciphering.

aharonzbaida said:

AlexanderRC said:

Spotware said:

Is tick data being downloaded from broker server to user's computer? if yes, where can i find them? if no, would like to know how reliable is tick data backtest in calgo. afaik, in mt4, tick data has to be downloaded locally so not sure if calgo does the same?

cAlgo automatically downloads tick data from Spotware's servers. Tick data is broker specific. Tick data is 100% reliable because it contains every tick. You can find tick data on your machine in the following directory:

C:\Users\%UserName%\AppData\Roaming\%BrokerName%-cAlgo\BacktestingCache\Ticks

For every date there is a separate file in folder named after the symbol. The files are in a proprietary format with scrambling (reversible encryption). The only value of looking at them is to check whether the specific date and symbol has been cached or not.

and it also seems like users need to be connected to the internet to run a backtest, why?

We plan to remove this restriction in the future. Thank you for your feedback.

Please add the ability to use user specified tick data from a CSV file at the same time.

Hi Alexander,

Spotware, and the community, does anyone have news about importing tick data into back tester? The files in the cache directory are not human readable format with .tdbc extension. Any suggestion on converting tick data to this format, or other method available to import historical tick data?

- Roni

aharonzbaida
24 Jan 2016, 11:42

RE: RE: RE: RE:

Dinger
24 Jan 2016, 21:08

RE: RE: RE: RE: RE:

This post was removed by moderator.

ThirtysixD
30 Jan 2016, 03:01

After loading my csv file when trying to start to backtest I get an error

Cannot read CSV file - IC Markets cAlgo
'Object reference not set to an instance of an object

Here is my csv file format
Date,Time,Open,High,Low,Close,Volume
2001.01.02,23:09:00,268.8,268.8,268.8,268.8,4
2001.01.02,23:12:00,268.8,268.8,268.8,268.8,4
2001.01.02,23:16:00,268.9,268.9,268.9,268.9,4
2001.01.02,23:38:00,268.8,268.8,268.8,268.8,4

Any ideas?

 

RhettG
24 Jun 2016, 04:14 ( Updated at: 21 Dec 2023, 09:20 )

Backtesting with CSV data

Hi

I've loaded up data for 4 years on the DAX from a CSV file but there's no option to start the testing or to select dates according to the data I have. What do I do next?

davidp13
27 Jul 2016, 08:12

I have also selected the CSV file, but have not ability to choose the dates same as the last note from back in 2014. Why is this so? Am I doing something wrong?

davidp13
05 Aug 2016, 07:00

RE:

Admin pls respond. When I try to run Opt over the csv data it does not work. Gives an error. WHy?

davidp13 said:

I have also selected the CSV file, but have not ability to choose the dates same as the last note from back in 2014. Why is this so? Am I doing something wrong?

jaredthirsk
11 Dec 2016, 01:43

One tip: make sure the file is not opened in another app (or in both backtesting/optimization tabs?)  Try making a copy of the file.

Harshenboldd
27 May 2020, 20:11

1M CSV files not behaving like the one from server.

I just downloaded 3 years of 1minute .CSV files for GBPJPY. 

I can successfully import the data to cAlgo, but when I backtest in visual, each bars (I always backtest on 4H timeframes.) appears right away. There is no fluctuation in the candle, it is all printed at once. In contrary to when I backtest with server data.

Is there a way to fix this? because my algo can't detect the set ups when the bars appear all at once. 

Thanks!!

PanagiotisCharalampous
28 May 2020, 08:08

Hi Harshenboldd,

Can you share your csv file so that we can reproduce the issue?

Best Regards,

Panagiotis 

Join us on Telegram

duongphuongtrinh92
09 Jan 2022, 14:21

Can I backtest strategy with daily data?

Hi Spotware,

I would like to backtest a swing trading strategy using Daily Close Price. And the problem is I do not have 1-minute data csv file, I only have daily data. Can I backtest the strategy using this daily csv file?

amusleh
10 Jan 2022, 08:36

RE: Can I backtest strategy with daily data?

duongphuongtrinh92 said:

Hi Spotware,

I would like to backtest a swing trading strategy using Daily Close Price. And the problem is I do not have 1-minute data csv file, I only have daily data. Can I backtest the strategy using this daily csv file?

No, you can't.

You can only import M1 bars data or use the server data.

ctid2514471
23 Oct 2022, 15:52

It's a bit bizarre that this feature has never gone beyond M1 bars after all these years...

It seems crazy to pull tick data from the server every time, in particular.

This feels like a change that would take about 15 minutes of software development...

What am I missing?

akirilarry
03 Jan 2023, 04:25

How do i see the quality of my brokers historical data after doing a backtest.

How do i see the quality of my brokers historical data after doing a backtest, just like in mt5/mt4

Hatting
19 Jan 2023, 21:08

Allow loading .csv data from different timeframes

Since the optimization feature was released, you can only load data from a csv with 1 minute candle data. Sometimes that delays backtesting and data collection too much because not all strategies work in 1 minute. In the future, do you plan to allow csv files to be uploaded in the temporality that you are trying to optimize? Example: if I want to optimize a strategy in 1h, will I be able to upload custom .csv files with 1h candle data or will I always have to upload the .csv file with 1 minute candle data? (which greatly increases the data to be processed).