cAlgo keeps on disconnecting when using HFT algo.
cAlgo keeps on disconnecting when using HFT algo.
09 Sep 2014, 14:16
Hi cAlgo Team
I am trading a few High Frequency Algorithms through cAlgo and I keep on being disconnected. My take profit falls in the 1 - 10 pip range, however I place the orders almost every 10 seconds maybe 10 Limit order positions at a time. This leads to hundreds of positions open and an extremely slow platform with constant connection issues while my other cAlgo accounts on the same computer does not experience it at all. This makes me believe that the client platform or the server implements some form of throttling and disconnecting the client if the client place a lot of orders in a short amount of time.
1. Is this true?
2. If yes, why does it occur and when?
3. Is this a potential issue with cAlgo platform not being able to handle such load?
4. Why are there limitations on the amount of open orders? Will this be changed to be more or less in the future?
Thank you.
Replies
jobenb
10 Sep 2014, 12:52
Hi
I have used the Spotware cAlgo 1.25 Yesterday in parallel with the broker cAlgo 1.24 (which continuously kept on disconnecting - with the little red dot next to the account number) and although the latest cAlgo did perform much better it still crashed early this morning unfortunately. I have started it again and will keep it running, so you guys should easily be able to get the load on your side. :-)
How sure can we be that this issue really has been resolved in cAlgo 1.25 and that it is not a server load issue because of the fact that your Spotware server may just be able to handle everything as it is maybe not a true reflection of the load my broker is experiencing?
I do wonder about cServer and if it is not maybe that my broker's cServer is under heavy load not just from my algo but also having thousands of other connections trading at the same time and they decide to do a periodic kill switch on my account due to it causing a lot of the trading order load every few seconds???
Are you guys doing heavy load testing on both the platform and cServer?
Why are there a limit of 2000 orders / positions within cAlgo per account? This limit full scale rollout of algorithms across currency pairs. Could this be increased to 10000 please?
What is the behaviour of cServer if it has too many orders coming in than it can handle?
Please explain the limitations with regards to cAlgo interacting with cServer, so that I can code with that in mind?
Many thanks and I look forward to your reply!
@jobenb
Spotware
10 Sep 2014, 14:24
Yes, we do load testing for both client and server side.
cServer has a limitation for the amount of open positions per account: 2000. We plan to increase this limit in the future. If cServer cannot handle all incoming requests it puts them to the queue. There is no any logic to disconnect clients in case of heavy load.
Please send to engage@spotware.com your account number and broker name. We will troubleshoot the issue.
@Spotware
jobenb
10 Sep 2014, 15:42
RE:
Spotware said:
Yes, we do load testing for both client and server side.
cServer has a limitation for the amount of open positions per account: 2000. We plan to increase this limit in the future. If cServer cannot handle all incoming requests it puts them to the queue. There is no any logic to disconnect clients in case of heavy load.
Please send to engage@spotware.com your account number and broker name. We will troubleshoot the issue.
Thank you. I have sent an email to you containing the details. :-)
I was also thinking about the connecting / disconnecting and I think on the platform side it could also be a timeout that cause the disconnect and then retries after about 3 minutes. I think the timeout might be too short for connection when a lot of orders / positions are involved due to the busyness of the system and context switching of the threads. Is the timeout configurable?
Thank you for all your help! It is much appreciated and I hope we can improve the next release at the same time. ;-)
@jobenb
jobenb
11 Sep 2014, 10:52
RE:
Spotware said:
Yes, we do load testing for both client and server side.
cServer has a limitation for the amount of open positions per account: 2000. We plan to increase this limit in the future. If cServer cannot handle all incoming requests it puts them to the queue. There is no any logic to disconnect clients in case of heavy load.
Please send to engage@spotware.com your account number and broker name. We will troubleshoot the issue.
Just to give you another update. I have had another unhandled exception today on cAlgo version 1.25.
I was also thinking about the connecting / disconnecting and I think on the platform side it could also be a timeout that cause the disconnect and then retries after about 3 minutes. I think the timeout might be too short for connection when a lot of orders / positions are involved due to the busyness of the system and context switching of the threads. Is the timeout configurable?
I look forward to your answers and feedback. Thank you!!
@jobenb
jobenb
12 Sep 2014, 00:59
11/09/2014 19:49:12.649 | cBot crashed: Error #65140095
Hi cAlgo Team
I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095
Could you please explain what it means?
I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.
Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?
Many thanks!!
@jobenb
Spotware
12 Sep 2014, 15:14
I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095
Could you please specify a scenario to reproduce the problem? Did you remove the account while robot was running?
I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.
Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?
We will investigate the problem. Thank you for reporting that.
@Spotware
jobenb
13 Sep 2014, 04:56
RE:
Spotware said:
I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095
Could you please specify a scenario to reproduce the problem? Did you remove the account while robot was running?
I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.
Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?
We will investigate the problem. Thank you for reporting that.
Hi,
I did not close any account while any cBot was running. Normally the platform crashed completely, however this time the cBot stopped running and by investigating the log it showed Error #65140095. This could also potentially be related (I think) to the regular connecting and disconnecting issue being experienced when there are a very high open order load with a lot of open positions.
I think this is quite critical to be resolved as it clearly place a hidden barrier for anyone's cBot to place a lot of orders on various instruments. So someone with a quiet cBot will probably never experience this issue but someone like me with advanced medium - high frequency algorithms will not be able to use the platform at all due to these unknown hidden capacity risks. Luckily I picked this up on a demo account, and luckily you guys are determined to fix it! :D
Thank you.
@jobenb
jobenb
17 Sep 2014, 11:46
RE:
Spotware said:
I have just had a look at the cAlgo 1.25 platform to verify if it is still running and it seems all the robots have crashed with the following error: 11/09/2014 19:49:12.649 | cBot crashed: Error #65140095
Could you please specify a scenario to reproduce the problem? Did you remove the account while robot was running?
I also experience connecting and disconnecting issues with cAlgo 1.25 now, but closing and restarting the platform seems to resolve it again for the next few hours. With cAlgo 1.25 it seems that it only happens once in awhile compared to cAlgo 1.24 with which it happens constantly unless it is during the quiet hours of the market.
Any ideas? Have you had time to verify whether it is a potential timeout which is too short for a high order load (which constantly changes due to TP being reached and new orders being placed)?
We will investigate the problem. Thank you for reporting that.
Dear Spotware
Any luck with finding and resolving the issue?
Thank you.
@jobenb
jobenb
17 Sep 2014, 15:10
RE:
Spotware said:
Do you still observe Error #65140095 in cAlgo 1.25? If so, please press Shift+Ctrl+Alt+T. It will send us the troubleshooting information.
Hi
I did not experience it again, however I will surely press Shift+Ctrl+Alt+T next time.
I still experience disconnecting issues during times of high volatility on an account with between 1000 and 2000 open orders / positions across the Majors. Every time an order is closed a new one is opened to take its place. Take profit is only 1 -10 pips, so there are constantly orders being opened and closed every 10 seconds. Somehow cAlgo struggles and keep on disconnecting. It then appears as if it keeps on being disconnected at which point I have to force close the platform and re-open it again to have another few hours of it working fine until the cycle repeats itself. This cannot be my internet connection because I have other accounts trading on other instances of cAlgo which is not so high frequency and they never experience the same issues of a regular red dot next to the account number...
I look forward to your replies.
@jobenb
jobenb
18 Sep 2014, 14:10
RE:
Spotware said:
Dear Trader,
We plan to optimize our platform in the future. For now we can recommend you to have not more than 500 positions and pending orders.
Thank you.
Could you please let me know how far into the future this is planned as it is currently a real limitation?
Also, one of my previous posts asked a question about a timeout for the client platform connecting to the server. Is this configurable and have you investigated whether that might be set too short considering the circumstances?
@jobenb
Spotware
19 Sep 2014, 10:02
Could you please let me know how far into the future this is planned as it is currently a real limitation?
We cannot provide a time estimate at the moment.
Also, one of my previous posts asked a question about a timeout for the client platform connecting to the server. Is this configurable and have you investigated whether that might be set too short considering the circumstances?
We have more complicated logic and it is not configurable by end users.
@Spotware
jobenb
19 Sep 2014, 19:11
RE:
Spotware said:
Could you please let me know how far into the future this is planned as it is currently a real limitation?
We cannot provide a time estimate at the moment.
Also, one of my previous posts asked a question about a timeout for the client platform connecting to the server. Is this configurable and have you investigated whether that might be set too short considering the circumstances?
We have more complicated logic and it is not configurable by end users.
Hi Spotware
Just to confirm the limitation that we are dealing with here and to put it into perspective...
My broker currently offers 64 currency pairs. If cTrader / cAlgo can only handle a maximum of 500 open orders / positions then that means I can only trade with about 7 - 8 orders / positions per currency pair. This is really limiting for traders since it is common to scale in and out of positions.
I sincerely hope that this issue is dealt with as soon as possible.
Thank you!!
@jobenb
jobenb
30 Sep 2014, 17:03
RE:
jobenb said:
Spotware said:
Could you please let me know how far into the future this is planned as it is currently a real limitation?
We cannot provide a time estimate at the moment.
Also, one of my previous posts asked a question about a timeout for the client platform connecting to the server. Is this configurable and have you investigated whether that might be set too short considering the circumstances?
We have more complicated logic and it is not configurable by end users.
Hi Spotware
Just to confirm the limitation that we are dealing with here and to put it into perspective...
My broker currently offers 64 currency pairs. If cTrader / cAlgo can only handle a maximum of 500 open orders / positions then that means I can only trade with about 7 - 8 orders / positions per currency pair. This is really limiting for traders since it is common to scale in and out of positions.
I sincerely hope that this issue is dealt with as soon as possible.
Thank you!!
Dear Spotware
When will the performance improvements (implemented in Spotware cAlgo) be rolled out to the brokers? At least the latest Spotware cAlgo version is better than my broker's version, however it still needs urgent improvement with regards to this issue of high order load.
Thank you!
I look forward to the improvement and to hear from you. :-)
@jobenb
AlexanderRC
30 Sep 2014, 17:15
RE: RE:
jobenb said:
Just to confirm the limitation that we are dealing with here and to put it into perspective...My broker currently offers 64 currency pairs. If cTrader / cAlgo can only handle a maximum of 500 open orders / positions then that means I can only trade with about 7 - 8 orders / positions per currency pair. This is really limiting for traders since it is common to scale in and out of positions.
I sincerely hope that this issue is dealt with as soon as possible.
Thank you!!
Please vote for Modify volume of a position on vote.spotware.com. If the feature is implemented, you will be able to change the volume of the current position(s) instead of opening additional ones.
@AlexanderRC
jobenb
01 Oct 2014, 01:08
RE: RE: RE:
AlexanderRC said:
jobenb said:
Just to confirm the limitation that we are dealing with here and to put it into perspective...My broker currently offers 64 currency pairs. If cTrader / cAlgo can only handle a maximum of 500 open orders / positions then that means I can only trade with about 7 - 8 orders / positions per currency pair. This is really limiting for traders since it is common to scale in and out of positions.
I sincerely hope that this issue is dealt with as soon as possible.
Thank you!!
Please vote for Modify volume of a position on vote.spotware.com. If the feature is implemented, you will be able to change the volume of the current position(s) instead of opening additional ones.
Sorry, but this does not resolve this issue. This issue is not about modifying volume.
It is about cAlgo not being able to handle the order load and thereby limiting the trader in how he / she can implement his / her strategies!
@jobenb
jobenb
02 Oct 2014, 16:14
RE:
Spotware said:
We plan to deliver current Spotware version to all brokers next week. We cannot provide a time estimate for the release of other performance optimizations.
Hi Spotware
Just a bit more information...
I watched my algos trade this evening and there were about 450 positions and 350 orders open. Every few hours I have to close the platform and open it again, because the platform would freeze and I can't do anything. Every time I notice that it has also disconnected together with the freezing, however the only way is to force close the platform and open it again. Anyways, I watched it trade ok and then a news release and volatility hits the market with a bang - the server seemed not to respond to the pending orders and it just disconnected again and froze. a Complete waste of time to trade with cAlgo like this. Anyways I hope you have found the issue and you are working on it as I would constantly monitor this thread for an update as this now becomes critical.
I have noticed that a lot of brokers advertise that you can run high frequency algorithms on ECN cAlgo, however I really would like to but if cAlgo struggles with a few hundred positions and a few hundred orders then I have my doubts...
Just a few questions...
Did you manage to reproduce this issue on your side?
Do you know what is causing this?
Lastly, thank you for all your hard work and for working to rectify this issue!
@jobenb
jobenb
17 Oct 2014, 16:19
RE:
Spotware said:
Hello jobenb,
Yes, we can reproduce the performance issue. Yes, we know how to fix it. However we cannot provide a time estimate for the release. We will send you an email message when it will be resolved.
Dear Spotware,
Will this be fixed in the next release?
@jobenb
jobenb
22 Oct 2014, 00:06
Update
jobenb said:
Spotware said:
Hello jobenb,
Yes, we can reproduce the performance issue. Yes, we know how to fix it. However we cannot provide a time estimate for the release. We will send you an email message when it will be resolved.
Dear Spotware,
Will this be fixed in the next release?
Dear Spotware
This disconnecting issue is still occurring although a bit less than before the release of last week Friday.
I can't wait for this issue to be resolved... :-)
Thank you!
@jobenb
jobenb
24 Oct 2014, 00:16
23/10/2014 11:38:20.322 | cBot crashed: Error #65140095
jobenb said:
Spotware said:
Hello jobenb,
Yes, we can reproduce the performance issue. Yes, we know how to fix it. However we cannot provide a time estimate for the release. We will send you an email message when it will be resolved.
Thank you!
Dear Spotware
Today I looked at my account and I saw that my cBot was not running anymore. Upon investigation in the log I found the following error:
23/10/2014 11:38:20.322 | cBot crashed: Error #65140095
What does this mean?
@jobenb
jobenb
28 Oct 2014, 10:28
Disconnecting Issue
Hi Spotware
Today I noticed another time when the cAlgo platform keeps on disconnecting from the server, and that is whenever I try and optimize more than 3 currency pairs at the same time. I have resources set at 12% per optimization. It tries to connect a few times and then after a while it just seems to give up trying and the red dot stays next to the account number.
Are you currently working on this issue or is it pushed back in the queue?
Thank you.
@jobenb
Spotware
29 Oct 2014, 09:16
RE: Disconnecting Issue
jobenb said:
Hi Spotware
Today I noticed another time when the cAlgo platform keeps on disconnecting from the server, and that is whenever I try and optimize more than 3 currency pairs at the same time. I have resources set at 12% per optimization. It tries to connect a few times and then after a while it just seems to give up trying and the red dot stays next to the account number.
Are you currently working on this issue or is it pushed back in the queue?
Thank you.
Does the problem still persist? Could you please send environment details (live/demo, broker name) to troubleshooting@spotware.com?
@Spotware
jobenb
31 Oct 2014, 12:27
( Updated at: 21 Dec 2023, 09:20 )
RE: RE: Disconnecting Issue
Spotware said:
jobenb said:
Hi Spotware
Today I noticed another time when the cAlgo platform keeps on disconnecting from the server, and that is whenever I try and optimize more than 3 currency pairs at the same time. I have resources set at 12% per optimization. It tries to connect a few times and then after a while it just seems to give up trying and the red dot stays next to the account number.
Are you currently working on this issue or is it pushed back in the queue?
Thank you.
Does the problem still persist? Could you please send environment details (live/demo, broker name) to troubleshooting@spotware.com?
Dear Spotware
I was using my broker cAlgo. I will try Spotware cAlgo as it has a newer version and will let you know if the disconnecting issue persist.
Also, some further details about the issue above... I initially started the optimizations on 87% resources, but then changed it to 15% during runtime. The UI was not very responsive and then after a while the disconnecting started happening. I guess the platform reached a point where the connection must have timed out due to all the processing from the optimizations at the same time. This might relate to the issue of running HFT algos on multiple currencies and then experiencing constant disconnection and UI unresponsiveness.
I will however do some testing with the Spotware cAlgo as it is the latest build and because you indicate that this issue should not persist in the latest build.
Thank you!!
My computer specification is:
@jobenb
jobenb
01 Nov 2014, 04:01
( Updated at: 21 Dec 2023, 09:20 )
Issues still occuring in cAlgo v 1.27.133
jobenb said:
Spotware said:
jobenb said:
Hi Spotware
Today I noticed another time when the cAlgo platform keeps on disconnecting from the server, and that is whenever I try and optimize more than 3 currency pairs at the same time. I have resources set at 12% per optimization. It tries to connect a few times and then after a while it just seems to give up trying and the red dot stays next to the account number.
Are you currently working on this issue or is it pushed back in the queue?
Thank you.
Does the problem still persist? Could you please send environment details (live/demo, broker name) to troubleshooting@spotware.com?
Dear Spotware
I was using my broker cAlgo. I will try Spotware cAlgo as it has a newer version and will let you know if the disconnecting issue persist.
Also, some further details about the issue above... I initially started the optimizations on 87% resources, but then changed it to 15% during runtime. The UI was not very responsive and then after a while the disconnecting started happening. I guess the platform reached a point where the connection must have timed out due to all the processing from the optimizations at the same time. This might relate to the issue of running HFT algos on multiple currencies and then experiencing constant disconnection and UI unresponsiveness.
I will however do some testing with the Spotware cAlgo as it is the latest build and because you indicate that this issue should not persist in the latest build.
Thank you!!
My computer specification is:
Dear Spotware
I have tested cAlgo v 1.27.133, running my HFT algos and the platform:
1. Is unresponsive when running HFT algos on multiple currencies.
2. Keeps on disconnecting when the market is very active and therefore a lot of ticks are being received by the platform.
My algos also ignore any new ticks if it is still running to prevent any form of possible build up or back log of tick events. It seems the problem is the platform not being able to cope with tracking all the open orders, pending orders and new orders. Obviously every system has its limits, but I do believe there is an issue here.
You mentioned in one of your previous posts that you can reproduce this issue and that you know how to fix it, so I guess my question is:
Is this issue currently being fixed or is this fix on hold?
@jobenb
alexgbg30@gmail.com
21 Jul 2017, 22:31
RE:
Spotware said:
Dear Trader,
We plan to optimize our platform in the future. For now we can recommend you to have not more than 500 positions and pending orders.
Dear spotwear
it's almost 2018 and and still struggling after opening over than 500 positions
@alexgbg30@gmail.com
Spotware
09 Sep 2014, 14:27
cServer doesn't have any throttling logic.
We prepared several performance optimizations in cAlgo 1.25. If it is possible please check how your cBots will perform in cAlgo 1.25. Currently cAlgo 1.25 is deployed for Spotware environment only:
http://spotware.ctrader.com/calgo-spotware-setup.exe
@Spotware