honeybadger
04 Jun 2021, 13:32 ( Updated at: 21 Dec 2023, 09:22 )

L2 Data

Ahoy,

mimoMoney said:

Thank you very much for the prompt response. 

Looking at the chart at the time of entry, the entry prices in the providers history aren't possible neither are mine. Below 5 mins graph indicates that between 6 55 AM and 7 AM UTC the high was 15294 and the low was 15265 . However the entry prices mentioned in our histories are far from this range.

 

I found this so intriguing that I want to offer some troubleshooting - cTrader records a market snapshot for each deal, which can be retrieved through position/deal info:

This will help you identify why you got the price you got. But I just realize this info is only available for cTrader and cAlgo, not for cCopy. In the spirit of transparency, could the Deal Info and Market Snapshot function be added to cCopy, Panagiotis?

In the end, with different brokers you get different prices, so maybe the strategy provider was just lucky to trade with some very bears or some very rookies who took his 10 lot limit order. But then again the order book shouldn't look that thin at German market opening for him to succeed with that. So it would be indeed interesting to know what happened there.

honeybadger
04 Jun 2021, 13:03

Same same, different broker

PanagiotisCharalampous said:

Hi ryclee,

The broker is responsible for historical data. Please contact them directly.

Best Regards,

Panagiotis 

Join us on Telegram

Hi Panagiotis,

So far people here have experienced the problem with USDCAD, USDCHF, EURUSD, and USDJPY on Pepperstone, TradeView Forex, Fondex, and now I am adding EURUSD and DE30 (haven't checked other symbols) on IC Markets.

Are you sure this is a broker problem when so many brokers have the problem all at once, on all kinds of symbols?

Btw, this issue doesn't happen on D1 to me. I've checked D1 and h1.

Best,

h.

honeybadger
24 Dec 2020, 03:55

Very helpful information, thanks a lot, Panagioti, and happy holidays!

honeybadger
17 Dec 2020, 05:09

RE:

Yes Please!

honeybadger
11 Dec 2020, 02:14

RE: RE: RE: RE: It seems to work with Python

Hi there,

happy to hear it worked.

mywebsidekicks said:

So how do I actually send the data?

It needs to be sent to 'live.ctraderapi.com:5035' Right?

Yes, as it is documented in the Open API Reference.

Via regular Python socket interface or with SSL wrapper?

I don't know if it works via a bare TCP Socket as I would never send my Client Secret unencrypted. TLS (more specifically: the TLS certificate) is the only piece in this setup ensuring that nobody will hijack your account and your money. 

My advice for these kinds of questions, however: just try it out! Even if your attempt does not work, you will gain valuable programming experience and help avoiding these kind of problems in the future.

Stay safe (off- and online),

h.

honeybadger
17 Nov 2020, 02:27

RE: RE: It seems to work with Python

mywebsidekicks said:

So how did you install protobuffs on Python, with pip or?

Also, do you find Open API really any better than Fix API for trading?

Hi there,

Thanks for your comments. You do not need to install protobuf for Python. Instead, you need the appropriate protobuf compiler for your system architecture which is available from Google (more detailed info is available under the link I posted above). The compiler generates Python classes for each type of protobuf message that you can import into your project and work with - and all classes come with the methods needed for handling them.

I can't really comment on your second question as I have no experience with FIX API - its functionality simply doesn't fit my needs.

Cheers,

h.

honeybadger
16 Nov 2020, 01:39

It seems to work with Python

depot051 said:

Protocol buffer doesn't support Python properly, you have to modify it manually and install pure python protocol buffer implementation (because C++ scoping rules ruins Python) to get them work.

Hi,

The above post is already 2 years old, so perhaps it was true at that time. But if you acknowledge that Protobuf is from Google and a company that smart and big is not super likely to cause major portability issues, then a bit of digging, hard work and reading protocol references gets you onto a promising path. Please see my post on formatting protobuf messages in Python for a first lead.

@irmscher9, the main building blocks for a Python app here seem to be the TCP Socket¹, and the Protobuf classes² with proper formatting and wrapping (see my linked post) - then you essentially have a communication interface with the server and what you do internally with that communication is entirely up to you. I am still at the beginning of my superpower-python-conquer-the-world-trading-app but at least I have managed to send a request to the server, and get a reply in accordance with the Open API reference.

Note that for the beginning and for testing, you can handle the OAuth2³ part manually - you can implement an HTTP client and proper redirects at a later stage. Still, once you have your credentials, your app should authenticate with the server using the ProtoOAApplicationAuthReq() and the ProtoOAAccountAuthReq() message.

Hope this helps,

h.

¹ Python Socket basics: https://docs.python.org/3.7/howto/sockets.html 
² More information on Protobuf for Python: https://developers.google.com/protocol-buffers/docs/reference/python-generated
³ OAuth2 basics: https://aaronparecki.com/oauth-2-simplified/, and for Python: https://requests-oauthlib.readthedocs.io/en/latest/oauth2_workflow.html
 

honeybadger
15 Nov 2020, 23:46

Workaround

Hi taharxeg and Spotware team,

I encountered the same problem and have literally just now found a solution/workaround as described here.

.

0. The Code Snippet in question (Python):

proto = OAMessages.ProtoOAApplicationAuthReq()
# following line is equal to proto.payloadType = 2100
proto.payloadType = int(OAMessagesModel.ProtoOAPayloadType.PROTO_OA_APPLICATION_AUTH_REQ)
proto.clientId = self.CID
proto.clientSecret = self.CSEC

### debug message
print(proto.serializeToString())
### end debug

# transmits the message to the server; tx(msg) adds a uint32 LENGTH message before
self.tx(proto.serializeToString())

.

1. The Problem

The generated Protobuf classes yield a different ByteString in Python vs. C#. This is true for the ProtoOAAuthReq class but perhaps for other classes, too. @taharxeg posted the two different ByteStrings, and I can reproduce this behavior with my own Client ID and Client Secret (both strings are the pure messages without the separate uint32 LENGTH message at the beginning):

C# ByteString (as in Spotware's C# sample project) represented in Hex:

08 B4 10 12 6D 12 37 [Client ID] 1A 32 [Client Secret]

Python ByteString represented in Hex:

08 B4 10 12 37 [Client ID] 1A 32 [Client Secret]

Problem Description: The Python-generated ByteString is missing 2 bytes between its 3rd and 4th byte.

.

2. The Result

As a result, sending my Python ProtoMessage yields the response:

\x08\xde\x10\x12>\x1a\x0fINVALID_REQUEST"+Protocol message tag had invalid wire type.\x1a2[Client Secret]

.

3. Error Tracing

Upon investigation of the Spotware example project, I find the following behavior in the file Open Api Library/OpenApiMessagesFactory.cs, lines 173-179:

public ProtoMessage CreateAppAuthorizationRequest(string clientId, string clientSecret, string clientMsgId = null)
{
  var _msg = ProtoOAApplicationAuthReq.CreateBuilder();
  _msg.SetClientId(clientId);
  _msg.SetClientSecret(clientSecret);
  return CreateMessage((uint)_msg.PayloadType, _msg.Build().ToByteString(), clientMsgId);
}

To me, the return statement looks like a generic ProtoMessage is being built that contains the the three fields (uint32) payloadType, (byte[]) payload, and (string) clientMsgId. The payload field, in contrast, contains the ByteString of a ProtoOAApplicationAuthReq message with the format (string) clientId, (string) clientSecret. This seems to be different than building a ProtoOAAuthReq with Python.

.

4. Workaround (EDITED)

This is not optimized, I just tried to confirm my suspicion - and there you go! The workaround is to mimic this behavior. Obviously the field values are preceded by a 2-byte "field indicator" or similar, so I cut off the first 3 bytes of my Python ByteString and embed the remainder in a generic ProtoMessage:

proto = OAMessages.ProtoOAApplicationAuthReq()
proto.clientId = self.CID
proto.clientSecret = self.CSEC
msg = bytearray(proto.SerializeToString())

wrapper = OACommon.ProtoMessage()
wrapper.payloadType = OAMessagesModel.ProtoOAPayloadType.PROTO_OA_APPLICATION_AUTH_REQ
wrapper.payload = bytes(msg[3:])
# EDIT: this works also with the original ProtoOAApplicationAuthReq, i.e.
# wrapper.payload = proto.SerializeToString()

#### debug messages
print("ProtoMessage:\n")
print(wrapper.SerializeToString())
#### end debug

self.tx(wrapper.SerializeToString())

Result: And suddenly, the server understands me! 

\x08\xde\x10\x12:\x1a\x16CH_CLIENT_AUTH_FAILURE" OA client is not in active state

(This error message is expected b/c my app has not been activated for the API yet).

.

5. The bug

Although I am not sure, the behavior of Google's/Python's ProtoOAApplicationAuthReq class looks unwanted to me. Judging from the Open API Reference, the server expects a generic ProtoMessage with the format (uint32) payloadType | (byte[]) payload | (string) clientMsgId. In my opinion, ProtoOAApplicationAuthReq.SerializeToString() should either

• wrap the payload in a ProtoMessage and add the payloadType in the ProtoMessage but not in the ProtoOAApplicationAuthReq message, or
• create the payload only - without the payloadType - and leave the payload ByteString for manual wrapping, as in the C# example project. (EDIT: After identifying The Solution - see below - it turns out this is exactly what the method does.)

I am not sure what exactly the desired behavior should be (maybe even the C# implementation has "undesired" behavior in this sense?) - and I hope you can shed some light on this. Certainly the desired behavior should not be to generate a ByteString first and then manipulate it manually (cut off the first 3 bytes) before continuing. If the OpenApiMessagesFactory.cs was automatically generated by ProtoGen or similar, this smells like a bug on Google's side - this is why the perspective of the Spotware team on this is so valuable.

.

EDIT: 6. The Solution

I just found out that the string manipulation is not necessary. So the correct procedure is 

1. Generate ProtoOAApplicationAuthReq message
2. Serialize message to ByteString
3. Generate generic ProtoMessage wrapper
   • Set payloadType according to the message in (1)
   • Set payload to the serialized message from (2)
   • Optionally set clientMsgId
4. Do what you like with your awesome, server-readable ProtoMessage!

After understanding this, I do see that the Open API reference somehow captures this, but I still think that the way it is described in the Reference is somewhat confusing to us amateur programmers. It would be amazing if you could update the reference to clarify the necessary extra step and the difference between the Open API Common Messages (the "message wrappers") and the Open API Messages (the payload of a ProtoMessage).

.

Thanks for taking the time to check this!

Best,

h.

honeybadger
28 Aug 2020, 20:13

Hi there,

My two cents:

• From a copying POV: I use those metrics to gauge the success of a strategy. Those metrics are vital to me and I wouldn't follow a strategy provider who doesn't disclose these (especially when others do).
• From a provider POV: I love the idea of having privacy controls. However, the reason I chose cTrader (also as a strategy provider) is its approach to transparency compared to other trading platforms (especially it aligns with my own philosophy towards transaprency). This is why - if I must choose - I'd prefer to have an open and transparent platform rather than competing with providers who choose not to disclose key metrics.

Cheers,

h.