Trading Platform
Overview
Challenge: Run any number of trading algorithms in parallel against over thousand trading pairs across several centralised crypto exchanges.
Solution: Built and deployed a distributed, scalable, banking-grade streaming platform with high throughput and low latency from exchange data receipt through decisioning to order placement.
Technologies: AWS (EC2, VPC, CodeCommit), Apache Kafka, Rust, Python, Wireguard
Limitations of Bots
There is an inflection point at around 5-10, sometimes even less, trading pairs, when monolithic bots fall over. Usually written in Python or JS/TS, they hit the limitations of the underlying platform, not being able to process all the necessary data fast enough.
NodeJS has a cooperative "async" multitasking logic like Windows 3.1 used to have (no wonder, as it was designed to run in a browser!), and Python is not much better, as threads are not running parallel, only one thread can run at a time. So, to the surprise of users and bot developers alike, putting ever bigger machines under the bot does not solve the problem, only makes the bot more expensive.
This is compounded by constraints of the underlying operating systems and network infrastructure that can be saturated by the high volume of real-time market data.
What is even worse is that updating or reconfiguring a monolithic bot requires stopping it, either leaving positions hanging or waiting for them, for an unpredictable time, to close.
Moving up from the bot concept to the platform concept, therefore, becomes necessary for any reasonably complex, multi-algorithm, multi-pair use case.
Platform to the Rescue
A trading platform is a distributed system of specialised components that collectively provide a robust, high throughput, low latency environment to run trading strategies and algorithms.
The cloud made setting up and scaling platforms viable and relatively low-cost compared to dedicated, high-speed internet connections and internal networks serving large servers. It is now quite possible to spin up many servers in an AWS data centre close to the exchanges they need to communicate with.
Specialised instances can then initiate separate connections to the exchanges to receive and process the real-time time market data feed, spreading it out across several endpoints, staying within rate limits and network capacity.
Screaming Fast Streaming
When processing large amounts of data in real time, a fast, reliable, high capacity core is required. I used Apache Kafka, a banking-grade event streaming solution, as the core for the platform.
It allowed all components of the platform to get, process and forward data without unnecessary latency, allowing the algorithms to quickly decide when to trade.
By separating the components with Kafka, each could be built against their own requirements, e.g., Rust for the most demanding components with a lot of data flowing through and Python for the algorithms, which only use the pre-digested data.
The easy platform for running projects of any size.
To discuss tailored projects, please connect on LinkedIn.