How Pepperstone made Kpow its first stop for Kafka visibility
Challenge
Pepperstone's engineering team was migrating data-intensive, batch-oriented processes to an event-based architecture on Apache Kafka. Limited specialist support and a lack of whole-system visibility meant senior engineers were often pulled into tasks the rest of the team couldn't self-serve.
Solution
Kpow gave Pepperstone's engineers:
- Direct visibility into topics and consumer groups after every deployment, without an internal command-line job
- A way to sample data and assess capacity requirements as the platform scaled
- A shared reference point for communicating Kafka's state to the wider IT team and the business
Result
- Kpow became the team's default first stop for anything happening in the cluster
- Faster turnaround on troubleshooting and capacity planning
- A clearer way to flag development and infrastructure issues before they became bigger problems
“If I want to see anything in our cluster, I can now go to Kpow first, rather than establishing an internal command-line job.”
Douglas Reith, Team Lead, Pepperstone
Scaling a Forex and CFD platform onto Kafka
Founded in 2010, Pepperstone is an online Forex and CFD broker built around a digitally-led trading platform. It has grown into one of the largest MetaTrader brokers in the world, processing an average of US$12.55 billion in trades a day across more than US$9.2 billion in retail trading accounts.
Team Lead Douglas Reith’s team was migrating data-intensive, batch-oriented processes over to an event-based architecture on Apache Kafka. Kafka sits at the center of the client services the team delivers, but the team had limited specialist support available as it scaled, and Kafka itself offers little tooling for whole-system visibility. That made it difficult to review the current state of the platform with confidence, whether the goal was resolving an issue or planning for future growth.
The challenge: scaling Kafka without whole-system visibility
Pepperstone’s rapid growth meant an increasing reliance on Kafka across the business, but managing that workflow without external support was becoming a difficult, laborious task. The automation needed to keep pace with demand on internal resources didn’t exist yet.
Limited availability of specialist Kafka support created a bottleneck. Senior engineers were often pulled into tasks that few others on the team could self-serve using the existing core software, leaving less time for their own priorities. Compounding that, Kafka doesn’t offer much native tooling for whole-system visibility, so the team often couldn’t review the current state of the platform with real confidence, whether they were troubleshooting an issue or planning for a future project.
Factor House and Pepperstone connected at the VoxxedDays conference in 2019, where Factor House was presenting its Apache Kafka Three Ways (AK3X) workshop and signing up beta testers for Kpow. Factor House delivered a tailored AK3X workshop to Pepperstone’s engineers, who trialled Kpow soon after in their staging environment. Since the team was still early in its Kafka journey, that trial surfaced a long list of quick wins.
The solution: Kpow as the team’s first stop for Kafka
Day to day, Kpow is where Pepperstone’s engineers go to work directly with Kafka. As Reith put it: “If I want to see anything in our cluster, I can now go to Kpow first, rather than establishing an internal command-line job.”
That usage has grown over time. “Initially we would open up Kpow to check topics and consumer groups after the deployment of a new application or if we wanted to get some statistics,” Reith said. “Now, we are doing that as well as sampling data, assessing capacity requirements, and troubleshooting issues.”
That visibility extends beyond the engineering team itself. Kpow gives Pepperstone a shared reference point for talking about Kafka with the rest of IT and the business. “Kpow has improved how we communicate Kafka to the rest of the IT team and the rest of the business,” Reith said. “It has helped us assess what we may need to do in the future, as well as identify issues related to development and infrastructure standards.”
The results: a faster, more confident way to run Kafka at scale
Kpow’s visibility turned a tool the team once opened only to check topics and consumer groups after a deployment into their default first stop for anything happening in the cluster. Reith described the shift as giving the team “a much greater perspective about Kafka and how it and our whole system is operating at any given time.”
That perspective now underpins troubleshooting, capacity planning, and how the team flags development and infrastructure issues before they escalate, work that previously depended on establishing an internal command-line job each time. Pepperstone is now a Kpow enterprise client, and Reith said the team would recommend it to others: “to raise the awareness and visibility of Apache Kafka, as well as assist data engineers in their work.”
Taken together, Kpow gave Pepperstone’s engineers a faster way to troubleshoot, a clearer way to plan for capacity as the platform scaled, and a shared language for talking about Kafka’s health across the wider business, letting a small team keep pace with one of the world’s largest MetaTrader brokers.
Kpow by Factor House makes the Kafka developer experience simple, so teams can save time and money while growing their Kafka expertise.
