First impressions of Redpanda

What is Redpanda and why was the technology interesting

Redpanda is a streaming data platform that is compatible with the Kafka APIs. It is built using C++ so there is no JVM and no Garbage Collection. There is no Zookeeper. It is based on the RAFT consensus algorithm. It uses a thread-per-core model among several other techniques that make it fast. We highly recommend watching Alex Gallego’s (Founder and CEO of Redpanda Data) talk on Co-designing Raft + thread-per-core execution model for the Kafka-API where he covers many of the techniques in detail. The compatibility with Kafka APIs, the fact that it was built from scratch to take advantage of the current state of hardware, and with operational simplicity as a goal made this very interesting for us.

Deploying Redpanda on IBM Cloud

To ease the installation process, provisioning the IBM infrastructure and configuring Redpanda can be automated. This automation is done utilizing Terraform for infrastructure provisioning and Ansible for automating Redpanda deployment. We have contributed to the [deploy-automation] repository for Redpanda to include IBM Cloud support. To get started:

  1. Clone the repository with terraform scripts and Ansible from [here]
  2. Follow the instructions in this [README] to set up your IBM Cloud account as well as provision the infrastructure using terraform.
  3. Once your infrastructure is successfully set up, follow the instructions [here]

Compatibility testing

Coming from Kafka, our first goal was to check that our existing stack of Kafka Streams applications based on SpringBoot and Avro Schemas work seamlessly with Redpanda as advertised. This was to simulate a scenario where we swap Kafka cluster with a Redpanda cluster. To do so, we took a demo application that used identical package versions and set up that we run in production at our clients and simply replaced our Kafka bootstrap servers and confluent schema registry with the Redpanda services.

First impressions

  • Simplicity of installation and management: The main take-way from setting up Redpanda was its simplicity. Redpanda comes as a single binary. There is no Zookeeper and Schema Registry is also built in. Given our day job with Kafka where we maintain three separated distributed applications consisting of brokers, zookeeper, and schema registry, this was a welcome change. This drastically reduces the complexity of chaining together deployments, managing multiple configuration files, as well as in case of debugging issues.
  • Redpanda CLI is fantastic: The CLI provided was very intuitive to use as well as providing a great amount of power too. It provides an easy way to check the status of your cluster and topics while not requiring the use of external tooling or additional GUIs. As an operator, we we really welcomed the developer friendliness of this tool.
  • Kafka client applications were fully compatible Redpanda was fully compatible with our existing Kafka Streams applications and Kafka Connectors. Initially, we were worried that we would need to reconfigure or change the dependencies of our applications, but through our compatibility testing, it was just as simple as pointing our application to a new brokers and schema registry and our applications functioned as intended.

Conclusion

We are continuing to work with Redpanda and it is a technology we have become comfortable proposing on our client engagements. If you are using Redpanda and have lessons you’ve learned, please feel free to share them via comments. We are always learning and would appreciate hearing what others experiences have been.

References

[1] “Technology Radar”, ThoughtWorks, 2022. [Online]. Available: https://www.thoughtworks.com/en-us/radar. [Accessed: 22- Mar- 2022].

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shahir A. Daya

Shahir A. Daya

Shahir Daya is an IBM Distinguished Engineer and Chief Architect for Business Transformation Services in IBM Consulting.