AutoMQ: A Cloud-Native fork of Kafka by separating storage to S3

---

🍵 AutoMQ vs Other Streaming Platforms

Feature	AutoMQ	Apache Kafka	Confluent	Apache Pulsar	Redpanda	Warpstream
Apache Kafka Compatibility[1]	Native Kafka			Non-Kafka	Kafka Protocol
Source Code Availability	Yes	Yes	No	Yes	Yes	No
Stateless Broker	Yes	No	No	Yes	No	Yes
P99 Latency	Single-digit ms latency					> 1200ms
Continuous Self-Balancing	Yes	No	Yes	Yes	Yes	Yes
Scale in/out	In seconds	In hours/days	In hours	In hours (scale-in); In seconds (scale-out)	In hours In seconds (Enterprise Only)	In seconds
Spot Instance Support	Yes	No	No	No	No	Yes
Partition Reassignment	In seconds	In hours/days	In hours	In seconds	In hours In seconds (Enterprise Only)	In seconds
Component	Broker	Broker Zookeeper (Non-KRaft)		Broker Zookeeper Bookkeeper Proxy(Optional)	Broker	Agent MetadataServer
Durability	Guaranteed by S3/EBS[2]	Guaranteed by ISR		Guaranteed by Bookkeeper	Guaranteed by Raft	Guaranteed by S3
Inter-AZ Networking Fees	No	Yes				No

[1] Apache Kafka Compatibility's definition is coming from this blog.

[2] EBS Durability: On Azure, GCP, and Alibaba Cloud, Regional EBS replicas span multiple AZs. On AWS, ensure durability by double writing to EBS and S3 Express One Zone in different AZs.

🔶Why AutoMQ

• Cost effective: The first true cloud-native streaming storage system, designed for optimal cost and efficiency on the cloud. Refer to this report to see how we cut Apache Kafka billing by 90% on the cloud.
• High Reliability: Leverage cloud-shared storage services(EBS and S3) to achieve zero RPO, RTO in seconds and 99.999999999% durability.
• Serverless:
  • Auto Scaling: Monitor cluster metrics and automatically scale in/out to align with your workload, enabling a pay-as-you-go model.
  • Scaling in seconds: The computing layer (broker) is stateless and can scale in/out within seconds, making AutoMQ a truly serverless solution.
  • Infinite scalable: Utilize cloud object storage as the primary storage solution, eliminating concerns about storage capacity.
• Manage-less: The built-in auto-balancer component automatically schedules partitions and network traffic between brokers, eliminating manual partition reassignment.
• High performance:
  • Low latency: Accelerate writing with high-performance EBS as WAL, achieving single-digit millisecond latency.
  • High throughput: Leverage pre-fetching, batch processing, and parallel technologies to maximize the capabilities of cloud object storage.

  Refer to the AutoMQ Performance White Paper to see how we achieve this.

• A superior alternative to Apache Kafka: 100% compatible with Apache Kafka and does not lose any key features, but cheaper and better.

✨Architecture

AutoMQ adopts a Shared-Storage architecture, replacing the storage layer of Apache Kafka with a shared streaming storage library called S3Stream in a storage-compute separation manner, making the Broker completely stateless.

Compared to the classic Kafka Shared-Nothing or Tiered-Storage architectures, AutoMQ's computing layer (Broker) is truly stateless, enabling features such as Auto-Scaling, Self-Balancing, and Partition Reassignment in Seconds that significantly reduce costs and improve efficiency.

⛄Get started with AutoMQ

Deploy Locally on a Single Host

curl https://download.automq.com/community_edition/standalone_deployment/install_run.sh | bash

The easiest way to run AutoMQ. You can experience features like Partition Reassignment in Seconds and Continuous Self-Balancing in your local machine. Learn more

There are more deployment options available:

• Deploy on Linux with 5 Nodes
• Deploy on Kubernetes(Enterprise Edition Only)
• Try AutoMQ on Alibaba Cloud Marketplace
• Try AutoMQ on AWS Marketplace

💬Community

You can join the following groups or channels to discuss or ask questions about AutoMQ:

• Ask questions or report a bug by GitHub Issues
• Discuss about AutoMQ or Kafka by Wechat Group

👥How to contribute

If you've found a problem with AutoMQ, please open a GitHub Issues. To contribute to AutoMQ please see Code of Conduct and Contributing Guide. We have a list of good first issues that help you to get started, gain experience, and get familiar with our contribution process. To claim one, simply reply with 'pick up' in the issue and the AutoMQ maintainers will assign the issue to you. If you have any questions about the 'good first issue' please feel free to ask. We will do our best to clarify any doubts you may have.

⭐License

AutoMQ is released under Business Source License 1.1. When contributing to AutoMQ, you can find the relevant license header in each file.