my experience is the opposite unless you really need the whiz-bang feature you should never use Kafka it’s the least reliable and hardest to run it require active babysitting by skilled team of admin and ops.
if you don’t need low end-to-end latency (tailing consumer blocked on long polling) it’s better to use something like HdFS or AWS S3, but if you need it but have low throughout it’s better to use something like RabbitMQ. if you have both high throughout and need polling then it’s worth investing into Apache Pulsar.
Pulsar is half baked with limited number of half baked clients even for major languages. Kafka instead is a bulletproof solution with tons of clients in any flavor, with instruments to detect issues, alert about them. Don't even compare them.
Pulsar is far better designed than Kafka and is much more reliable and scalable. Clients in every language are an entirely different issue and mostly down to developer bandwidth because it's a small team.
Kafka actually doesn't have that many great clients either, they all are just wrappers around the C++ librdkafka library.
Pulsar clients are also easy to create because of the stateless protocol, per-message acknowledgements, and optional websockets API. Or you can just use their Kafka bridge adapter and use your existing Kafka clients.
Pulsar itself is solid but it’s true number of good clients is limited. Same is true for kafka the only reliable client is the java one. All other are based on librdkafka which still have many major bugs causing it to be stuck and not producing to new leader during leader changes.
One benefit of pulsar is that the client library is completely stateless and much simple. It does not need to know where data is stored it’s all handled by the pulsar broker.
I have used the C++, python and Java clients and they are all pretty usable. Pulsar deployment support is great. It seems the teams deploy it themselves.
I don't think this is coming up enough in the comments! Kafka is a gigantic pain in the ass to run. Configuration is torture. I haven't used Pulsar, but please if someone has a message broker that's easy to run, let me know.
Pulsar currently is easy to setup if you're on Kubernetes, especially if you can dedicate an entire cluster to it. You can run a small 3-node cluster on GKE with local SSD instance storage and GCS/S3 tiering that can handle GBit throughput.
Otherwise look at NATS with NATS Streaming (although they're currently redesigning a better version of it). There's also Liftbridge which is an alternative NATS Streaming fork.
Like most Apache products it comes with a lot of knobs and switches, the more you tweak those the worse it gets. Kafka runs completely fine as long as you aren’t trying to tune every config option, unless you have a super custom use case and you know what you’re doing.
I had trouble accomplishing simple things like setting the maximum message size. You have to:
1. Either increase the default max message size on the broker, or set it on the specific topic(s) involved.
2. Set the max message size at every producer.
3. Set the max batch size at every consumer, which has some interaction with a message count (if I'm remembering correctly).
Because we increased the allowed message size, I wanted to make sure that disk space was freed after a given interval. This was super opaque to get working. I don't remember the specific config keys, but there was about 4 parameters involved, and 3 had no effect if the 4th wasn't set properly. They all used sligtly different naming conventions, so didn't appear in context in the documentation.
Still, I agree that defaults are usually the way to go with Apache stuff. If you find yourself turning a lot of knobs to stabilize something, there's probably a design issue.
if you don’t need low end-to-end latency (tailing consumer blocked on long polling) it’s better to use something like HdFS or AWS S3, but if you need it but have low throughout it’s better to use something like RabbitMQ. if you have both high throughout and need polling then it’s worth investing into Apache Pulsar.