Rendered at 18:10:11 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
spetz 1 days ago [-]
Apache Iggy™ has officially graduated from the Apache Incubator and is now an Apache Software Foundation Top-Level Project (TLP)
It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache project with a growing community. And its biggest feature yet, being the Viewstamped Replication Revisited (VSR) clustering, is coming soon.
junon 1 days ago [-]
I believe spetz is on the project, so I'd like to ask:
Is this a MQ implementation, or can this be used in place of e.g. something like protobufs and iroh/tonic? Does it have something akin to grants/capabilities/authorization? Does it have RPC? Can I have services speak a more privileged set of instructions/commands and clients needing auth to speak dumber/more high level administrative commands just like gRPC?
Also, is the performance due to the (what I assume is custom) zero copy serialization?
The landing page simply states it's essentially "fast messaging" but beyond that it's not entirely clear when I'd use this or what its features/limitations are.
spetz 1 days ago [-]
It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how the data is stored/organized https://iggy.apache.org/docs/ - for example, on top of the topics, we also have "stream" which is just an extra hierarchy that can be used for something like multi-tenancy isolation or anything else depending on the use case. And yes, there's a built-in RBAC (read/manage particular streams, topics, servers, etc.). As for protocols, we have native support for TCP, QUIC, WebSocket (these 3 are stateful binary protocols), and HTTP as well.
There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.
renszarv 6 hours ago [-]
The real important topic (pun intended) for these kind of systems, is what happens, when one of the server is killed, what would the clients see from that. (Not to mention the juice network/DNS or EBS volume problems)
genxy 24 hours ago [-]
For those confused about what Iggy is, the post appears to be written for the Iggy community.
Iggy is basically Kafka written in Rust that has yet to reach the distributed, clustered, fault tolerant stage. Looks cool for sure, but it has a ways to go.
spetz 24 hours ago [-]
Sort of, but anyone familiar with Kafka should feel like at home, although there a few subtle differences when it comes to the transport protocol, hierarchy etc.
And VSR clustering is hopefully coming this week!
languagehacker 1 days ago [-]
I love my Italian Greyhounds so much, so this is great news to me.
spetz 1 days ago [-]
Awesome! You can find mine too in one of the older blog posts, hence the name :)
oulipo 1 days ago [-]
Will it integrate directly with MQTT? Or should I use a MQTT broker and pipe into Iggy?
Also, since we've been asked about this a lot, there's a high chance, that we might implement MQTT Gateway, just like we currently do for Kafka (proxy on top, as a separate runtime).
oulipo 10 hours ago [-]
Could be nice indeed, if it can handle MQTT5 and replace EMQX!
0x457 1 days ago [-]
Iggy is different kind of message queue. This is what you would use instead of Kafka. There is no sink or source for MQTT out of the box.
anony-123 1 days ago [-]
Why the website is looking AI slop?
anony-123 1 days ago [-]
Why this comment is being flagged?
fenestella 1 days ago [-]
[flagged]
whalesalad 1 days ago [-]
[flagged]
nozzlegear 1 days ago [-]
Because it used orange as an accent color? The docs are Astro Starlight with some tweaks to the CSS.
anony-123 1 days ago [-]
Docs looks good, but the landing page is having gradients, pills, badges which looks vibe coded AI slop :)
ramraj07 1 days ago [-]
I always thought Im just not good enough to understand, hey Im just too dumb, surely there's people out there who understand what differentiates all these apache products! Nope never found one.
kitd 1 days ago [-]
Apache is just an umbrella organisation for managing open source software projects. There's nothing that links those projects otherwise.
hofo 1 days ago [-]
Differentiation from each other or from competitor projects?
adolph 1 days ago [-]
[dead]
otabdeveloper4 1 days ago [-]
Don't worry about it, Apache is the hospice where software projects go to die.
0x457 24 hours ago [-]
There are roughly 400 projects on that page; about 90 are in the attic, meaning formally retired. Another 25 are incubating, and some of those look like they’re already dead; nobody has checked the pulse and filled out the paperwork yet.
A huge chunk of Apache is Java libraries.
Some projects are just academia doing academia things; they graduate, the original people move on, and the project dies. Others are corporate donations, including the occasional “we don’t want to play with this anymore; maybe Apache can keep it alive” project.
That being said, the incubator graduation rate is still roughly 75–80%.
How useful or alive those graduates are five or ten years later is a different question.
Saying Apache is where software go to die is just wrong.
otabdeveloper4 8 hours ago [-]
> Saying Apache is where software go to die is just wrong.
...and yet you proceeded to list all the ways that dead projects are being kept undead by Apache? I'm getting conflicting vibes here from your post.
0x457 2 hours ago [-]
Yeah, the incubation projects. Do you not understand the difference between an incubation project and a TLP?
No conflicting vibes: 80% of projects survive the incubation, become TLP, and live a long life. How is that a hospice?
I only listed one way the undead project kept undead - "nobody has checked the pulse and filled out the paperwork yet." That was about _some_ of the 25 projects out of 400 projects. As for academia projects - some of them survive.
There is only one category that has a high mortality rate in Apache: government-donated projects.
skeeter2020 1 days ago [-]
What a crock of shite. Of course there's going to be some dead one when you have this many, but there are also projects that are foundational and many exciting new ones too:
It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache project with a growing community. And its biggest feature yet, being the Viewstamped Replication Revisited (VSR) clustering, is coming soon.
Is this a MQ implementation, or can this be used in place of e.g. something like protobufs and iroh/tonic? Does it have something akin to grants/capabilities/authorization? Does it have RPC? Can I have services speak a more privileged set of instructions/commands and clients needing auth to speak dumber/more high level administrative commands just like gRPC?
Also, is the performance due to the (what I assume is custom) zero copy serialization?
The landing page simply states it's essentially "fast messaging" but beyond that it's not entirely clear when I'd use this or what its features/limitations are.
There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.
Iggy is basically Kafka written in Rust that has yet to reach the distributed, clustered, fault tolerant stage. Looks cool for sure, but it has a ways to go.
Also, since we've been asked about this a lot, there's a high chance, that we might implement MQTT Gateway, just like we currently do for Kafka (proxy on top, as a separate runtime).
A huge chunk of Apache is Java libraries.
Some projects are just academia doing academia things; they graduate, the original people move on, and the project dies. Others are corporate donations, including the occasional “we don’t want to play with this anymore; maybe Apache can keep it alive” project.
That being said, the incubator graduation rate is still roughly 75–80%.
How useful or alive those graduates are five or ten years later is a different question.
Saying Apache is where software go to die is just wrong.
...and yet you proceeded to list all the ways that dead projects are being kept undead by Apache? I'm getting conflicting vibes here from your post.
No conflicting vibes: 80% of projects survive the incubation, become TLP, and live a long life. How is that a hospice?
I only listed one way the undead project kept undead - "nobody has checked the pulse and filled out the paperwork yet." That was about _some_ of the 25 projects out of 400 projects. As for academia projects - some of them survive.
There is only one category that has a high mortality rate in Apache: government-donated projects.
https://projects.apache.org/projects.html