Apache Kafka terminal UI: Kpow demo

Yeah, really excited to show you the first few things that we've been working on: a CLI, a terminal user interface (TUI), and a set of agent skills.

We'll take a look at those one by one. They're all quite tightly coupled together. If you get the CLI, you get the terminal UI, and you also get the agent skills as part of that. There's also new documentation on our website covering all of these.

So come and check out our documentation: agent skills, CLI, and terminal user interface. There are instructions there on how to get set up and install. It's integrated with Homebrew, so you can add the Factor House tap and brew install the CLI. There are also binaries available to download for various platforms and architectures.

Pretty good documentation there for getting set up. The one caveat for using the CLI is that you need the Kpow API set up. As long as you've enabled the Kpow API, everything today will just work. We support various login models, so you can use it unauthenticated if you're running locally or have any unauthenticated instances of Kpow.

It can also use basic auth (username/password) as well as OpenID Connect (OIDC). OIDC is important, and we'll tell you why later when we give you a sneak peek of what we're working on next. Okay. I think the best way to show a CLI is to just run it.

So `fh`, short for Factor House. It's not just a Kafka UI. I'll give you a tip: there are a few other products coming as well. There'll be a little bit of Flink, a little bit of Iceberg, and maybe a few other things. So you're going to have one CLI that can be used across more of your streaming infrastructure and your streaming architecture.

I was really inspired by the AWS CLI with this one, where you have the one CLI and you can use all of their services. We want the same sort of experience here: one tool for everything you need to do in your streaming infrastructure. So let's do `fh version` and see what we get.

We get a pretty cool bit of ASCII art there, and we're at version nine and the build. Now, as I said, because additional tools are coming, everything we'll see today is `fh kafka`. So we can do `fh kafka topic list`, and we get the topics we have there.

It's integrated with all the niceties you get with modern CLIs, like scrolling tables. You probably also noticed, since my code completion gave it away, that you can add limits, sort by topic name, and even add filters.

You can really narrow down your searches. So this is going to be great any time you're at the terminal. I was a platform engineer for many years. My life was at the terminal, which inspired a lot of what you're going to see today. I think this is going to make engineers and platform engineers pretty happy. Basically everything available in Kpow is available in the API, and therefore available in the CLI as well.

So you can create topics on the CLI, you can delete topics, you can reset consumer group offsets, you can view your lag. All the things you would expect to do in the web app are available from the CLI. But now it's scriptable, so you can write scripts, CI scripts, anything like that to perform actions.

So this is going to be a big enabler of your automation initiatives. If you don't want to type `kafka topic list` all the time, you can just type `k topic list`, because I like to type less. There will be little shortcuts like that as well.

What else can we do here? Let's take a look at the group list. We can see all of the groups that are stable. You could obviously search for unstable as well. We can get all sorts of information about groups, so we can check out what our airline analytics service is doing there.

Now this is wrapped a little, which is fine. It's responsive to your current screen real estate. I'll make it smaller for a second and you'll see that if there's enough screen real estate, we don't wrap.

Let me bring it back to a reasonable size. Connectors. We can check out the Connect clusters. Great, we've got three Connect clusters there. So what's happening on one of these clusters? `fh kafka connect`, and I'm going to cheat and use code completion again, and we'll check out what's happening on the airline cluster.

Great, so we've got a whole bunch of sinks, a couple of sources, and they're all running. Remember that they're all running for a little later. This often trips people up. They look at the basic connector metrics and go, "Great, all my Connect infrastructure is running, so we're all good."

We're going to show something about that a little later. Again, we can do Schema Registry, so we can check out what our registries are and view schemas. We can even view diffs of schemas, or specific versions, which is quite cool.

Let's clear that for a second. Now, everything you just saw... I was actually sitting on a train. Any platform engineer knows the Kubernetes tool k9s. It's a terminal user interface, and I used to use it all the time.

I was sitting on the train and thought, "I wonder what k9s for Kafka would look like." An idea was born, and now we essentially have k9s for Kafka. Don't sue me, k9s. They're open source, I don't think they will. So we now have the Factor House terminal UI.

And here we go. A lovely terminal UI, which looks a little like the Kpow web app. You can see we've got our signals. I think Derek is going to show a bit about signals later. We've got brokers, topic statistics, groups, Schema Registry. We've even got some charts here for throughput and lag.

On the landing page here we have signals as well. Sorry Derek, I'm stealing a bit of your thunder, since I think you're going to demo signals. But yes, we also have signals in the terminal UI. They're a new set of alerts, warnings, and notifications that help keep your cluster healthy.

This is all keyboard navigation based. If I hit 2, it takes me to signals, and if I hit 7, it takes me to the schema page. We'll come back here. I can also use Shift plus the left and right arrows to navigate through these.

For a deeper look at signals, I won't spend too much time here, and Derek can show you those from the web app. But the signals are integrated. I'm going to skip to 4. We'll come back to Data Inspect. This is our topic view. We can look at topics and select them.

Let's take a look at the turnaround topic. I pressed Enter, and what's happened is it's selected my topic. And we have these across here, which are basically tabs. I can hit N and P to go to the next and previous tab.

If I hit N, it goes to the Config tab, and because I selected that topic earlier, it's showing me the config for that particular topic. I can go to reassignments. There are no reassignments and no ACLs on that topic.

If I hit Enter again, it clears the selection, and if we go to the next tab, we can see the config for all of the topics. Interestingly, on any of these table-view pages, you can hit the forward slash key and it brings up a filter up here, and you can just start typing.

Let's do turnaround. So we can filter and quickly find the information we're looking for. My favourite part of this page, though... let's see, which topic do I want? Let's use the search. Let's do baggage derived. If I hit I now, it drops me into Data Inspect. There we go. Now we're in Data Inspect. So that's a shortcut to get you into Data Inspect. But I'll come up here and select a different topic. Let's select baggage arrived. What we've got now is multi-topic search across topics.

You can see we've got some airline baggage mixed in with some transactions. The transactions aren't terribly useful because that topic isn't integrated with Schema Registry, which is why that data looks a bit messy. Again, I've now got tabs along here, so I can use N and P to switch between my data sets.

Let's do one more so I can show off the really cool thing. If I type TX, again we can filter in here. I'll get rid of that one and come back. What was the one I wanted before? Baggage arrived. I'll hit Enter.

Now I've got the most recent hundred or so messages. Most of the features available in the web app are available here too. We can set our window, for example from earliest. And in here I can start typing kJQ queries as well.

Destination. Let's find all the baggage that has gone through Singapore. I'm going to make that a streaming search and set the limit to 5,000. And here we go: two thousand, three thousand, four thousand. There we go, we've got 5,000 records loaded into our session.

Now we can start looking at all of our baggage and find out where our data might be. So I'm really excited by Data Inspect. You can drill in and see more detail about any record just by hitting Enter. Okay, a quick look at groups, because groups is a bit exciting as well.

Again, we can select a row here, TX partner, and get information about that group. If I hit N, we'll see the members just for that group. We'll see the assignments just for that group. You'll see we've got our lag and our rates, everything you would get from the web app.

We'll see the topics that member is consuming. And if there are any ACLs, we'll see those too. We can come back. We could clear that, but actually, let's leave it there.

Let's go back into assignments. If I hit A, I can change the offsets for that member. I can set everything to latest, I can set everything to current, which is essentially a no-op, or I can put everything back to the earliest offset we know about, which in this case is zero on this topic.

So that lets us do staged mutations for resetting offsets. You can do that for all of the partitions on that topic, or press O and do it for a single partition, which is quite cool. This is the first of the mutating actions available in the terminal user interface.

Very soon we'll have other mutating actions as well: topic creation, topic deletion, things like that. I see this being super helpful for local development. Spin it up locally with Community Edition, or however you run your stuff locally, and create your test topics and so on.

Okay. If you're using Kafka Streams, we have Kafka Streams metrics as well. This one is quite cool. We can see the topology and navigate through the topology for our Kafka Streams app. We can look at all the different operations happening there.

The foreach, flatMap, and mapValues nodes, and we can get the current throughput, lag, and metrics for the nodes in our Kafka Streams topology all the way through to our sink. I really like that one. I mentioned earlier we have Schema Registry, so we can come in here.

Again, we can look for data. There's my favourite topic, baggage arrived. And we can see what the schema looks like. We have multiple schema registries here, which is less common, but we have them really to show off that we work with all of the schema registries.

Here you can see we've got AWS Glue, Karapace, and Confluent Schema Registry. Again, I can hit N and P to navigate through the different schema registries. We can get broker information as well: disk configuration, KRaft information, and ACLs.

We can take a look at all of our ACLs. A quick look at Connect. If I hit 9... oh, that's interesting. I wonder what's wrong with that red line there. We'll find out in a minute, I think. Remember how I said that when you ask all of the connectors whether they're healthy, they say, "Yeah, we're good"?

Turns out one of them is not, and we're about to learn why, thanks to the magic of AI. We also have some Flink, just a sneak peek. If you're using Flink, you'll be able to see the state of your Flink clusters as well.

Same setup here. You can look at your Flink jobs and check out the throughput on all of the nodes within that topology. Coming soon will be restarting jobs, taking snapshots, things like that. Okay, I think I've exhausted the terminal UI.