What is Mcp Kafka

mcp-kafka is a server-side implementation of the model context protocol (MCP) designed for Apache Kafka. It facilitates reliable interactions between language models and the Kafka ecosystem, including Kafka Connect, Burrow, and Cruise Control.

Use cases

Use cases for mcp-kafka include real-time data processing, monitoring Kafka cluster health, managing data pipelines with Kafka Connect, and optimizing resource allocation through Cruise Control.

How to use

To use mcp-kafka, set the appropriate environment variables for Kafka, Kafka Connect, Burrow, and Cruise Control APIs. The server responds based on the presence of these variables and supports various Kafka operations such as consuming and producing messages, as well as managing Kafka clusters and topics.

Key features

Key features of mcp-kafka include support for core Kafka APIs (excluding Streams), integration with Burrow and Cruise Control REST APIs, and capabilities for managing Kafka resources like topics and consumer groups.

Where to use

mcp-kafka is suitable for applications that require interaction with Apache Kafka and its ecosystem, particularly in environments where language models are utilized for data processing and message handling.

Overview

mcp-kafka is a server-side implementation of the model context protocol (MCP) for Apache Kafka. It allows language models (LLM/SLM) to reliably interact with Kafka & its ecosystem, including Kafka Connect, Burrow, & Cruise Control.

NOTE: This is a WIP, changes/potential errors are expected.

Features

The server supports capabilities based on the core Kafka APIs, excluding Streams (for now), along with the Burrow & Cruise Control REST APIs.

TODO

• [ ] Use asyncio and aiohttp.
• [ ] Set env config values in the data class.
• [ ] Finish admin/consumer/producer API support.
• [ ] Support Burrow API.
• [ ] Support CC API.
• [ ] Service + integration tests.
• [ ] Publish to PyPI (uv pip install mcp-kafka)
• [ ] Test Dockerfile & push to Dockerhub (docker pull bkpowers/mcp-kafka)
• [ ] Usage/config/deployment options + demo in README.
• [ ] Consider Strimzi/kcctl integrations.
• [ ] Prompts with auto-complete support for certain resources/tools.

MCP Capabilities

Tools

Kafka

• consume
• produce
• describe_kafka_cluster
• describe_kafka_topics
• describe_kafka_consumer_groups
• describe_kafka_delegation_tokens
• describe_kafka_log_dirs
• describe_kafka_configs
• describe_kafka_acls

Kafka Connect

• get_kafka_connect_cluster_info
• get_kafka_connect_config
• get_kafka_connect_connectors
• get_kafka_connect_connector_plugins
• get_kafka_connect_loggers

Burrow

• burrow_healthcheck
• burrow_list_clusters
• burrow_describe_cluster
• burrow_list_consumers_with_group_detail
• burrow_list_topics
• burrow_check_consumer_group_status

Cruise Control

• cruise_control_get_state
• cruise_control_get_kafka_cluster_load
• cruise_control_get_partition_resource_utilization_and_load
• cruise_control_get_partition_and_replica_state
• cruise_control_get_optimization_proposals
• cruise_control_get_user_request_result

Resources

• topic, connector, consumer group

Prompts

Usage / Configuration

1. Claude Desktop
2. Cursor
3. Windsurf
4. Langchain MCP adapter
5. Azure OpenAI

Environment Variables

Supported APIs are (currently) enabled by the presence of their specific environment variable. If none are present, the server responds as empty. If 1 or more variables are present, then the respective tools are also present.

• KAFKA_BOOTSTRAP_SERVERS: Kafka Admin, Consumer, Producer APIs
• KAFKA_CONNECT_API_URL: Kafka Connect API
• KAFKA_BURROW_API_URL: Burrow API
• KAFKA_CRUISE_CONTROL_API_URL: Cruise Control API