Mcp Vllm Benchmark
What is Mcp Vllm Benchmark
mcp-vllm-benchmark is a proof-of-concept tool designed to benchmark vLLM using MCP, allowing users to interactively test and measure the performance of various models.
Use cases
Use cases include benchmarking different AI models to evaluate their performance, conducting comparative analysis of model outputs, and optimizing model selection for specific tasks.
How to use
To use mcp-vllm-benchmark, clone the repository and add it to your MCP servers configuration. Then, you can run benchmarks by providing specific commands to test different models and endpoints.
Key features
Key features include the ability to run multiple benchmark iterations, compare results, and the flexibility to specify various parameters such as the number of prompts and models to benchmark.
Where to use
mcp-vllm-benchmark can be used in fields such as machine learning, AI model evaluation, and performance testing, particularly for applications involving large language models.
Clients Supporting MCP
The following are the main client software that supports the Model Context Protocol. Click the link to visit the official website for more information.
Overview
What is Mcp Vllm Benchmark
mcp-vllm-benchmark is a proof-of-concept tool designed to benchmark vLLM using MCP, allowing users to interactively test and measure the performance of various models.
Use cases
Use cases include benchmarking different AI models to evaluate their performance, conducting comparative analysis of model outputs, and optimizing model selection for specific tasks.
How to use
To use mcp-vllm-benchmark, clone the repository and add it to your MCP servers configuration. Then, you can run benchmarks by providing specific commands to test different models and endpoints.
Key features
Key features include the ability to run multiple benchmark iterations, compare results, and the flexibility to specify various parameters such as the number of prompts and models to benchmark.
Where to use
mcp-vllm-benchmark can be used in fields such as machine learning, AI model evaluation, and performance testing, particularly for applications involving large language models.
Clients Supporting MCP
The following are the main client software that supports the Model Context Protocol. Click the link to visit the official website for more information.
Content
MCP vLLM Benchmarking Tool
This is proof of concept on how to use MCP to interactively benchmark vLLM.
We are not new to benchmarking, read our blog:
This is just an exploration of possibilities with MCP.
Usage
- Clone the repository
- Add it to your MCP servers:
{ "mcpServers": { "mcp-vllm": { "command": "uv", "args": [ "run", "/Path/TO/mcp-vllm-benchmarking-tool/server.py" ] } } }
Then you can prompt for example like this:
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888 benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.
Todo:
- Due to some random outputs by vllm it may show that it found some invalid json. I have not really looked into it yet.
Dev Tools Supporting MCP
The following are the main code editors that support the Model Context Protocol. Click the link to visit the official website for more information.
