What is Playwright Mcp Testing Framework

The playwright-mcp-testing-framework is a testing framework designed for automating tests using Model Context Protocol (MCP) servers and Large Language Models (LLMs). It enables users to write tests in markdown format and execute them through an LLM-powered agent.

Use cases

Use cases include automating end-to-end tests for web applications, validating user interactions, and ensuring the functionality of features in a consistent manner across different environments.

How to use

To use the framework, clone the repository, install the required Python and npm dependencies, and set up your environment variables in a .env file. You can run individual tests or all tests in a directory using specific Python commands.

Key features

Key features include writing tests in markdown format, executing tests with LLM-powered automation, support for Playwright MCP server, flexible test execution, multi-layered fallback mechanisms for element selection, and robust error handling and reporting.

Where to use

The framework can be used in software testing environments where automated testing is required, particularly in web application testing, quality assurance processes, and scenarios involving LLMs.

MCP Test Framework

A framework for running automated tests using Model Context Protocol (MCP) servers and LLMs.

Overview

This framework allows you to write tests in markdown files and execute them using an LLM-powered agent through the MCP protocol. The agent interprets the test instructions and executes them using Playwright or other supported MCP servers.

Features

• Write tests in markdown format
• Execute tests with LLM-powered automation
• Support for Playwright MCP server
• Flexible test execution from files or directories
• Multi-layered fallback mechanisms for reliable element selection
• Robust error handling and reporting

Installation

1. Clone the repository

2. Install Python dependencies using requirements.txt:

   pip install -r requirements.txt
   

3. Install required npm packages:

   npm install @playwright/mcp@latest
   

Configuration

Create a .env file in the project root with your OpenAI API key:

OPENAI_API_KEY=
GOOGLE_API_KEY=
ANTHROPIC_API_KEY=

Usage

Run a single test:

python main.py tests/simple_test.md

Run all tests in a directory:

python main.py /tests

Test Format

Tests are written in markdown format. Here’s a simple example:

# Simple Test

## Test Steps

1. Navigate to https://todomvc.com/examples/vanilla-es6/
2. Add a todo item "Buy groceries"
3. Add another item "Finish documentation"
4. Mark "Buy groceries" as completed
5. Verify that the item is crossed out
6. Click the "Active" filter
7. Verify that only "Finish documentation" is shown

Project Structure

├── src/
│   ├── agent/            # MCP agent related code
│   ├── test_runner/      # Test execution logic
│   └── utils/            # Utility functions
├── tests/                # Test case markdown files
├── main.py               # Main entry point
├── requirements.txt      # Python dependencies
├── .env                  # Environment variables
├── README.md             # Project documentation

Future Improvements

Here are some potential areas for enhancing the framework:

Parallel Test Execution

• Implement concurrent test execution to improve throughput
• Add support for test sharding across multiple machines
• Create a dependency system to manage tests that can’t run in parallel

Enhanced Reporting

• Implement detailed HTML test reports with screenshots
• Add support for video recording of test executions
• Create a dashboard for monitoring test results over time

Multi-LLM Support

• Expand beyond OpenAI to support Gemini, Anthropic, and other LLMs
• Implement a fallback mechanism to try alternative LLMs if the primary one fails
• Add LLM response caching to reduce API costs

Resilience Improvements

• Enhance the multi-layered fallback system for element selection
• Add automatic retry logic for flaky tests
• Implement smart waiting strategies based on application state

Test Management

• Create a test suite organization system
• Add tags/categories for selective test execution
• Implement test prioritization based on historical success rates

CI/CD Integration

• Add GitHub Actions workflow templates
• Create Jenkins pipeline configuration examples
• Support for common CI/CD platforms

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.