mcpengine/infra/factory-tools/mcp-validator/README.md

# MCP Validator

A testing suite and reference implementation for the [Model Context Protocol (MCP)](https://github.com/modelcontextprotocol).

## Summary

The MCP Validator provides a comprehensive environment for testing and validating MCP server implementations. It includes reference implementations and a testing framework to ensure compliance with the MCP specification.

**Latest Protocol Support**: The validator now supports MCP protocol version **2025-06-18**, which includes major enhancements like structured tool output, OAuth 2.1 authentication, elicitation support, removal of JSON-RPC batching, and enhanced security features.

## 🔐 OAuth 2.1 Authentication Support

The validator includes comprehensive OAuth 2.1 authentication support for the 2025-06-18 protocol version, providing a complete framework for secure MCP implementations.

### Authentication Features

- **OAuth 2.1 Compliance**: Full RFC 6749, RFC 6750, and OAuth 2.1 draft compliance
- **Bearer Token Support**: Proper Bearer token extraction and validation
- **WWW-Authenticate Headers**: Standards-compliant 401 responses with authentication challenges
- **Resource Server Capabilities**: Complete OAuth 2.1 resource server implementation
- **MCP-Protocol-Version Headers**: Protocol version negotiation via HTTP headers
- **Security Headers**: CORS, Origin validation, and DNS rebinding attack prevention

### OAuth 2.1 Configuration

The reference HTTP server supports OAuth 2.1 authentication through environment variables:

```bash
# Enable OAuth 2.1 authentication
export MCP_OAUTH_ENABLED=true
export MCP_OAUTH_INTROSPECTION_URL=https://auth.example.com/oauth/introspect
export MCP_OAUTH_REQUIRED_SCOPES=mcp:read,mcp:write
export MCP_OAUTH_REALM=mcp-server

# Optional: Configure resource indicators (RFC 8707)
export MCP_OAUTH_RESOURCE_INDICATORS=https://api.example.com/mcp
```

### Authentication Testing

Test OAuth 2.1 compliance with the built-in authentication test suite:

```bash
# Test OAuth 2.1 features (framework validation)
python mcp_testing/scripts/http_compliance_test.py --debug

# Test with authentication enabled
MCP_OAUTH_ENABLED=true python ref_http_server/reference_mcp_server.py --port 8088 &
python mcp_testing/scripts/http_compliance_test.py --server-url http://localhost:8088

# Test Bearer token handling
curl -H "Authorization: Bearer your-token-here" \
     -H "MCP-Protocol-Version: 2025-06-18" \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"client_info":{"name":"Test"},"client_capabilities":{"protocol_versions":["2025-06-18"]}}}' \
     http://localhost:8088/messages
```

## 🚀 2025-06-18 Protocol Features

### Core Enhancements

- **Structured Tool Output**: Tools return structured responses with `content`, `isError`, and `structuredContent` fields
- **Enhanced Tool Schema**: Tools use `inputSchema` (renamed from `parameters`), `title`, and `outputSchema` fields
- **JSON-RPC Batching Removal**: Batch requests are properly rejected for 2025-06-18 protocol
- **Elicitation Support**: Framework for user interaction and data collection
- **Protocol Version Headers**: HTTP requests include `MCP-Protocol-Version` header

### Security Improvements

- **OAuth 2.1 Authentication**: Complete authentication framework
- **Enhanced Error Handling**: Improved error responses with security considerations
- **Origin Validation**: DNS rebinding attack prevention
- **Session Security**: Secure session management with per-session protocol versions

## STDIO Compliance Testing

The validator includes a comprehensive testing suite for STDIO-based MCP servers.

### Running STDIO Tests

```bash
# Run compliance tests for the STDIO server (latest protocol)
python -m mcp_testing.scripts.compliance_report --server-command "python ref_stdio_server/stdio_server_2025_03_26.py" --protocol-version 2025-06-18

# Run with previous protocol versions
python -m mcp_testing.scripts.compliance_report --server-command "python ref_stdio_server/stdio_server_2025_03_26.py" --protocol-version 2025-03-26
```

### STDIO Test Coverage

The STDIO compliance tests verify:
1. Protocol Initialization
2. Tools Functionality
   - Basic tools (echo, add)
   - Async tools (sleep) for 2025-03-26+ versions
   - Structured tool output (2025-06-18)
3. Error Handling
4. Protocol Version Negotiation
5. Advanced Features (2025-06-18)
   - Elicitation support
   - Enhanced validation
   - JSON-RPC batching restrictions

### Testing Different STDIO Server Types

The validator supports testing any STDIO-based MCP server, whether it's run directly from a command or installed via pip. Here's how to test different types of servers:

#### Direct Command Testing

For servers that run directly from a Python file or command:

```bash
# Test a local Python file (latest protocol)
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/your/server.py" --protocol-version 2025-06-18

# Test with specific timeouts
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/server.py" --protocol-version 2025-06-18 --test-timeout 30 --tools-timeout 15

# Focus on tools testing with dynamic discovery
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/server.py" --protocol-version 2025-06-18 --test-mode tools --dynamic-only

# Test with previous protocol versions for compatibility
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/server.py" --protocol-version 2025-03-26
```

#### Testing Pip-Installed Servers

For servers installed via pip (like `mcp-server-fetch`):

```bash
# Ensure you're in the correct virtual environment
source .venv/bin/activate

# Install the server and dependencies
pip install your-mcp-server  # Replace with actual package name

# Run compliance tests
python -m mcp_testing.scripts.compliance_report --server-command "python -m your_server_module" --protocol-version 2024-11-05

# Run tools-only tests
python -m mcp_testing.scripts.compliance_report --server-command "python -m your_server_module" --protocol-version 2024-11-05 --test-mode tools

# example brave search server
BRAVE_API_KEY=api-key python -m mcp_testing.scripts.compliance_report --server-command "npx -y @modelcontextprotocol/server-brave-search" --protocol-version 2024-11-05 
```

#### Test Configuration Options

Common options for both types:

- `--test-mode tools`: Focus on testing tool functionality
- `--dynamic-only`: Automatically discover and test available tools
- `--test-timeout 30`: Set timeout for regular tests (seconds)
- `--tools-timeout 15`: Set timeout for tool-specific tests (seconds)
- `--required-tools tool1,tool2`: Specify required tools to test
- `--skip-tests test1,test2`: Skip specific tests
- `--skip-async`: Skip async tool testing

Note: Tool-related tests that timeout are treated as non-critical, allowing testing to continue.

### Test Reports

Each test run generates a detailed report containing:
- Server information (command, protocol version)
- Test execution timestamp
- Test duration
- Success rate
- Detailed results for each test case
- Server capabilities
- Session information


### Running Tests

You can run different types of tests using module-style commands:

```bash
# Basic interaction test (latest protocol)
python -m mcp_testing.scripts.basic_interaction --server-command "python -m mcp_server_fetch" --protocol-version 2025-06-18

# Compliance tests with tools-only mode
python -m mcp_testing.scripts.compliance_report --server-command "python -m mcp_server_fetch" --protocol-version 2025-06-18 --test-mode tools

# Set custom timeouts for tools tests vs. other tests
python -m mcp_testing.scripts.compliance_report --server-command "python -m mcp_server_fetch" --protocol-version 2025-06-18 --test-timeout 30 --tools-timeout 15

# Test only specific functionality
python -m mcp_testing.scripts.compliance_report --server-command "/path/to/server" --protocol-version 2025-06-18 --test-mode tools

# Skip async tests if experiencing hanging issues
python -m mcp_testing.scripts.compliance_report --server-command "/path/to/server" --protocol-version 2025-06-18 --skip-async

# HTTP debug output
python -m mcp_testing.scripts.http_test --server-url http://localhost:8000/mcp --debug

# Test backward compatibility with older protocols
python -m mcp_testing.scripts.compliance_report --server-command "python -m mcp_server_fetch" --protocol-version 2024-11-05
```

Note: Tool-related tests that timeout are treated as non-critical, allowing testing to continue.

## HTTP Compliance Testing

The validator includes a comprehensive compliance testing suite for HTTP-based MCP servers with full 2025-06-18 protocol support.

### Running HTTP Tests

```bash
# Start the reference HTTP server (runs on port 8088)
python ref_http_server/reference_mcp_server.py

# Run compliance tests and generate a detailed report
python -m mcp_testing.scripts.http_compliance_test --output-dir reports

# Test with OAuth 2.1 authentication enabled
MCP_OAUTH_ENABLED=true python ref_http_server/reference_mcp_server.py --port 8088 &
python -m mcp_testing.scripts.http_compliance_test --server-url http://localhost:8088 --debug
```

### HTTP Test Coverage

The HTTP compliance test suite verifies:

1. **Protocol Initialization**
   - 2025-06-18 protocol negotiation
   - Session management with protocol-specific features
   - MCP-Protocol-Version header handling

2. **Tools Functionality**
   - Structured tool output (2025-06-18)
   - Legacy tool responses (older protocols)
   - Enhanced tool schema validation

3. **Error Handling**
   - Enhanced error responses (2025-06-18)
   - OAuth 2.1 authentication errors
   - Proper HTTP status codes

4. **Batch Request Processing**
   - Batch request restrictions (2025-06-18)
   - Legacy batch support (older protocols)

5. **Session Management**
   - Per-session protocol versions
   - Session security and validation

6. **Protocol Negotiation**
   - Multi-version support (2024-11-05, 2025-03-26, 2025-06-18)
   - Intelligent version selection

7. **Authentication & Security**
   - OAuth 2.1 Bearer token validation
   - WWW-Authenticate header compliance
   - CORS and origin validation

8. **Ping Utility**
   - Connection testing and validation

### Production Deployment

For production deployments with 2025-06-18 protocol:

```bash
# Enable HTTPS (required for OAuth 2.1 in production)
export MCP_TLS_CERT_FILE=/path/to/cert.pem
export MCP_TLS_KEY_FILE=/path/to/key.pem

# Configure OAuth 2.1 authentication
export MCP_OAUTH_ENABLED=true
export MCP_OAUTH_INTROSPECTION_URL=https://auth.example.com/oauth/introspect
export MCP_OAUTH_REQUIRED_SCOPES=mcp:read,mcp:write

# Configure CORS for web clients
export MCP_CORS_ORIGINS=https://myapp.example.com,https://anotherapp.example.com

# Enable rate limiting and security features
export MCP_RATE_LIMIT_ENABLED=true
export MCP_MAX_REQUESTS_PER_MINUTE=100

# Start the server
python ref_http_server/reference_mcp_server.py --port 443 --production
```

## Testing Scripts Overview

The following scripts are available in `mcp_testing/scripts/`:

### Active and Maintained
- `http_compliance_test.py`: Primary script for HTTP server testing with 2025-06-18 support (7/7 tests passing)
- `compliance_report.py`: Primary script for STDIO server testing with 2025-06-18 support (enhanced test coverage)

### Supporting Scripts mixed working/in progress
- `basic_interaction.py`: Simple tool for testing basic server functionality
- `http_test.py`: Lower-level HTTP testing utilities with OAuth 2.1 support
- `http_compliance.py`: Core HTTP compliance testing logic
- `http_compliance_report.py`: Report generation for HTTP tests
- `run_stdio_tests.py`: Lower-level STDIO testing utilities
- `session_test.py`: Session management testing utilities

## 📋 Protocol Version Support

| Protocol Version | Status | Features |
|-----------------|--------|----------|
| 2025-06-18 | ✅ **Full Support** | Structured tool output, OAuth 2.1, elicitation, no batching |
| 2025-03-26 | ✅ **Full Support** | Async tools, enhanced capabilities, batch support |
| 2024-11-05 | ✅ **Full Support** | Basic MCP functionality, legacy compatibility |

## 🧪 Test Results Summary

Current test coverage for 2025-06-18 protocol:

```
✅ HTTP Compliance Test: 7/7 tests passed
✅ OAuth 2.1 Framework: 6/6 tests passed  
✅ Protocol Features: 7/7 tests passed
✅ Multi-Protocol Support: 3/3 versions supported
✅ Backward Compatibility: 100% maintained
✅ Security Features: Authentication, CORS, Origin validation
```

## 📚 Additional Resources

- **JSON Schema**: Complete schemas available in `schema/` directory
- **Specification Documents**: Full specification files in `specification/` directory
- **Reference Implementations**: HTTP and STDIO servers in `ref_http_server/` and `ref_stdio_server/`
- **Test Reports**: Generated reports available in `reports/` directory

## License

SPDX-License-Identifier: AGPL-3.0-or-later
Copyright (c) 2025 Scott Wilcox