Introduction
In this tutorial, you'll learn how to implement and use Anthropic's new plugin evaluation framework for Claude Code. This system allows developers to evaluate the effectiveness of plugins by running them against realistic prompts and comparing results with a baseline (no-plugin) scenario. You'll build a plugin evaluation pipeline that uses six different grader types to assess plugin performance.
Prerequisites
- Basic understanding of Python and command-line interfaces
- Anthropic Claude Code API access
- Installed
anthropicPython package - Basic familiarity with plugin development concepts
Step-by-Step Instructions
1. Setting Up Your Plugin Evaluation Environment
1.1 Install Required Dependencies
First, create a virtual environment and install the necessary packages:
python -m venv plugin_eval_env
source plugin_eval_env/bin/activate # On Windows: plugin_eval_env\Scripts\activate
pip install anthropic
Why: This creates an isolated environment to prevent package conflicts and ensures you have the Anthropic SDK for interacting with Claude Code.
1.2 Create Project Structure
Create the following directory structure for your evaluation project:
plugin_evals/
├── evals/
│ ├── __init__.py
│ ├── grader_types.py
│ └── eval_runner.py
├── plugins/
│ ├── __init__.py
│ └── sample_plugin.py
├── prompts/
│ └── test_prompts.json
└── main.py
Why: This structure organizes your evaluation components, making it easy to maintain and scale your plugin evaluation system.
2. Implementing the Grader Types
2.1 Define Your Grader Types
Create evals/grader_types.py with six different grader implementations:
import json
from typing import Dict, Any
class GraderBase:
def grade(self, response: str, expected: str) -> Dict[str, Any]:
raise NotImplementedError
class ExactMatchGrader(GraderBase):
def grade(self, response: str, expected: str) -> Dict[str, Any]:
return {
"pass": response.strip() == expected.strip(),
"score": 1.0 if response.strip() == expected.strip() else 0.0,
"type": "exact_match"
}
class FuzzyMatchGrader(GraderBase):
def grade(self, response: str, expected: str) -> Dict[str, Any]:
# Simple fuzzy matching using string similarity
from difflib import SequenceMatcher
similarity = SequenceMatcher(None, response.strip(), expected.strip()).ratio()
return {
"pass": similarity > 0.8,
"score": similarity,
"type": "fuzzy_match"
}
class ToolUseGrader(GraderBase):
def grade(self, response: str, expected: str) -> Dict[str, Any]:
# Check if response contains expected tool usage
tool_used = expected.lower() in response.lower()
return {
"pass": tool_used,
"score": 1.0 if tool_used else 0.0,
"type": "tool_use"
}
class CodeGenerationGrader(GraderBase):
def grade(self, response: str, expected: str) -> Dict[str, Any]:
# Check if response contains valid code
has_code = "def " in response or "class " in response
return {
"pass": has_code,
"score": 1.0 if has_code else 0.0,
"type": "code_generation"
}
class JSONOutputGrader(GraderBase):
def grade(self, response: str, expected: str) -> Dict[str, Any]:
try:
parsed_response = json.loads(response)
parsed_expected = json.loads(expected)
matches = parsed_response == parsed_expected
return {
"pass": matches,
"score": 1.0 if matches else 0.0,
"type": "json_output"
}
except json.JSONDecodeError:
return {
"pass": False,
"score": 0.0,
"type": "json_output"
}
class MultiStepGrader(GraderBase):
def grade(self, response: str, expected: str) -> Dict[str, Any]:
# Check for multiple steps in response
steps = response.split('\n')
has_multiple_steps = len(steps) > 1
return {
"pass": has_multiple_steps,
"score": 1.0 if has_multiple_steps else 0.0,
"type": "multi_step"
}
Why: These six grader types cover different aspects of plugin performance - exact matching, fuzzy matching, tool usage, code generation, JSON output validation, and multi-step processing. This comprehensive approach helps measure plugin effectiveness from multiple angles.
3. Creating a Plugin Runner
3.1 Implement the Plugin Runner
Create evals/eval_runner.py:
import os
from anthropic import Anthropic
from grader_types import GraderBase
class PluginEvalRunner:
def __init__(self, api_key: str):
self.client = Anthropic(api_key=api_key)
self.graders = {
"exact_match": ExactMatchGrader(),
"fuzzy_match": FuzzyMatchGrader(),
"tool_use": ToolUseGrader(),
"code_generation": CodeGenerationGrader(),
"json_output": JSONOutputGrader(),
"multi_step": MultiStepGrader()
}
def run_plugin_eval(self, prompt: str, plugin_name: str, expected_output: str, use_plugin: bool = True) -> dict:
# Construct the prompt with or without plugin instructions
if use_plugin:
full_prompt = f"{prompt} [Use the {plugin_name} plugin]"
else:
full_prompt = prompt
# Call Claude Code API
response = self.client.messages.create(
model="claude-3-haiku-2026-09-11",
max_tokens=1000,
messages=[
{"role": "user", "content": full_prompt}
]
)
# Get Claude's response
actual_output = response.content[0].text
# Grade the response using all grader types
results = {}
for grader_name, grader in self.graders.items():
results[grader_name] = grader.grade(actual_output, expected_output)
return {
"prompt": prompt,
"plugin_used": use_plugin,
"plugin_name": plugin_name,
"actual_output": actual_output,
"expected_output": expected_output,
"results": results
}
Why: This runner handles the core evaluation logic, including calling the Claude API, processing responses, and applying all six grader types to measure plugin performance comprehensively.
4. Setting Up Test Prompts
4.1 Create Test Prompts
Create prompts/test_prompts.json:
[{
"id": "prompt_1",
"text": "Write a Python function to calculate the factorial of a number",
"expected": "def factorial(n):\n if n == 0:\n return 1\n else:\n return n * factorial(n-1)"
}, {
"id": "prompt_2",
"text": "Create a JSON structure for a user profile",
"expected": "{\n \"name\": \"John Doe\",\n \"age\": 30\n}"
}, {
"id": "prompt_3",
"text": "Explain how to use the requests library to make HTTP GET requests",
"expected": "import requests\nresponse = requests.get('https://api.example.com/data')"
}]
Why: These test prompts represent realistic use cases for plugin evaluation, covering different programming tasks and data formats to test various plugin capabilities.
5. Running Plugin Evaluations
5.1 Create Main Evaluation Script
Create main.py:
import json
from evals.eval_runner import PluginEvalRunner
def load_prompts(file_path: str):
with open(file_path, 'r') as f:
return json.load(f)
def main():
# Initialize the runner
runner = PluginEvalRunner(api_key=os.getenv('ANTHROPIC_API_KEY'))
# Load test prompts
prompts = load_prompts('prompts/test_prompts.json')
# Run evaluations for each prompt
for prompt_data in prompts:
print(f"\nEvaluating prompt: {prompt_data['text']}")
# Run with plugin
result_with_plugin = runner.run_plugin_eval(
prompt=prompt_data['text'],
plugin_name="code_generator",
expected_output=prompt_data['expected'],
use_plugin=True
)
# Run without plugin (baseline)
result_without_plugin = runner.run_plugin_eval(
prompt=prompt_data['text'],
plugin_name="code_generator",
expected_output=prompt_data['expected'],
use_plugin=False
)
# Compare results
print("\nWith Plugin:", result_with_plugin['results']['exact_match']['pass'])
print("Without Plugin:", result_without_plugin['results']['exact_match']['pass'])
# Print detailed comparison
print("\nDetailed Results:")
for grader_type, result in result_with_plugin['results'].items():
if result['pass']:
print(f" {grader_type}: PASS (Score: {result['score']})")
else:
print(f" {grader_type}: FAIL (Score: {result['score']})")
if __name__ == "__main__":
main()
Why: This script orchestrates the entire evaluation process, running each prompt with and without the plugin to establish a baseline for comparison.
6. Running Your Evaluation
6.1 Set Environment Variables
Set your Anthropic API key:
export ANTHROPIC_API_KEY='your-api-key-here'
Why: The API key is required to access Claude Code's capabilities for your plugin evaluations.
6.2 Execute the Evaluation
python main.py
Why: This runs your complete evaluation pipeline, comparing plugin performance against the baseline to determine effectiveness.
Summary
In this tutorial, you've built a comprehensive plugin evaluation system for Claude Code that implements six different grader types to measure plugin effectiveness. The system runs prompts both with and without plugins, creating a baseline comparison that answers critical questions about plugin performance. This framework provides developers with a robust way to measure and improve their plugin capabilities, ensuring that plugins deliver real value to users.
