Ignite UI MCP Testbed: An Open-Source Bench for Measuring What Your AI Agent’s Tooling Actually Does

AI coding agents are only as good as the context you give them. Hand an agent a Model Context Protocol (MCP) server that looks up live component docs, or a set of Agent Skills that spell out your framework’s conventions, and it should write better code. But “should” isn’t a benchmark. Does the Ignite UI Theming MCP server actually reduce hallucinated APIs? Do skills help more on Angular than on React? Is Sonnet worth it over a free hosted model for your particular prompts?

Until now, answering those questions meant running the agent by hand, over and over, eyeballing the output, and hoping you remembered what you changed between runs. This tool turns that guesswork into a repeatable experiment.

---

The core idea: matrix executions

The heart of the testbed is a simple but powerful loop. You take one shared prompt and run it across a grid of combinations:

• Platforms: Angular, React, Blazor, and Web Components out of the box.
• Variants: a specific bundle of tooling – which MCP servers are enabled, plus a four-way skills choice (none, generated skills, local skills, or both overlaid) – so you can pit your draft skills directly against the shipped ones.
• Models: anything opencode can drive – Anthropic, OpenAI, a local OpenAI-compatible endpoint (Ollama, LM Studio), or even keyless free hosted models.

Every instance in that grid becomes its own isolated agent run, executed one at a time – deliberately sequential, so a big matrix stays within your provider’s rate limits and each run gets the container to itself. The testbed scaffolds a fresh project, hands the agent the prompt, and builds the resulting app.

Then it screenshots every route the agent generated, runs your verification tests against it, and lays every run next to each other so you can see, concretely, what changed.

Want to know whether skills matter? Run “with skills” versus “without” across all four frameworks and read the difference off a single grid. That’s the whole pitch: stop guessing whether your tooling helps, and start measuring it.

---

Built on open source, isolated by design

The testbed drives opencode, an open-source AI coding agent, and exercises the Ignite UI AI toolchain through it: two MCP servers (live component docs / API lookup and theming queries) plus the Agent Skills the agent loads automatically.

The only thing you need on your machine is rootless Podman. Everything else – Node, .NET, the Ignite UI CLIs, opencode, and a headless Chromium – is baked into a single container image. Every session spins up its own fresh, ephemeral container, so nothing leaks between runs and every result starts from a clean slate.

Your model API key is handled with the same care: you type it into the wizard, it’s passed to the agent as an environment variable, and it is never written to disk inside the container or echoed back to the browser.

Getting started is two commands:

./run.sh build      # build the image (once)
./run.sh            # launch a fresh session at http://localhost:8080

On Windows, the equivalent PowerShell scripts (run.ps1 / stop.ps1) handle the path and flag differences for you.

---

The four views

The web wizard organizes everything into four modes.

Configuration

Before you run anything, the Configuration view lets you manage provider packs: JSON files that teach the testbed how to scaffold and configure a library – its scaffold and dev-server commands, its MCP servers, and where its skills come from. Ignite UI ships supported out of the box, but packs are how the testbed reaches beyond Ignite UI. Drop in a pack and its frameworks show up as new platforms everywhere else in the tool.

And the configuration file for Angular Material looks like this:

{
  "name": "angular-material",
  "displayName": "Angular Material",
  "description": "Angular scaffolded by the Angular CLI with @angular/material installed - the agent does the Material wiring (theme, typography, components).",
  "version": "22.0.0",
  "frameworks": [
    {
      "id": "angular-material",
      "label": "Angular Material",
      "scaffold": {
        "cmd": "npx",
        "argv": ["-y", "@angular/cli", "new", "{{name}}", "--defaults", "--style", "scss", "--skip-git"],
        "cwdIsParent": true
      },
      "install": ["@angular/material", "@angular/cdk"],
      "dev": {
        "cmd": "npx",
        "argv": ["ng", "serve", "--host", "0.0.0.0", "--port", "{{port}}", "--poll", "1500"]
      }
    }
  ],
  "configure": {
    "mcpServers": [
      {
        "name": "angular-cli",
        "command": "npx",
        "args": ["-y", "@angular/cli", "mcp"],
        "class": "angular",
        "label": "Angular CLI MCP",
        "description": "Angular CLI's built-in MCP server - docs, best practices, project queries"
      }
    ]
  },
  "containerDeps": { "npmGlobal": ["@angular/cli"] }
}

Interactive

The default view scaffolds a single project, wires up your chosen MCPs, skills, and model, and hands you off to a live opencode web session with streaming token and cost stats. This is the mode for exploring: poking at a prompt, watching the agent work, switching models mid-session to feel the difference.

Matrix

This is where the comparison happens. You pick your platforms, define your variants, choose a model, and write one prompt. The testbed then runs every combination headless: scaffold, run the agent, build, screenshot every route, run the verification tests, and drop the results into History. It’s the fastest way to answer “with skills vs. without across all four frameworks” in a single submission.

History

Every run, interactive or matrix, lands in a sortable grid showing its configuration, per-stage timings, token and cost stats, screenshots, and logs. Rows expand into full detail, you can rate each run 1-5 stars, and the whole thing exports to Excel. Crucially, History persists in ./sessions/history/ on the host, so it survives across containers: your results don’t vanish when a session ends.

The grid rendering History isn’t a stand-in. It’s Ignite UI’s own commercial Data Grid (sortable, filterable, Excel-exportable), the same component you’d ship in your own app. Using it here means History doubles as a working example of the grid under a real workload: expandable rows, live sorting, and export against actual benchmark data, not a demo dataset. By default it builds as a watermarked trial. Set IG_NPM_TOKEN (plus IG_NPM_USERNAME / IG_NPM_EMAIL) in .env to build the licensed grid instead – see the README’s “Licensed grid” section for the three-line setup.

---

Verification tests: quality gates, not vibes

Screenshots tell you what the agent produced. Verification tests tell you whether it works.

The testbed runs your own Playwright end-to-end tests against each generated app as a post-generation quality gate. You author plain specs (no node_modules, no config, no package.json) and drop them under ./tests/. The container provides @playwright/test and a headless Chromium, and each spec runs against the served app.

Tests are organized as a shared set that runs for every platform, plus optional per-framework overlays:

tests/
  shared/          # runs for every platform
    smoke.spec.ts
  angular/         # runs only for Angular entries (plus shared/)
    grid.spec.ts
  react/
  webcomponents/
  blazor/

A spec looks exactly like the Playwright you already know:

import { test, expect } from '@playwright/test';

test('home page renders a grid', async ({ page }) => {
  await page.goto('/');
  await expect(page.locator('igc-grid, igx-grid, .igr-grid')).toBeVisible();
});

When a spec fails, that run is marked test-failed in History, distinct from a clean success or a build-error, and the pass/fail counts show right in the grid. A grouped picker lets you choose exactly which specs run per framework, so you can dial verification up or down per experiment.

---

Made for CI, not just the browser

Everything the Matrix view collects can also live in a JSON config file, so a full matrix can run without ever opening the UI:

./run.sh --matrix-config ./matrix.json            # run it
./run.sh --matrix-config ./matrix.json --validate  # just check the config

// matrix.json
{
  "name": "grid+chart baseline",
  "platforms": ["angular", "react"],
  "variants": [
    { "mcps": ["igniteui", "theming"], "skills": true, "localSkills": false },
    { "mcps": [], "skills": false, "localSkills": false }
  ],
  "model": "anthropic/claude-sonnet-4-5",
  "prompt": "Build a dashboard page with a data grid showing sample sales data and a bar chart summarizing it.",
  "apiKeyEnv": "ANTHROPIC_API_KEY",
  "selectedTests": ["angular::shared/smoke.spec.ts", "react::shared/smoke.spec.ts"],
  "autoRun": true,
  "exitOnDone": false
}

Progress mirrors to the terminal with per-entry pass/fail outcomes, so a config-driven run is followable from a log stream. When the matrix settles, two artifacts land on the host: a self-contained report.html (summary table, stage timings, token/cost usage, test results, and embedded screenshots, openable straight from the filesystem) and a machine-readable summary.json that a CI job can read to see which combination regressed.

For pipelines, exitOnDone gives you meaningful exit codes: 0 if every entry succeeded, 2 if everything built but some verification tests failed, and 1 for anything worse. That’s enough to gate a merge on “did the agent, with this tooling, still produce a working app?”

---

Extend it to your own stack

The testbed ships tuned for Ignite UI, but it isn’t locked to it. Provider packs let you register any library (its scaffold commands, its MCP servers, its skills) as a first-class platform. A matrix config can even carry a pack inline, making a terminal run fully self-contained in a single file.

The repository includes a complete worked example: an Angular Material provider defined inline, scaffolded with the Angular CLI, with @angular/material and @angular/cdk installed post-scaffold, and then the agent is left to do the Material wiring, which is the thing being tested. It runs one prompt with and without the Angular CLI MCP, so you can see the tool’s effect directly. If it works for Angular Material, it can work for your component library too.

---

Get started with the Ignite UI MCP Testbed

It’s open source and free for everyone. Whether you’re evaluating whether MCP servers earn their keep, tuning skills for a specific framework, comparing models against your real prompts, or wiring an agent-quality check into CI, it gives you a repeatable, isolated, side-by-side bench to do it on.

git clone https://github.com/IgniteUI/igniteui-mcp-testbed
cd igniteui-mcp-testbed
./run.sh build      # build the image (once)
./run.sh            # launch a fresh session at http://localhost:8080

Clone the repository, build the image, and run your first matrix. Then let the grid tell you what your tooling is actually worth.

The Ignite UI MCP Testbed is open source and available now. Contributions, provider packs, and feedback are welcome.