LLM Eval Harness Builder

Name: OpenModels
Creator: OpenModels
License: https://github.com/openmodelsrun/openmodels

AvanzadadataContexto mínimo: 32K

Designs evaluation harnesses for LLM applications, covering dataset construction, task-specific metrics, LLM-as-judge rubrics with bias controls, and regression gates. Helps teams measure quality, catch regressions across model or prompt changes, and report results with confidence intervals rather than vibes.

llm evaluation benchmarking llm-as-judge quality regression-testing

Casos de uso

Building an eval set and metrics for a RAG assistant
Designing an LLM-as-judge rubric with bias mitigations
Adding a regression gate for prompt and model changes in CI
Reporting eval results with statistical confidence

Prompt de ejemplo

We have a support chatbot backed by an LLM. Design an evaluation harness: propose an eval
dataset structure, define metrics for helpfulness/faithfulness/safety, write an LLM-as-judge
rubric that controls for position and verbosity bias, and outline a CI regression gate.

Modelos recomendados

claude-opus-4-8 gpt-5-6-sol gemini-3-5-pro deepseek-v4

Herramientas compatibles

claude-codecursorkiroany

Modalidades

Entrada: text, code

→

Salida: text, code

Skills relacionadas

rag-pipeline-builder prompt-engineering synthetic-data-generator

Autor

OpenModels Community

@openmodelsrun