AgentSea - Unite and Orchestrate AI Agents

Installation

bash

pnpm add @lov3kaizen/agentsea-evaluate

Key Features

📊

Built-in Metrics

Accuracy, relevance, coherence, toxicity, and more

⚖️

LLM-as-Judge

Rubric-based and comparative scoring

👥

Human Feedback

Ratings, rankings, and corrections

📦

Dataset Management

Create and import datasets with HuggingFace integration

🔄

Continuous Evaluation

Monitor production quality with alerts

🎯

Preference Learning

Generate datasets for RLHF/DPO training

Quick Start

typescript

import {
  EvaluationPipeline,
  AccuracyMetric,
  RelevanceMetric,
  EvalDataset,
} from '@lov3kaizen/agentsea-evaluate';

// Create metrics
const accuracy = new AccuracyMetric({ type: 'fuzzy' });
const relevance = new RelevanceMetric();

// Create evaluation pipeline
const pipeline = new EvaluationPipeline({
  metrics: [accuracy, relevance],
  parallelism: 5,
});

// Create dataset
const dataset = new EvalDataset({
  items: [
    {
      id: '1',
      input: 'What is the capital of France?',
      expectedOutput: 'Paris',
    },
    {
      id: '2',
      input: 'What is 2 + 2?',
      expectedOutput: '4',
    },
  ],
});

// Run evaluation
const results = await pipeline.evaluate({
  dataset,
  generateFn: async (input) => {
    // Your LLM generation function
    return await myAgent.run(input);
  },
});

console.log(results.summary);
// { passRate: 0.95, avgScore: 0.87, ... }

Built-in Metrics

Metric	Description
`AccuracyMetric`	Exact, fuzzy, or semantic match against expected output
`RelevanceMetric`	How relevant the response is to the input
`CoherenceMetric`	Logical flow and consistency of the response
`ToxicityMetric`	Detection of harmful or inappropriate content
`FaithfulnessMetric`	Factual accuracy relative to provided context (RAG)
`ContextRelevanceMetric`	Relevance of retrieved context (RAG)
`FluencyMetric`	Grammar, spelling, and readability
`ConcisenessMetric`	Brevity without losing important information
`HelpfulnessMetric`	How helpful the response is to the user
`SafetyMetric`	Detection of unsafe or harmful outputs

Custom Metrics

typescript

import { BaseMetric, MetricResult, EvaluationInput } from '@lov3kaizen/agentsea-evaluate';

class CustomMetric extends BaseMetric {
  readonly type = 'custom';
  readonly name = 'my-metric';

  async evaluate(input: EvaluationInput): Promise<MetricResult> {
    // Your evaluation logic
    const score = calculateScore(input.output, input.expectedOutput);

    return {
      metric: this.name,
      score,
      explanation: `Score: ${score}`,
    };
  }
}

LLM-as-Judge

Rubric-Based Evaluation

Use LLMs to evaluate responses with custom rubrics:

typescript

import { RubricJudge } from '@lov3kaizen/agentsea-evaluate';

const judge = new RubricJudge({
  provider: anthropicProvider,
  rubric: {
    criteria: 'Response Quality',
    levels: [
      { score: 1, description: 'Poor - Incorrect or irrelevant' },
      { score: 2, description: 'Fair - Partially correct' },
      { score: 3, description: 'Good - Correct but incomplete' },
      { score: 4, description: 'Very Good - Correct and complete' },
      { score: 5, description: 'Excellent - Correct, complete, and well-explained' },
    ],
  },
});

const result = await judge.evaluate({
  input: 'Explain quantum entanglement',
  output: response,
});

Comparative Evaluation

Compare two responses head-to-head:

typescript

import { ComparativeJudge } from '@lov3kaizen/agentsea-evaluate';

const judge = new ComparativeJudge({
  provider: openaiProvider,
  criteria: ['accuracy', 'helpfulness', 'clarity'],
});

const result = await judge.compare({
  input: 'Summarize this article',
  responseA: modelAOutput,
  responseB: modelBOutput,
});
// { winner: 'A', reasoning: '...', criteriaScores: {...} }

Human Feedback

Rating Collector

Collect ratings from human annotators:

typescript

import { RatingCollector } from '@lov3kaizen/agentsea-evaluate/feedback';

const collector = new RatingCollector({
  scale: 5,
  criteria: ['accuracy', 'helpfulness', 'clarity'],
});

// Collect feedback
await collector.collect({
  itemId: 'response-123',
  input: 'What is ML?',
  output: 'Machine Learning is...',
  annotatorId: 'user-1',
  ratings: {
    accuracy: 4,
    helpfulness: 5,
    clarity: 4,
  },
  comment: 'Good explanation',
});

// Get aggregated scores
const stats = collector.getStatistics('response-123');

Preference Collection

Collect A/B preferences for RLHF/DPO training:

typescript

import { PreferenceCollector } from '@lov3kaizen/agentsea-evaluate/feedback';

const collector = new PreferenceCollector();

// Collect A/B preferences
await collector.collect({
  input: 'Explain recursion',
  responseA: '...',
  responseB: '...',
  preference: 'A',
  annotatorId: 'user-1',
  reason: 'More concise explanation',
});

// Export for RLHF/DPO training
const dataset = collector.exportForDPO();

Datasets

Create Dataset

typescript

import { EvalDataset } from '@lov3kaizen/agentsea-evaluate/datasets';

const dataset = new EvalDataset({
  name: 'qa-benchmark',
  items: [
    {
      id: '1',
      input: 'Question 1',
      expectedOutput: 'Answer 1',
      context: ['Relevant context...'],
      tags: ['factual', 'science'],
    },
  ],
});

// Filter and sample
const subset = dataset
  .filter(item => item.tags?.includes('science'))
  .sample(100);

// Split for train/test
const [train, test] = dataset.split(0.8);

HuggingFace Integration

Import from the Hub

EvalDataset.fromHuggingFace loads rows from the public HuggingFace datasets-server REST API — no SDK dependency is required. Rows are mapped onto eval items using the configured field names.

typescript

import { EvalDataset } from '@lov3kaizen/agentsea-evaluate';

const dataset = await EvalDataset.fromHuggingFace('squad', {
  split: 'validation', // defaults to 'train'
  subset: 'default', // datasets-server config / subset, defaults to 'default'
  inputField: 'question', // defaults to 'input'
  outputField: 'answers', // defaults to 'output'
  contextField: 'context', // optional, maps to item.context[]
  limit: 1000, // defaults to 100
});

console.log(dataset.size);
console.log(dataset.getItems()[0]);

To read gated or private datasets, set the HF_TOKEN environment variable. It is sent as a bearer token on the datasets-server request.

Export to the Hub

DatasetExporter pushes a preference dataset to the HuggingFace Hub as DPO-format JSONL, generating a dataset card (README) alongside the data. Hub upload uses the optional @huggingface/hub package and requires a write-scoped token.

typescript

import { DatasetExporter } from '@lov3kaizen/agentsea-evaluate/datasets';

const exporter = new DatasetExporter();

const result = await exporter.exportPreferences(preferenceDataset, {
  format: 'huggingface',
  formatOptions: {
    name: 'my-org/preference-dataset', // target Hub repo
    token: process.env.HF_TOKEN, // write-scoped HF token
    private: true,
    license: 'mit',
    tags: ['preference', 'dpo', 'rlhf'],
  },
});

console.log(result.url);
// https://huggingface.co/datasets/my-org/preference-dataset

name and token may also be supplied at the top level as repoName and token on the export options.

Continuous Evaluation

Monitor production quality with automated evaluation pipelines:

typescript

import { ContinuousEvaluator } from '@lov3kaizen/agentsea-evaluate/continuous';

const evaluator = new ContinuousEvaluator({
  metrics: [accuracy, relevance, toxicity],
  sampleRate: 0.1, // Evaluate 10% of requests
  alertThresholds: {
    accuracy: 0.8,
    toxicity: 0.1,
  },
});

// Set up alerts
evaluator.on('alert', (alert) => {
  console.error(`Quality alert: ${alert.metric} below threshold`);
  notifyOncall(alert);
});

// Log production interactions
await evaluator.log({
  input: userQuery,
  output: agentResponse,
  expectedOutput: groundTruth, // Optional
});

Alert Channels

The AlertManager delivers quality-degradation alerts over multiple channels. Define rules per metric, then attach one or more channels. Webhook, Slack, email, and PagerDuty are all supported.

typescript

import { AlertManager } from '@lov3kaizen/agentsea-evaluate/continuous';

const alerts = new AlertManager({
  rules: {
    accuracy: { metric: 'accuracy', threshold: 0.8, direction: 'below', severity: 'critical' },
    toxicity: { metric: 'toxicity', threshold: 0.1, direction: 'above', severity: 'warning' },
  },
  cooldownMs: 300_000, // suppress repeats for 5 minutes (default)
  channels: [
    { type: 'slack', webhook: 'https://hooks.slack.com/services/...' },
    { type: 'webhook', webhook: 'https://example.com/alerts' },
  ],
});

// Feed metric values; a breached rule fires the channels.
alerts.check('accuracy', 0.74);

Email (SMTP)

The email channel sends alerts over SMTP using the optional nodemailer package (install it to enable this channel). It requires to recipients and an smtp.host; the from address defaults to the SMTP auth user.

typescript

const channel = {
  type: 'email',
  to: ['oncall@example.com', 'quality@example.com'],
  from: 'alerts@example.com', // optional, defaults to smtp.auth.user
  smtp: {
    host: 'smtp.example.com',
    port: 587, // defaults to 587
    secure: false, // defaults to false
    auth: { user: 'apikey', pass: process.env.SMTP_PASSWORD },
  },
};

PagerDuty

The pagerduty channel triggers an incident through the PagerDuty Events API v2 enqueue endpoint. Provide an integration routingKey (it falls back to apiKey when not set). Repeated alerts for the same metric are de-duplicated into a single incident.

typescript

const channel = {
  type: 'pagerduty',
  routingKey: process.env.PAGERDUTY_ROUTING_KEY, // Events API v2 integration key
};

Alert severity (info, warning, critical) comes from the matching rule and is mapped to the PagerDuty severity on each event.

API Reference

EvaluationPipeline

typescript

interface EvaluationPipelineConfig {
  metrics: MetricInterface[];
  llmJudge?: JudgeInterface;
  parallelism?: number;
  timeout?: number;
  retries?: number;
}

// Methods
pipeline.evaluate(options: PipelineEvaluationOptions): Promise<PipelineEvaluationResult>

EvalDataset

typescript

interface EvalDatasetItem {
  id: string;
  input: string;
  expectedOutput?: string;
  context?: string[];
  reference?: string;
  metadata?: Record<string, unknown>;
  tags?: string[];
}

// Methods
dataset.getItems(): EvalDatasetItem[]
dataset.filter(predicate): EvalDataset
dataset.sample(count): EvalDataset
dataset.split(ratio): [EvalDataset, EvalDataset]

PipelineEvaluationResult

typescript

interface PipelineEvaluationResult {
  results: SingleEvaluationResult[];
  metrics: MetricsSummary;
  failures: FailureAnalysis[];
  summary: EvaluationSummary;
  exportJSON(): string;
  exportCSV(): string;
}

LLM Evaluation

Installation

Key Features

Built-in Metrics

LLM-as-Judge

Human Feedback

Dataset Management

Continuous Evaluation

Preference Learning

Quick Start

Built-in Metrics

Custom Metrics

LLM-as-Judge

Rubric-Based Evaluation

Comparative Evaluation

Human Feedback

Rating Collector

Preference Collection

Datasets

Create Dataset

HuggingFace Integration

Import from the Hub

Export to the Hub

Continuous Evaluation

Alert Channels

Email (SMTP)

PagerDuty

API Reference

EvaluationPipeline

EvalDataset

PipelineEvaluationResult

Next Steps