v1.1.0 release - Contributors, Sponsors and Enquiries are most welcome 😌

Browser Automation (Surf)

Computer-use agent for controlling desktop environments. Screen capture, mouse, keyboard actions using Claude's vision capabilities.

Surf enables AI agents to interact with desktop applications and browsers through 8 computer-use tools with Claude vision integration.

Installation

bash
pnpm add @lov3kaizen/agentsea-surf

# Optional dependencies
pnpm add puppeteer   # Browser automation (default engine)
pnpm add playwright  # Browser automation (chromium/firefox/webkit)
pnpm add sharp       # Image processing
pnpm add rfb2        # VNC remote-display backend

Key Features

🖥️

8 Computer-Use Tools

Screenshot, click, type, scroll, drag, key press, cursor move, wait

🔌

Multiple Backends

Native, Puppeteer, Playwright, Docker, Kubernetes, VNC, RDP

👁️

Claude Vision

Automatic screen analysis and action determination

🔄

Streaming

Real-time event streaming during execution

🛡️

Security Sandbox

Rate limiting, command blocking, domain restrictions

🎯

NestJS Integration

REST API and WebSocket support

Computer-Use Tools

📷

Screenshot

🖱️

Click

⌨️

Type

📜

Scroll

🎯

Drag

⏎

Key Press

➡️

Cursor Move

⏱️

Wait

Quick Start

typescript
import {
  SurfAgent,
  createNativeBackend,
} from '@lov3kaizen/agentsea-surf';

async function main() {
  // Create a native backend for your platform
  const backend = createNativeBackend();
  await backend.connect();

  // Create the agent
  const agent = new SurfAgent('session-1', backend, {
    maxSteps: 20,
    vision: {
      model: 'claude-sonnet-4-20250514',
      maxTokens: 4096,
      includeScreenshotInResponse: true,
    },
  });

  // Execute a task
  const result = await agent.execute(
    'Open Chrome and navigate to google.com'
  );

  console.log('Result:', result.response);
  console.log('Steps taken:', result.state.actionHistory.length);

  await backend.disconnect();
}

main().catch(console.error);

Backends

Surf ships with backends for native desktops, browsers (Puppeteer or Playwright), Docker, Kubernetes, and remote displays (VNC/RDP). Each can be constructed directly, or selected by type through the createBackend(config) factory:

typescript
import { createBackend } from '@lov3kaizen/agentsea-surf';

// createBackend returns a Promise<DesktopBackend>
const backend = await createBackend({
  type: 'browser',
  options: { engine: 'playwright', browserType: 'chromium' },
});

await backend.connect();

Supported type values: native, browser, docker, kubernetes, vnc, and rdp.

Native Backend

Automatically selects the appropriate backend for your platform (macOS, Linux, Windows):

typescript
import { createNativeBackend } from '@lov3kaizen/agentsea-surf';

const backend = createNativeBackend({ displayIndex: 0 });
await backend.connect();

Browser Backend (Puppeteer or Playwright)

Automate web browsers with Puppeteer (the default engine) or Playwright:

typescript
import { PuppeteerBackend } from '@lov3kaizen/agentsea-surf';

const backend = new PuppeteerBackend({
  headless: false,
  viewport: { width: 1920, height: 1080 },
  initialUrl: 'https://example.com',
});

await backend.connect();

Playwright adds Firefox and WebKit support via browserType. Select it through createBackend by setting engine: 'playwright' (the default engine remains 'puppeteer'):

typescript
import { createBackend } from '@lov3kaizen/agentsea-surf';

const backend = await createBackend({
  type: 'browser',
  options: {
    engine: 'playwright',
    browserType: 'webkit', // 'chromium' | 'firefox' | 'webkit'
    headless: true,
    viewport: { width: 1920, height: 1080 },
    initialUrl: 'https://example.com',
  },
});

await backend.connect();

Docker Backend

Run in an isolated Docker container:

typescript
import { DockerBackend } from '@lov3kaizen/agentsea-surf';

const backend = new DockerBackend({
  image: 'agentsea/desktop:ubuntu-22.04',
  resolution: { width: 1920, height: 1080, scaleFactor: 1 },
  removeOnDisconnect: true,
});

await backend.connect();

Kubernetes Backend

Run a desktop pod in a Kubernetes cluster, driven via kubectl exec. Requires kubectl on PATH and an image that ships an X server plus xdotool/scrot (the same desktop image used for Docker works):

typescript
import { createBackend } from '@lov3kaizen/agentsea-surf';

const backend = await createBackend({
  type: 'kubernetes',
  options: {
    namespace: 'agentsea',
    image: 'agentsea/desktop:ubuntu-22.04',
    resolution: { width: 1920, height: 1080, scaleFactor: 1 },
    resources: {
      requests: { cpu: '500m', memory: '1Gi' },
      limits: { cpu: '2', memory: '4Gi' },
    },
    deleteOnDisconnect: true,
  },
});

await backend.connect();

Remote-Display Backends (VNC / RDP)

Drive an existing remote desktop over the VNC (RFB) or RDP protocol. Both translate Surf actions into pointer/key events:

typescript
import { createBackend } from '@lov3kaizen/agentsea-surf';

// VNC
const vnc = await createBackend({
  type: 'vnc',
  options: { host: '127.0.0.1', port: 5900, password: 'secret' },
});

// RDP
const rdp = await createBackend({
  type: 'rdp',
  options: {
    host: '127.0.0.1',
    port: 3389,
    username: 'user',
    password: 'secret',
    domain: 'WORKGROUP',
  },
});

await vnc.connect();
Pointer/key/input translation is fully implemented and unit-tested, but frame capture (screenshots) and the RDP transport are experimental. The optional rfb2 (VNC) and RDP client packages cannot assemble framebuffers into PNGs on their own. For production, inject a custom RemoteDisplayClient via the backend's { client } dependency.
typescript
import {
  VNCBackend,
  type RemoteDisplayClient,
} from '@lov3kaizen/agentsea-surf';

// Inject your own client for reliable frame capture + transport
const client: RemoteDisplayClient = createMyRemoteDisplayClient();

const backend = new VNCBackend(
  { host: '127.0.0.1', port: 5900 },
  { client },
);

await backend.connect();

Streaming Execution

Stream events as the agent executes:

typescript
const agent = new SurfAgent('session-1', backend, config);

for await (const event of agent.executeStream('Search for weather')) {
  switch (event.type) {
    case 'screenshot':
      console.log('Screenshot taken');
      break;
    case 'action':
      console.log(`Executing: ${event.action.description}`);
      break;
    case 'complete':
      console.log('Task completed:', event.response);
      break;
  }
}

Using Individual Tools

Use tools independently for fine-grained control:

typescript
import {
  createSurfTools,
  createNativeBackend,
} from '@lov3kaizen/agentsea-surf';

const backend = createNativeBackend();
await backend.connect();

const tools = createSurfTools(backend);

// Use individual tools
await tools.screenshot.execute({});
await tools.click.execute({ x: 100, y: 200 });
await tools.typeText.execute({ text: 'Hello World' });
await tools.scroll.execute({ direction: 'down', amount: 200 });
await tools.keyPress.execute({ key: 'Enter' });
await tools.drag.execute({ startX: 0, startY: 0, endX: 100, endY: 100 });
await tools.cursorMove.execute({ x: 500, y: 300 });
await tools.wait.execute({ duration: 1000 });

Security Sandboxing

Restrict agent capabilities for safety:

typescript
const agent = new SurfAgent('session', backend, {
  sandbox: {
    enabled: true,
    maxActionsPerMinute: 60,
    blockedDomains: ['malicious-site.com'],
    blockedCommands: ['rm -rf', 'sudo'],
    blockedPaths: ['/etc', '/root'],
  },
});

NestJS Integration

typescript
import { Module } from '@nestjs/common';
import { SurfModule } from '@lov3kaizen/agentsea-surf/nestjs';

@Module({
  imports: [
    SurfModule.forRoot({
      backend: { type: 'native' },
      config: {
        maxSteps: 50,
        sandbox: { enabled: true },
      },
      enableRestApi: true,
      enableWebSocket: true,
    }),
  ],
})
export class AppModule {}

REST API Endpoints

When using NestJS integration:

MethodEndpointDescription
POST/surf/executeExecute a task
POST/surf/actionExecute single action
POST/surf/screenshotTake a screenshot
GET/surf/screenGet screen state
GET/surf/sessionsList active sessions
GET/surf/statusGet backend status

WebSocket Events

EventDescription
executeStart task execution (emits stream, complete, error)
actionExecute single action (emits actionResult)
screenshotTake screenshot (emits screenshotResult)
stopStop current execution
statusGet backend status

API Reference

SurfAgent

typescript
// Constructor
new SurfAgent(
  sessionId: string,
  backend: DesktopBackend,
  config?: Partial<SurfConfig>
)

// Methods
agent.execute(task: string, context?: AgentContext)  // Execute a task
agent.executeStream(task: string, context?: AgentContext)  // Execute with streaming
agent.stop()  // Stop the current execution
agent.getState()  // Get current agent state

SurfConfig

typescript
interface SurfConfig {
  maxSteps: number;  // Maximum steps before stopping
  vision: {
    model: string;  // Claude model to use
    maxTokens: number;
    includeScreenshotInResponse: boolean;
  };
  sandbox?: {
    enabled: boolean;
    maxActionsPerMinute?: number;
    blockedDomains?: string[];
    blockedCommands?: string[];
    blockedPaths?: string[];
  };
}

Use Cases

  • Web Automation - Fill forms, navigate sites, scrape data
  • Desktop Automation - Control native applications
  • Testing - Automated UI testing with AI
  • Data Entry - Automate repetitive data entry tasks
  • Research - Gather information from multiple sources

Next Steps