Annotation Format Schema v1.0

A portable format for structured UI feedback

Overview

The Annotation Format Schema (AFS) is an open format created and used by Agentation for capturing UI feedback in a way that AI coding agents can reliably parse and act on. Think of it like smart Figma comments for your running app — persistent annotations attached to specific elements, with threads, status tracking, resolution workflows, and structured metadata that agents can actually understand.

This spec defines the annotation object shape. Tools can emit annotations in this format, and agents can consume them regardless of how they were created.

Human

Reviewer

Annotation

Object

Agent

Claude

→pending
→acknowledged
→resolved

What This Unlocks

A structured schema isn't just about clean data — it enables entirely new workflows:

Two-way communication — Agents can reply to annotations, asking “Should this be 24px or 16px?” and get responses in the same thread
Status tracking — See what's pending, acknowledged, resolved, or dismissed at a glance
Cross-page queries — “What annotations do I have?” works across your entire site
Bulk operations — “Clear all annotations” or “Show me blocking issues only”
Persistent history — Feedback survives page refreshes and browser sessions

Without a schema, feedback is fire-and-forget. With one, it becomes a conversation.

Design Goals

Agent-readable — Structured data that LLMs can parse without guessing
Framework-agnostic — Works with any UI, though React gets extra context
Tool-agnostic — Any tool can emit, any agent can consume
Human-authored — Designed for feedback from humans (or automated reviewers)
Minimal core — Few required fields, many optional for richer context

Annotation Object

An annotation represents a single piece of feedback attached to a UI element.

Note: The server may add metadata fields (sessionId, createdAt, updatedAt) when syncing annotations.

Required Fields

{
  id: string;           // Unique identifier (e.g. "ann_abc123")
  comment: string;      // Human feedback ("Button is misaligned")
  elementPath: string;  // CSS selector path ("body > main > button.cta")
  timestamp: number;    // Unix timestamp (ms)
  x: number;            // % of viewport width (0-100)
  y: number;            // px from document top (or viewport if isFixed)
  element: string;      // Tag name ("button", "div", "input")
}

Recommended Fields

{
  url: string;          // Page URL where annotation was created
  boundingBox: {        // Element position at annotation time
    x: number;
    y: number;
    width: number;
    height: number;
  };
}

Optional Context Fields

{
  // React-specific (when available)
  reactComponents: string;  // Component tree ("App > Dashboard > Button")
  sourceFile: string;       // Source path + line ("src/Button.tsx:42:5")

  // Element details
  cssClasses: string;       // Class list ("btn btn-primary disabled")
  computedStyles: string;   // Key CSS properties
  accessibility: string;    // ARIA attributes, role
  nearbyText: string;       // Visible text in/around element
  selectedText: string;     // Text highlighted by user

  // Feedback classification
  intent: "fix" | "change" | "question" | "approve";
  severity: "blocking" | "important" | "suggestion";
}

Lifecycle Fields

{
  sessionId: string;        // Server session this annotation belongs to
  status: "pending" | "acknowledged" | "resolved" | "dismissed";
  thread: ThreadMessage[];  // Back-and-forth conversation
  createdAt: number;        // Unix timestamp (ms)
  updatedAt: number;        // Unix timestamp (ms)
  resolvedAt: number;       // Unix timestamp (ms)
  resolvedBy: "human" | "agent";
  authorId: string;         // Optional actor identity
}

Browser Component Fields

These optional fields are set by the Agentation browser component for UI rendering:

{
  isFixed: boolean;         // Element has fixed/sticky positioning
  isMultiSelect: boolean;   // Created via drag selection
  fullPath: string;         // Full DOM path (vs shorter elementPath)
  nearbyElements: string;   // Info about nearby DOM elements
  elementBoundingBoxes: Array<{ x: number; y: number; width: number; height: number }>;
}

Full TypeScript Definition

type Annotation = {
  // Core fields
  id: string;
  x: number;
  y: number;
  comment: string;
  element: string;
  elementPath: string;
  timestamp: number;

  // Optional element metadata
  selectedText?: string;
  boundingBox?: { x: number; y: number; width: number; height: number };
  nearbyText?: string;
  cssClasses?: string;
  nearbyElements?: string;
  computedStyles?: string;
  fullPath?: string;
  accessibility?: string;
  isMultiSelect?: boolean;
  isFixed?: boolean;
  reactComponents?: string;
  sourceFile?: string;
  elementBoundingBoxes?: Array<{ x: number; y: number; width: number; height: number }>;

  // Server-backed fields
  sessionId?: string;
  url?: string;
  intent?: "fix" | "change" | "question" | "approve";
  severity?: "blocking" | "important" | "suggestion";
  status?: "pending" | "acknowledged" | "resolved" | "dismissed";
  thread?: ThreadMessage[];
  createdAt?: number;
  updatedAt?: number;
  resolvedAt?: number;
  resolvedBy?: "human" | "agent";
  authorId?: string;
};

type ThreadMessage = {
  id: string;
  role: "human" | "agent";
  content: string;
  timestamp: number;
};

Event Envelope

For real-time streaming, annotations are wrapped in an event envelope:

type AgentationEvent = {
  type: "annotation.created" | "annotation.updated" | "annotation.deleted"
      | "session.created" | "session.updated" | "session.closed"
      | "thread.message" | "action.requested";
  timestamp: number;     // Unix milliseconds
  sessionId: string;
  sequence: number;      // Monotonic for persisted ordering/replay; bootstrap sync snapshots may use 0
  payload: Annotation | Session | ThreadMessage | ActionRequest;
};

type ActionRequest = {
  sessionId: string;
  annotations: Annotation[];
  output: string;
  timestamp: number;
};

The sequence number enables clients to detect missed persisted events and request replay. Bootstrap sync snapshots may use 0 before live events begin. See the server docs for SSE streaming details.

JSON Schema

For validation in any language:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://agentation.dev/schema/annotation.v1.json",
  "title": "Annotation",
  "type": "object",
  "required": ["id", "comment", "elementPath", "timestamp", "x", "y", "element"],
  "properties": {
    "id": { "type": "string" },
    "comment": { "type": "string" },
    "elementPath": { "type": "string" },
    "timestamp": { "type": "number" },
    "x": { "type": "number", "description": "% of viewport width (0-100)" },
    "y": { "type": "number", "description": "px from document top or viewport for fixed elements" },
    "element": { "type": "string" },
    "sessionId": { "type": "string" },
    "url": { "type": "string", "format": "uri" },
    "boundingBox": {
      "type": "object",
      "properties": {
        "x": { "type": "number" },
        "y": { "type": "number" },
        "width": { "type": "number" },
        "height": { "type": "number" }
      },
      "required": ["x", "y", "width", "height"]
    },
    "reactComponents": { "type": "string" },
    "sourceFile": { "type": "string" },
    "cssClasses": { "type": "string" },
    "computedStyles": { "type": "string" },
    "accessibility": { "type": "string" },
    "nearbyText": { "type": "string" },
    "selectedText": { "type": "string" },
    "isFixed": { "type": "boolean" },
    "isMultiSelect": { "type": "boolean" },
    "fullPath": { "type": "string" },
    "nearbyElements": { "type": "string" },
    "elementBoundingBoxes": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "x": { "type": "number" },
          "y": { "type": "number" },
          "width": { "type": "number" },
          "height": { "type": "number" }
        },
        "required": ["x", "y", "width", "height"]
      }
    },
    "intent": { "enum": ["fix", "change", "question", "approve"] },
    "severity": { "enum": ["blocking", "important", "suggestion"] },
    "status": { "enum": ["pending", "acknowledged", "resolved", "dismissed"] },
    "createdAt": { "type": "number" },
    "updatedAt": { "type": "number" },
    "resolvedAt": { "type": "number" },
    "resolvedBy": { "enum": ["human", "agent"] },
    "authorId": { "type": "string" }
  }
}

Example Annotation

{
  "id": "ann_k8x2m",
  "comment": "Button is cut off on mobile viewport",
  "elementPath": "body > main > .hero-section > button.cta",
  "timestamp": 1705694400000,
  "x": 45.5,
  "y": 480,
  "element": "button",
  "url": "http://localhost:3000/landing",
  "boundingBox": { "x": 120, "y": 480, "width": 200, "height": 48 },
  "reactComponents": "App > LandingPage > HeroSection > CTAButton",
  "cssClasses": "cta btn-primary",
  "nearbyText": "Get Started Free",
  "intent": "fix",
  "severity": "blocking",
  "status": "pending"
}

Markdown Output Format

For pasting into chat-based agents, annotations can be serialized as markdown:

## Annotation #1
**Element:** button.cta
**Path:** body > main > .hero-section > button.cta
**React:** App > LandingPage > HeroSection > CTAButton
**Position:** 120px, 480px (200×48px)
**Feedback:** Button is cut off on mobile viewport
**Severity:** blocking

See Output Formats for detail level options (Compact → Forensic).

Implementations

Tools that emit or consume this format:

Agentation (React)	Click-to-annotate toolbar for React apps
Agentation CLI / HTTP server	Exposes annotations over HTTP, SSE, and CLI workflows for coding agents

Building an Implementation

To emit Agentation Format annotations from your tool:

Capture the required fields: id, comment, elementPath, timestamp, x, y, element
Add recommended fields for better agent accuracy: url, boundingBox
For React apps, traverse the fiber tree to get reactComponents
Output as JSON for HTTP API / CLI consumption, or markdown for chat pasting

See the Agentation source for reference implementations of element detection and React component traversal.

Why This Format?

Existing agent protocols (MCP, A2A, ACP) standardize tools and messaging, but they don't define a UI feedback grammar. They rely on whatever structured context you feed them.

This format fills that gap: a portable wire format specifically for "human points at UI, agent needs to find and fix the code." We hope it's useful to others building similar tools.

Versioning

Current version: v1

Schema URL: https://agentation.dev/schema/annotation.v1.json