AgentOps

Agents

Mon, 01 Jan 0001 00:00:00 +0000

An Agent is the central CRD in AgentOps. It declares everything the operator needs to run an AI agent as a Kubernetes workload: which model to use, what tools are available, how memory works, who can delegate to it, and what safety guardrails to enforce.

Agent CRD spec

The Agent CRD (agents.agents.agentops.io, short name ag) carries the full agent definition. Here is a representative example:

apiVersion: agents.agentops.io/v1alpha1
kind: Agent
metadata:
 name: platform-engineer
 namespace: agents
spec:
 # ── Lifecycle ──
 mode: daemon  # daemon (always-on) or task (one-shot)

 # ── Model ──
 model: anthropic/claude-sonnet-4-20250514
 fallbackModels:
 - openai/gpt-4o
 titleModel: openai/gpt-4o-mini  # cheap model for auto-titling sessions
 providerRefs:
 - name: anthropic
 - name: openai

 # ── Identity ──
 systemPrompt: |
 You are a senior platform engineer specializing in Kubernetes,
 GitOps, and infrastructure automation.
 contextFiles:
 - configMapRef:
 name: platform-context
 key: AGENTS.md

 # ── Built-in tools ──
 builtinTools:
 - bash
 - read
 - edit
 - write
 - grep
 - glob
 - fetch

 # ── External tools (AgentTool CRs) ──
 tools:
 - name: kubectl
 permissions:
 mode: deny
 rules: ["kubectl_delete", "kubectl_scale"]
 - name: flux
 - name: git

 # ── Memory ──
 memory:
 serverRef: agentops-memory  # service name or AgentTool CR
 project: platform-engineer
 contextLimit: 5 # observations injected per turn
 autoSummarize: true
 autoSave: true
 autoSearch: true

 # ── Discovery & delegation ──
 discovery:
 description: "Kubernetes and GitOps specialist. Handles cluster operations, Helm releases, Flux reconciliation, and infrastructure debugging."
 tags: ["kubernetes", "gitops", "infrastructure"]
 scope: namespace  # namespace | explicit | hidden
 # allowedCallers: [...] # only used with scope: explicit

 # ── Safety ──
 toolHooks:
 blockedCommands:
 - "rm -rf /"
 - "kubectl delete namespace"
 allowedPaths:
 - "/workspace"
 - "/tmp"
 maxSteps: 100 # safety limit on agent loop iterations
 timeout: "10m" # per-prompt timeout

 # ── Infrastructure ──
 image: ghcr.io/samyn92/agent-runtime-fantasy:latest
 resources:
 requests:
 memory: "256Mi"
 cpu: "100m"
 limits:
 memory: "1Gi"
 storage:
 size: "10Gi"
 env:
 LOG_LEVEL: "info"
 secrets:
 - name: GITHUB_TOKEN
 secretRef:
 name: agent-tokens
 key: github-token

Spec reference

Field	Type	Default	Description
`mode`	`daemon` / `task`	required	Lifecycle mode. Daemon = Deployment, task = Job template.
`model`	string	required	Primary model in `provider/model` format.
`fallbackModels`	[]string	`[]`	Tried in order if the primary model fails.
`titleModel`	string	primary model	Cheap model for auto-titling daemon sessions.
`providers`	–	–	Removed. Use `providerRefs` to reference Provider CRs.
`providerRefs`	[]ProviderBinding	required (min 1)	References to Provider CRs with optional per-agent call default overrides.
`systemPrompt`	string	`""`	Injected at the start of every session.
`contextFiles`	[]ContextFileRef	`[]`	ConfigMap-backed context files (e.g. AGENTS.md).
`builtinTools`	[]string	all defaults	Built-in Fantasy tools: `bash`, `read`, `edit`, `write`, `grep`, `ls`, `glob`, `fetch`. Set to `[]` to disable all.
`tools`	[]AgentToolBinding	`[]`	References to AgentTool CRs with optional permission overrides.
`permissionTools`	[]string	`[]`	Tools requiring user approval before execution.
`enableQuestionTool`	bool	`false`	Allow the agent to ask interactive questions.
`memory`	MemorySpec	nil	Memory configuration (see Memory).
`discovery`	DiscoverySpec	nil	Controls delegation visibility and access.
`toolHooks`	ToolHooksSpec	nil	Blocked commands, allowed paths, audit tools, memory save rules.
`maxSteps`	int	`100`	Maximum agent loop iterations (prevents infinite loops).
`timeout`	string	`"10m"`	Per-prompt timeout for daemons, job timeout for tasks.
`image`	string	`ghcr.io/samyn92/agent-runtime-fantasy:latest`	Runtime container image.
`resources`	ResourceRequirements	nil	CPU/memory requests and limits.
`storage`	StorageSpec	nil	PVC config for daemon agents (ignored for tasks).
`concurrency`	ConcurrencySpec	nil	Controls parallel AgentRun execution.
`networkPolicy`	NetworkPolicySpec	nil	Whether to create a NetworkPolicy for this agent.
`schedule`	string	`""`	Cron expression for periodic AgentRuns.
`schedulePrompt`	string	`""`	Prompt used when schedule triggers a run.
`resourceBindings`	[]AgentResourceBinding	`[]`	External resources (repos, buckets) bound to this agent.

Lifecycle modes

Daemon mode

A daemon agent is a long-running Deployment with a Service and optional PVC. It stays up, accepts prompts via HTTP POST on port 4096, and maintains a conversation session with working memory.

Building Custom MCP Tools

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks through creating a custom MCP tool server from scratch, packaging it, and deploying it into an AgentOps-managed agent.

Prerequisites

Go 1.22+
agent-tools CLI installed
Access to an OCI registry (e.g. ghcr.io)
A running AgentOps cluster with the operator installed

1. Create a Go Module

mkdir my-tool && cd my-tool
go mod init github.com/myorg/my-tool
go get github.com/modelcontextprotocol/go-sdk@v0.8.0

2. Implement the Tool Server

Create main.go:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/modelcontextprotocol/go-sdk/mcp"
)

func main() {
	server := mcp.NewServer(
		&mcp.Implementation{
			Name: "my-tool",
			Version: "0.1.0",
		},
		nil,
	)

	// Check mode for gating dangerous operations
	mode := os.Getenv("MODE") // "readonly" or "readwrite"

	// Register a read-safe tool (always available)
	mcp.AddTool(server, &mcp.Tool{
		Name: "lookup_item",
		Description: "Look up an item by ID.",
		InputSchema: json.RawMessage(`{
			"type": "object",
			"properties": {
				"id": {"type": "string", "description": "The item ID to look up."}
			},
			"required": ["id"]
		}`),
	}, func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
		id := req.Arguments["id"].(string)
		// Your lookup logic here
		return &mcp.CallToolResult{
			Content: []mcp.Content{
				{Type: "text", Text: fmt.Sprintf("Found item: %s", id)},
			},
		}, nil
	})

	// Register a write tool (only in readwrite mode)
	if mode != "readonly" {
		mcp.AddTool(server, &mcp.Tool{
			Name: "create_item",
			Description: "Create a new item.",
			InputSchema: json.RawMessage(`{
				"type": "object",
				"properties": {
					"name": {"type": "string", "description": "Item name."},
					"value": {"type": "string", "description": "Item value."}
				},
				"required": ["name", "value"]
			}`),
		}, func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
			name := req.Arguments["name"].(string)
			value := req.Arguments["value"].(string)
			// Your creation logic here
			return &mcp.CallToolResult{
				Content: []mcp.Content{
					{Type: "text", Text: fmt.Sprintf("Created item %s=%s", name, value)},
				},
			}, nil
		})
	}

	if err := server.Run(context.Background(), &mcp.StdioTransport{}); err != nil {
		fmt.Fprintf(os.Stderr, "server error: %v\n", err)
		os.Exit(1)
	}
}

Mode Gating Pattern

The MODE environment variable controls which tools are exposed. This is set per-agent in the AgentTool CR or the Agent CR’s toolRefs config:

CRD Reference

Mon, 01 Jan 0001 00:00:00 +0000

All CRDs belong to the API group agents.agentops.io/v1alpha1. The operator installs them automatically via the Helm chart.

kubectl get crds | grep agentops

agents.agents.agentops.io
agentruns.agents.agentops.io
agenttools.agents.agentops.io
agentresources.agents.agentops.io
channels.agents.agentops.io
providers.agents.agentops.io

Agent

Defines an AI agent workload. The mode field determines the lifecycle: daemon creates a Deployment + PVC + Service (always running), task creates a Job template (one prompt, exits).

apiVersion: agents.agentops.io/v1alpha1
kind: Agent
metadata:
 name: my-agent
 namespace: agents

Short name: ag

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

This guide gets you from zero to a running AI agent on Kubernetes in three steps.

Prerequisites

Kubernetes cluster v1.28+ (k3s, kind, EKS, GKE, AKS all work)
Helm 3.12+
kubectl configured for your cluster
An LLM API key (Anthropic, OpenAI, or Google)

Step 1: Install the platform

helm install agentops oci://ghcr.io/samyn92/charts/agentops-platform \
 --namespace agent-system --create-namespace

This deploys the operator, console, memory service, and Tempo into agent-system, and creates the agents namespace for workloads.

Repositories

Mon, 01 Jan 0001 00:00:00 +0000

AgentOps is composed of eight repositories, each with a focused responsibility. All container images are published to ghcr.io/samyn92/ and Helm charts to ghcr.io/samyn92/charts/.

agentops

Purpose: Central documentation hub (this site).

Hugo site with the Docsy theme
Architecture docs, CRD reference, guides, and project information
No runtime artifacts

Repository: github.com/samyn92/agentops

agentops-core

Purpose: Kubernetes operator that manages the full lifecycle of AI agent workloads.

6 CRDs: Agent, AgentRun, AgentTool, AgentResource, Channel, Provider
Reconciles Agent CRs into Deployments (daemon) or Job templates (task) with sidecars, storage, networking, and MCP tool injection
Handles delegation orchestration, concurrency control, and schedule-based runs
Built with kubebuilder

Key tech: Go, kubebuilder, controller-runtime Image: ghcr.io/samyn92/agentops-operator Current version: v0.8.1 (~70 commits) Repository: github.com/samyn92/agentops-core

Installation

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers installation from prerequisites through custom configuration, verification, and uninstalling.

Prerequisites

Requirement	Minimum version
Kubernetes	1.28+
Helm	3.12+
kubectl	1.28+
Container runtime	containerd, CRI-O
Storage	Default StorageClass with dynamic provisioning (for memory + Tempo PVCs)

Required secrets (created in the agents namespace):

At least one LLM API key (Anthropic, OpenAI, or Google) in a Kubernetes Secret

Full install (all components)

The default install deploys the operator, console, memory service, and Tempo:

Memory

Mon, 01 Jan 0001 00:00:00 +0000

Memory is the key differentiator of the AgentOps platform. Instead of treating each agent session as stateless, AgentOps implements a three-layer memory architecture that gives agents persistent, searchable, relevance-ranked context across sessions and restarts.

The system is split between the Fantasy runtime (in-process working memory) and agentops-memory (persistent short-term and long-term storage). No LLM calls are involved in memory management — summarization is deterministic, retrieval is BM25-ranked full-text search.

Three-layer model

Layer 1: Working memory (ephemeral)

Working memory is an unbounded list of fantasy.Message objects held in Go runtime memory. It represents the current conversation — every user prompt, assistant response, tool call, and tool result.

Memory API

Mon, 01 Jan 0001 00:00:00 +0000

The agentops-memory service provides a REST API for managing sessions, observations, search, and context injection. It runs as a standalone Go binary backed by SQLite with FTS5 for BM25 relevance-ranked full-text search.

Image: ghcr.io/samyn92/agentops-memory Default port: 7437 In-cluster DNS: agentops-memory.agents.svc.cluster.local:7437

Input validation

All endpoints enforce the following limits:

Constraint	Limit
Request body size	1 MiB
Title length	500 characters
Content length	50,000 characters
Topic key length	200 characters
Scope length	50 characters
Type length	50 characters
Tag length	100 characters per tag
Tag count	20 tags per observation
Project length	200 characters
Query result limit	1,000 rows maximum

Health

GET /health

Health check endpoint.

Multi-Agent Orchestration

Mon, 01 Jan 0001 00:00:00 +0000

AgentOps supports multi-agent orchestration natively. An orchestrator agent discovers and delegates to task agents using built-in runtime tools — no custom wiring required.

This guide walks through setting up an orchestrator + task agent pattern.

How Delegation Works

The delegation flow is:

The orchestrator agent calls list_task_agents (built-in) to discover available task agents in its namespace.
It decides which agents to delegate to based on their descriptions and tags.
It calls run_agents (built-in) with target agent names and input.
The runtime creates AgentRun CRs for each delegated task.
The DelegationWatcher tracks progress via Kubernetes Watch on those AgentRun resources.
Results are collected and returned to the orchestrator when all delegated runs complete.

The orchestrator does not need any toolRefs for delegation — list_task_agents and run_agents are built into the Fantasy runtime. However, the orchestrator can also have its own tools for direct work.

Providers

Mon, 01 Jan 0001 00:00:00 +0000

A Provider extracts LLM provider configuration from individual Agent CRs into a shared, reusable Kubernetes resource. Instead of repeating provider type, credentials, and endpoint settings across every agent, you define them once in a Provider CR and reference it from any number of agents via spec.providerRefs.

Provider CRD spec

The Provider CRD (providers.agents.agentops.io, short name prov) maps directly to the Fantasy SDK’s provider surface. Here is a representative example:

apiVersion: agents.agentops.io/v1alpha1
kind: Provider
metadata:
 name: anthropic
 namespace: agents
spec:
 # ── Backend ──
 type: anthropic  # Fantasy SDK provider backend

 # ── Credentials ──
 apiKeySecret:
 name: llm-api-keys
 key: ANTHROPIC_API_KEY

 # ── Per-call defaults (optional) ──
 defaults:
 anthropic:
 effort: high  # extended thinking effort
 thinkingBudgetTokens: 8192

Spec reference

Field	Type	Required	Default	Description
`type`	`anthropic` \| `openai` \| `google` \| `azure` \| `bedrock` \| `openrouter` \| `openaicompat`	Yes	–	Fantasy SDK backend type.
`apiKeySecret`	SecretKeyRef	Yes	–	Secret containing the API key. Validated on reconcile.
`endpoint`	ProviderEndpoint	No	–	API endpoint overrides (base URL, custom headers).
`config`	ProviderConfig	No	–	Type-specific configuration (org, project, Vertex, Bedrock, Responses API).
`defaults`	ProviderCallDefaults	No	–	Per-call options applied to all agents using this provider.

See the CRD Reference for the full field-level specification.

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

This roadmap covers the major features planned for AgentOps. Items are roughly ordered by priority. Timelines are not committed – this is an independent project and development happens as capacity allows.

AgentSkill CRD

Curated knowledge injected into agent system prompts. Static expertise (runbooks, procedures, domain knowledge) complementing agentops-memory’s dynamic experience (lessons learned, decisions).

Sources:

Git sync – pull markdown from a Git repository, auto-update on push
OCI – skill packages as OCI artifacts, versioned and cached
Inline – small markdown blocks directly in the CR (prototyping)
ConfigMap – mount existing ConfigMaps as skill content

Injection modes:

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

AgentOps separates the platform control plane from user agent workloads using a two-namespace model. The operator manages the full lifecycle of agents, tools, resources, and channels as native Kubernetes workloads.

Component overview

Two-namespace model

Namespace	Purpose	Contents
`agent-system`	Platform control plane	Operator, console, Tempo
`agents`	User workloads	Agent pods, memory service, AgentTool/AgentResource/Channel workloads, PVCs

This separation provides:

RBAC isolation — the operator has broad permissions in agents but users interact only through CRDs
Resource quota boundaries — platform overhead is independent of agent workload limits
Clear ownership — platform upgrades don’t disrupt running agents

Custom Resource Definitions

AgentOps defines 6 CRDs in the agents.agentops.io/v1alpha1 API group:

Tool Catalog

Mon, 01 Jan 0001 00:00:00 +0000

AgentOps ships six MCP tool servers as OCI artifacts. Each server is a compiled Go binary implementing MCP stdio transport. Agents reference them via AgentTool CRs, and the operator pulls the binary at pod startup via a crane init container.

All tool servers are published to ghcr.io/samyn92/agent-tools/.

kube-explore

Intent-based Kubernetes exploration. Higher-level than raw kubectl – the agent describes what it wants to understand, and the tool returns structured, relevant information.

Tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools extend what an agent can do beyond generating text. AgentOps uses the Model Context Protocol (MCP) for tool integration. MCP tools are compiled Go binaries that communicate over stdio — no HTTP servers, no network hops for tool calls within the pod.

The AgentTool CRD (agents.agents.agentops.io/agenttools, short name agtool) defines the tool catalog. The operator provisions tools based on their source type, and the Fantasy runtime spawns them at startup.

Agent Delegation and Orchestration

Mon, 01 Jan 0001 00:00:00 +0000

Agent delegation lets one agent spawn work on other agents and collect structured results — without polling, without consuming LLM tokens while waiting, and with full trace propagation across the delegation chain.

Discovery

Before an agent can delegate, it needs to know what’s available. The built-in list_task_agents tool queries the Kubernetes API for Agent CRs in the same namespace and returns a filtered list based on:

Visibility scope — namespace (default, visible to all), explicit (visible only to allowedCallers), or hidden (never listed).
Tags — categorization labels like kubernetes, security, frontend that agents use to find the right specialist.
Allowed callers — when scope is explicit, only agents named in allowedCallers can discover and delegate.

The runtime knows its own identity (AGENT_NAME env var) and uses it to filter the discovery response.

Helm Values Reference

Mon, 01 Jan 0001 00:00:00 +0000

The agentops-platform chart is an umbrella chart that deploys all AgentOps platform components. This page documents every configurable value.

Global

Key	Type	Default	Description
`global.imagePullSecrets`	list	`[]`	Image pull secrets applied to all sub-charts. Each entry is an object with a `name` field.
`agentNamespace`	string	`agents`	Namespace where agent workloads are deployed.
`createNamespace`	bool	`true`	Whether to create the agent namespace if it does not exist.

global:
 imagePullSecrets:
 - name: ghcr-secret
agentNamespace: agents
createNamespace: true

agentops-operator

The Kubernetes operator that reconciles Agent, AgentTool, and related CRDs.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Every component in AgentOps emits OpenTelemetry traces. The runtime follows the GenAI semantic conventions for LLM-specific attributes. The console proxies traces from Tempo and enriches them with delegation metadata from AgentRun CRDs.

Runtime tracing

The agent runtime produces rich OTEL spans for every operation in the agent loop:

Span hierarchy

GenAI semantic convention attributes

Every LLM span carries standard attributes:

Attribute	Example
`gen_ai.operation.name`	`chat`, `invoke_agent`
`gen_ai.provider.name`	`anthropic`, `openai`, `google`
`gen_ai.request.model`	`claude-sonnet-4-20250514`
`gen_ai.response.model`	`claude-sonnet-4-20250514`
`gen_ai.usage.input_tokens`	`12847`
`gen_ai.usage.output_tokens`	`3291`
`gen_ai.usage.reasoning_tokens`	`1024`
`gen_ai.usage.cache_read_tokens`	`8192`
`gen_ai.request.temperature`	`0.3`
`gen_ai.response.finish_reasons`	`stop`

Tool call events

Tool executions are recorded as tool.call events on the gen_ai.generate span. Each event includes:

The Web Console

Mon, 01 Jan 0001 00:00:00 +0000

The AgentOps console is a progressive web app backed by a Go BFF (Backend-for-Frontend). It connects to agent runtimes via the Fantasy Event Protocol (FEP) over Server-Sent Events for real-time streaming of responses, tool calls, delegation events, and interactive controls.

Architecture

Tech stack

Layer	Technology
Backend	Go, chi v5 router, controller-runtime informers for CRD access
Frontend	SolidJS 1.9, Vite 7, Tailwind 4
Transport	REST (60+ endpoints) + SSE (FEP multiplexer)
PWA	Workbox service worker, offline-capable static assets, NetworkFirst API caching

API surface

The BFF exposes 60+ endpoints under /api/v1, organized by domain:

Agent Resources

Mon, 01 Jan 0001 00:00:00 +0000

An AgentResource is a Kubernetes CRD that represents an external resource agents can work with. It provides a declarative catalog entry for things like GitHub repositories, GitLab projects, S3 buckets, and documentation — along with the credentials needed to access them.

Resources are separate from agents by design. You declare a resource once, then bind it to any number of agents.

Why a separate CRD?

Without AgentResource, every agent that needs access to a repository would duplicate the owner, repo name, branch, and credential reference in its own spec. AgentResource centralizes that: