Background Job System

NATS-backed workers for long-running tasks with real-time progress reporting.

The Problem

AI agents that can only respond synchronously are fundamentally limited. Research tasks, document summarization, multi-step analysis, infrastructure checks -- these take time. If the agent blocks on every operation, the user waits. If it times out, work is lost. You need asynchronous execution with reliable delivery, progress visibility, and the ability to run multiple jobs in parallel.

How Baker Street Solves It

Baker Street uses NATS JetStream as a durable job queue connecting the Brain to a pool of stateless Worker pods. When the agent decides a task needs background processing, it publishes a job message to NATS. Any available Worker picks it up from the shared queue, and load balancing happens automatically.

Three job types cover different workloads:

Agent jobs give Claude its own reasoning loop on the Worker. Used for tasks that require thinking: research, summarization, drafting, analysis.
Command jobs execute shell commands from a strict allowlist. Used for quick infrastructure queries: kubectl get pods, df -h, uptime.
HTTP jobs make API calls from inside the cluster. Used for service health checks, webhook triggers, and data fetching.

Workers report real-time status updates (received, running, completed, failed) back through NATS pub/sub, so the Brain always knows the state of every job. If a Worker crashes mid-job, NATS automatically redelivers the message to another Worker. Multiple Workers can run in parallel, scaling throughput horizontally.

Example

// Brain dispatches a background research job
await nats.publish("jobs.dispatch", {
  id: "job-abc-123",
  type: "agent",
  payload: {
    prompt: "Research the latest Kubernetes 1.32 features and summarize the key changes",
    model: "claude-sonnet-4-20250514",
  },
  replyTo: "jobs.status.job-abc-123",
});

// Worker picks up the job and reports progress
await nats.publish("jobs.status.job-abc-123", {
  id: "job-abc-123",
  status: "running",
  progress: "Analyzing Kubernetes changelog...",
});

Learn More

See the Workers documentation for details on job types, concurrency configuration, and the command allowlist.

Read the documentation