Vivid Sydney · Builder's Field Guide 2026 Edition · Open-Source Edition
Open Source Edition

How AI Agents
Are Built

Open-source(ish) Edition.

The five components behind every production agent, the two key architectures, and a stack you can mostly self-host in 2026 — MIT or Apache-2.0 licensed unless flagged otherwise.

AChatbot vs Agent

A chatbot waits for input and replies. An agent is given a goal.

  • Decides its own next action with an LLM, step by step
  • Plans multi-step work and self-corrects mid-run
  • Calls tools, reads memory, acts on external systems
The test: remove the human. If work still moves forward, it's an agent.

BTwo Architectures

Single agent: one model, many tools. It's an OK stack for most v1 builds.

Manager pattern: one agent decomposes the goal, delegates to specialists
Peer network: agents coordinate directly, no top-down control
Go multi-agent only when tasks demand parallelism or deep specialisation.

Five Components of Every Agent

MODEL → MEMORY → TOOLS → INSTRUCTIONS → ORCHESTRATION
01
Model
The reasoning core

Picks the next action, plans the sequence, recovers from its own errors. Optimise for task accuracy first, then cost and latency.

GPT-5.6 Claude Fable 5 GLM-5.3 Qwen-3.8 Max Kimi-K3
02
Memory
State that survives the session

Short-term: the context of the current run. Long-term: user preferences and past decisions, usually in a vector store. MongoDB Atlas and Redis are both swapped below — one for licensing, one for a cleaner permissive fork.

QdrantApache-2.0 MilvusApache-2.0 pgvectorPostgreSQL ValkeyBSD-3
03
Tools
Hands on the real world

APIs, databases, email, CRMs. A model without tools can only talk about the work. Tools let it do the work. Zapier and Make dropped — both closed, metered SaaS with no self-host path.

ActivepiecesMIT n8nfair-code WindmillAGPLv3 Hugging FaceApache-2.0
04
Instructions
The operating contract

Your SOPs rewritten as prompts: tight scope, explicit edge cases, defined fallback paths for every failure mode.

system prompts few-shot examples fallback paths
05
Orchestration
The conductor

Wires model, memory and tools into one loop. Routes work between agents when the system grows past one. CrewAI is MIT-licensed and actively maintained, so it stays. AutoGen is out — Microsoft put it in maintenance mode in October 2025 and pointed everyone to its successor.

LangChainMIT LangGraphMIT CrewAIMIT MS Agent FrameworkMIT

CGuardrails

Boundaries are a feature, not a constraint.

  • PII: detect and mask personal data before it hits a tool call or leaves your network
  • Content safety: block harmful or off-policy output
  • Dynamic rules: permissions adapt to user role and task risk
PresidioMIT NeMo GuardrailsApache-2.0 Guardrails AIApache-2.0

DHuman in the Loop

Autonomy is earned, never assumed.

  • Error thresholds: repeated failures escalate to a person
  • High-stakes gates: refunds, deletions, payments need sign-off
  • Full traces: log every step so failures are auditable
LangfuseMIT Arize PhoenixApache-2.0
Field note · does an agent stack need a dedicated PII layer?

Yes — treat PII detection as its own guardrail, not a line item under "content safety"

Perplexity's PII-TRACE research (1 Sep 2026) tested 12 detectors on long, multi-turn conversations and found the real failure mode isn't missing PII entirely — it's missing the second or third mention of the same identifier later in a run. That's exactly the shape of an agent workflow: a customer's email or phone number gets picked up in turn one, then re-quoted across several tool calls and memory writes. A generic safety filter checks each message in isolation and lets the repeats through.

69.1%of identifiers mentioned 6–10 times were still fully caught by a dedicated local detector (PII-Tracer) vs 4.5% for a general frontier model doing the same job
0.6Bparameters — small enough to run locally as a pre-flight gate before data crosses a trust boundary
13languages tested; recurring-identifier detection held at 80–93% consistency across all of them

The practical open-source version of that gate today is Microsoft Presidio (MIT) — it's the same tool Perplexity's own pipeline uses to rescan synthetic data for leaked PII, and it already runs the identical "screen before it leaves the boundary" pattern locally. Wire it in as a pre-processing step ahead of every tool call and memory write, not just at the chat input.

The 2026 Agent Stack

PURPOSE → TOOLS → LICENSE
Frontier models
GPT-5.6 Claude Fable 5 GLM-5.3 Qwen-3.8 Max Kimi-K3
Vector memory
QdrantApache-2.0 MilvusApache-2.0 pgvectorPostgreSQL ValkeyBSD-3
Orchestration
LangChainMIT LangGraphMIT CrewAIMIT MS Agent FrameworkMIT
Integration & actions
ActivepiecesMIT n8nfair-code WindmillAGPLv3 Hugging FaceApache-2.0
Guardrails & PII
PresidioMIT NeMo GuardrailsApache-2.0 Guardrails AIApache-2.0
Observability
LangfuseMIT Arize PhoenixApache-2.0
MIT / Apache-2.0 / BSD-3 / PostgreSQL — OSI-approved, unrestricted self-host and commercial use fair-code / AGPLv3 — source-available or copyleft; check terms before reselling as a hosted service