Using AI Agents to the Extreme — How I Accidentally Built a Cognitive Architecture

I didn’t set out to build a cognitive architecture. I set out to stop losing context every time Claude Code’s memory evaporated.

One thing led to another. I wrote a session recovery command. Then a handoff protocol. Then a delegation framework with weighted scope checks. Then cognitive trap detection. Then a virtual feature crew of four specialist agents. Then, today, a parallel orchestration skill that spawns 2-4 engineers across separate repo clones and collects their results in a status table.

At some point I stopped configuring an AI tool and started building a brain.

The moment it clicked

A researcher agent I’d spawned came back with a comparison I didn’t ask for. It had mapped my setup — all the markdown files, the slash commands, the subagent personas — against formal agent frameworks. LangGraph. CrewAI. AutoGen. The Claude Agent SDK.

The mapping was uncomfortably precise:

And the overall shape — Perception → Memory (short + long term) → Planning → Action → Reflection — that’s textbook cognitive architecture. Not metaphorically. Literally the components, wired together, doing the things.

The kicker: LESSONS.md. Most agent frameworks do reflection within a session — the agent tries something, it fails, it adjusts. My setup persists corrections across sessions. When Claude makes a mistake and I correct it, that correction gets logged with root cause analysis and applies forever. The frameworks haven’t caught up to that yet.

I built this by accident. Each piece solved a specific friction point. I never looked at a cognitive architecture diagram and thought “I should implement this.” I just kept hitting problems and writing markdown files until the problems went away.

What it actually looks like

The whole system lives in ~/.claude/ — 13,000+ lines across memory modules, agent personas, slash commands, and skills. Here’s the part that matters for this story.

The virtual feature crew

Four specialist agents, defined in markdown, with distinct perspectives:

They’re not decorative personas. They have operational protocols. Every task in the main session goes through a SCOPE CHECK — a weighted complexity estimate that determines whether I do the work directly or delegate it:

Task: Add new API endpoint
Steps: 2 exploratory (4.0) + 3 standard (3.0) = 7.0
Total: 7.0
Trap check: Multi-file exploration
Decision: DELEGATE to Engineer

Weight >= 3? Delegate. Under 3? Proceed directly, but track accumulation — after 3 direct tasks, the next one gets delegated regardless. It’s a context hygiene circuit breaker.

Cognitive trap detection

This is the part that makes people raise an eyebrow. The system detects when the main session is rationalizing a task as simpler than it is.

The heuristic is brutally simple:

“Am I about to explain why this task is small?”

If yes, that explanation IS the cognitive trap.

Genuine small tasks don’t need justification. The moment you’re constructing an argument for why something is “just cleanup” or “a quick fix,” your reasoning is doing persuasive work on itself. The system catches specific linguistic fingerprints — “just,” “quick,” “obviously,” “cleanup” — and adds a weight correction. That correction often tips the task over the delegation threshold.

Task: Clean up old config file
Steps: 1 read (1.0) + 2 edits (1.0) = 2.0
Trap check: "Cleanup" framing detected — add 1.0 correction
Corrected total: 3.0
Decision: DELEGATE (crosses threshold)

The word “cleanup” itself changed the decision.

Context economics

Here’s the framing that makes the whole delegation system make sense instead of feeling like bureaucracy.

Direct work = paying cash from your wallet. Every file you read, every search, every edit — that’s context window budget spent from a finite pool. Once spent, it’s gone. Context compacts, and you’ve lost everything.

Delegation = paying with someone else’s budget. A subagent has its own context window. A 10-step task done directly costs 10 context steps. Delegated, it costs ~2 (spawn + receive). That’s 80% savings.

The main session is a coordinator, not a worker. It understands user intent, dispatches subagents, synthesizes results, and maintains conversation continuity. It does not read files. It does not grep through codebases. It does not implement features. Every context step spent on “doing” is a step stolen from “coordinating.”

This reframe changed everything. The system went from “sometimes delegate big tasks” to “delegate by default, do directly only when the overhead of delegation exceeds the work itself.”

The thing I built today

Today I wrote /user:zparallel-work — a slash command that orchestrates 2-4 independent tasks in true parallel.

You describe the tasks. It plans them into a table — each gets a fresh yolo monorepo clone, its own branch, its own engineer agent. You confirm. It clones the repos serially (you can’t parallelize git clones to the same remote without getting rate-limited). Then it launches all the engineer agents in a single response with run_in_background: true.

Each agent gets:

• The task description
• The repo path and branch name
• A condensed conventions block extracted from the repo’s CLAUDE.md
• A workflow: explore → implement → verify → commit → push → create draft PR → report back

While they work, the main session is free. Doing nothing. Waiting. Which is exactly the point — the coordinator’s job is to coordinate, not to work.

When they finish, a status table collects the results:

## Parallel Work Report

| # | Task | Clone | Branch | PR | Status |
|---|------|-------|--------|----|--------|
| 1 | Port 2-phase to rocket | ~/mcode/yolo-rocket-2phase | shiyuanzheng/port-2phase-rocket-build | #22454 | Done |
| 2 | Early mai_kernels | ~/mcode/yolo-early-kernels | shiyuanzheng/early-mai-kernels-install | #22455 | Done |

Two PRs. Parallel execution. One command. This is the thing agent frameworks sell as their killer feature — multi-agent orchestration with task distribution and result aggregation. I built it as a 155-line markdown file.

The fresh-context principle

There’s a design decision in the system that sounds paranoid until you see it in action: revisions always get a new agent.

When you write something and then try to fix it, you see what you meant to write, not what you actually wrote. Your brain fills in the gaps. You read past your own errors because you have the intent loaded in working memory. This is why copy editors exist — they’ve never seen your draft before, so they see what’s on the page.

The same thing happens with AI context windows. An agent that wrote version 1 of a document has the original intent loaded. It will read past its own mistakes. It will patch instead of rethink. It has sunk-cost attachment to its approach.

So the iterative orchestration pattern works like this:

1. Draft — Agent writes the thing
2. Review — Fresh agent evaluates against the original goal
3. Revise — Another fresh agent incorporates feedback
4. Sign-off — Reviewer approves against the original goal, not just “does this look better”

Each step is a new context window. No intent pollution. No sunk-cost bias. The math works out: spawning costs ~2 context steps per cycle. A loaded agent trying to self-correct costs more context AND produces worse output.

This is the Reflexion pattern from the agent research literature. I didn’t know that when I built it. I just knew that self-review didn’t work and external review did.

The memory system that frameworks don’t have

Most agent frameworks handle memory within a session. LangGraph has checkpointing — save state, restore it later. AutoGen has conversation history. These solve the “what happened 5 minutes ago” problem.

My system solves the “what happened 5 sessions ago” problem.

Short-term memory: SESSION-BOOTSTRAP.md. Current focus, active specs, next actions. ~20 lines. Gets read at session start, updated at milestones, archived at handoff. This is the LangGraph checkpoint equivalent.

Episodic memory: DECISIONS.md. Design decisions with rationale. When someone asks “why did we do it this way,” the answer is in a file, not in someone’s head.

Correction memory: LESSONS.md. This is the one that’s ahead of the frameworks. When I correct Claude’s approach — “no, don’t use argparse, use tyro” or “that API returns XML, not JSON” — the correction gets logged with root cause analysis:

## [Date] Wrong assumption about API response format

**What happened:** Assumed /api/v2/results returns JSON
**Root cause:** Inferred from endpoint naming convention instead of checking docs
**Correction:** Always verify response format from API documentation or a test call
**Applies to:** General pattern — never assume wire format

This persists. Next session, next month, next project — that lesson is loaded. The mistake doesn’t repeat. Most frameworks do reflection within a single execution. They don’t do cross-session learning. They don’t build a corpus of “things I got wrong and why.”

Procedural memory: The memory modules themselves — subagents.md, quality-assurance.md, security.md. These aren’t memories of events. They’re memories of how to behave. The difference between knowing what happened (episodic) and knowing how to do things (procedural). Agent frameworks encode this in code. I encode it in markdown that gets loaded into the system prompt.

What frameworks would give me

I’m not going to pretend my markdown files are better than actual software. There are real things I don’t have:

Automatic checkpointing. I manage SESSION-BOOTSTRAP.md manually. LangGraph would save state automatically at every node transition. That’s strictly better — the manual step is where I lose context, because the whole point of needing a checkpoint is that I’m about to forget things, and people who are about to forget things don’t reliably remember to save first.

Graph visualization. I can’t see the agent flow as a diagram. When /user:zparallel-work spawns four engineers and they each call tools and make decisions, there’s no visual trace. I reconstruct the execution from output. A framework would give me a DAG I could inspect.

Observability. No OpenTelemetry integration. No tracing spans. When a subagent takes 10 minutes, I don’t know if it spent 9 minutes thinking or 9 minutes waiting on a tool call. Debugging is vibes-based.

Time-travel. No ability to roll back to “the state before that bad decision.” My system is append-only — DECISIONS.md records what happened, but I can’t rewind to a previous checkpoint and replay from there. LangGraph can.

Typed state. My state is markdown files. Framework state is Pydantic models with validation. When SESSION-BOOTSTRAP.md gets malformed — and it does, because markdown is forgiving in all the wrong ways — nothing catches it until a session loads gibberish.

The honest question

Here’s where I’m sitting: I’m closer to a framework builder than a framework consumer.

The question isn’t “which agent framework should I adopt.” It’s “should I formalize what I’ve already built, or adopt someone else’s formalization?”

Adopting a framework means translating my protocols into their abstractions. My scope check becomes a node evaluation function. My subagent dispatch becomes a router. My markdown memory becomes typed state objects. The behavior is preserved, but the implementation changes completely. And the thing about frameworks is they’re opinionated — they have opinions about how agents should work, and those opinions might conflict with what I’ve learned by actually running this system for months.

Formalizing my own system means… building a framework. Which is a completely different project than the one I’m supposed to be doing, which is migrating a monorepo to Bazel and making Docker builds go faster.

The pragmatic answer is probably neither. Keep the markdown. Keep the slash commands. They work. They’re debuggable with cat. They evolve by editing a text file. The day I need typed state validation and OpenTelemetry tracing for my blog-writing subagent is the day I’ve lost the plot.

But it’s a strange feeling — building something you didn’t realize was a thing, and then recognizing it in a textbook.

How this grew

None of this was planned. Here’s the actual sequence:

1. Context kept evaporating. I wrote /user:recover to reload state from git. Solved the immediate pain.
2. Multi-session work kept losing thread. I added SESSION-BOOTSTRAP.md and the handoff protocol. Now state persisted across sessions.
3. Big tasks ate the context window. I added subagent delegation with scope checks. Now expensive work happened in someone else’s context.
4. I kept underestimating task complexity. I added cognitive trap detection. Now the system catches my own rationalizations.
5. Self-review didn’t catch errors. I added the fresh-context review protocol. Now revisions get clean eyes.
6. I kept making the same mistakes. I added LESSONS.md. Now corrections persist.
7. Independent tasks ran sequentially. I added /user:zparallel-work. Now multiple engineers work simultaneously.

Each step was a response to a specific friction point. The cognitive architecture emerged from the accumulation of fixes. It wasn’t top-down design — it was bottom-up evolution under selection pressure.

That’s probably why it maps so cleanly onto formal frameworks. The frameworks were designed by people who studied how intelligent agents should work. My system was shaped by how intelligent agents need to work when they’re doing real engineering tasks in a monorepo with 200 packages and a Docker build that takes 40 minutes.

Convergent evolution. Different starting points, same destination. Because the problems are the same — memory, planning, delegation, reflection, error correction — regardless of whether you’re building from theory or from pain.

The punchline

I have 13,000 lines of markdown that constitute a hand-rolled cognitive architecture with working memory, long-term memory, multi-agent orchestration, cognitive trap detection, iterative refinement with fresh-context review, parallel task execution, and cross-session learning.

It runs on text files and vibes.

It works better than anything I’ve tried that came in a pip package.

I don’t know what to do with that information yet. But I’m pretty sure the answer is “keep writing markdown files.”