feat: Deep research upgrade so sisyphus and the architect now have 3 possible research types: librarian standard, librarian deep, and true deep research with reflexion

assets/agents/architect/config.yaml

@@ -6,7 +6,7 @@ description: |
   plan-conformance check (plus a black-box usage-pattern probe for consumer-facing surface), and
   finishes with ONE draft PR (CI checks watched to green) plus tracked follow-up tasks. Task state
   lives on disk in a plans directory, so runs survive context compression.
-version: 2.6.0
+version: 2.8.0
 agent_session: temp
 auto_continue: true
 max_auto_continues: 100
@@ -18,6 +18,7 @@ spawnable_agents:
   - oracle
   - explore
   - librarian
+  - deep-research
   - adversary
   - probe
   - review-gauntlet
@@ -180,7 +181,9 @@ instructions: |
   <label>; authoritative sources preferred; return a distilled, deduplicated checklist"), put the
   returned checklist through the same accept/drop round with the user, and write the ACCEPTED
   items into the plan — as measurable acceptance criteria on the relevant tasks where possible,
-  otherwise as a checklist under `## Quality bar → Long-tail criteria`.
+  otherwise as a checklist under `## Quality bar → Long-tail criteria`. Spawn checklist
+  librarians with `--variables {"research_depth": "deep"}` — checklist distillation is
+  broad-topic research and the default standard depth (3 searches / 8 fetches) starves it.
 
   Ground the breakdown in real code: fan out `explore` agents (load `parallel-research`) across
   {{project_dir}} — and `librarian` for unfamiliar external libraries — to confirm the design's
@@ -593,7 +596,8 @@ instructions: |
   |-------|---------|
   | `sisyphus` | Implement ONE task's code in {{project_dir}} (its own explore/coder/verify/review loop). One per task. |
   | `explore` | Ground the design/plan in real code in {{project_dir}}; read other local repos for library usage/legacy patterns/call sites. Fan out in parallel. |
-  | `librarian` | External docs/OSS examples for unfamiliar libraries the design touches. |
+  | `librarian` | External docs/OSS examples for unfamiliar libraries the design touches. Design-time/broad topics: pass `--variables {"research_depth": "deep"}` (~3× budgets); default depth is for narrow lookups. One librarian per question beats one broad librarian — deep is for inherently broad questions. |
+  | `deep-research` | Open-ended/exhaustive research that should churn until its critique loop passes (plans sub-questions, deep librarian per question, credibility vetting). Spawn with `--variables {"autonomy": "autonomous"}` — MANDATORY, the supervised default blocks on a human approval gate; raise `reflexion_budget` (default 2) for longer churn. |
   | `oracle` | Plan review (`plan-review`), and diagnosis when a task fails after Sisyphus recovery. |
   | `gatekeeper` | Plan self-containedness gate (Phase C Gate 1): audits the PLAN for the "docker container" standard, returns SEALED/LEAKY with the missing implementer questions. |
   | `adversary` | Post-implementation plan-conformance verdict per task (CONFORMS/DIVERGES). |

assets/agents/deep-research/README.md

@@ -13,6 +13,13 @@ this agent runs a fixed graph: every request goes through the same
 `plan -> parallel research -> vet -> critique -> synthesize -> verify -> approve`
 pipeline.
 
+Two variables tune a run (pass via `--variables`): `reflexion_budget`
+(default 2) caps the automated critique -> re-research passes — raise it
+to let the run churn until the critique passes — and
+`autonomy: autonomous` skips the final human approval gate so other
+agents can spawn this one autonomously. Each sub-question is researched
+by a spawned `librarian` running at deep budgets.
+
 This agent is also the **canonical reference for the Coyote graph
 system**: it exercises every node type (`script`, `llm`, `rag`, `map`,
 `agent`, `input`, `approval`, `end`) and both static fan-out and

assets/agents/deep-research/graph.yaml

@@ -7,7 +7,12 @@ description: |
   delegates the final write-up to a focused sub-agent, checks that the
   cited sources are reachable, and gates the result behind human
   approval. A reviewer's free-form feedback at the approval step feeds
-  back into another research pass.
+  back into another research pass. Each sub-question is researched by a
+  spawned librarian running at deep budgets. Tunable via --variables:
+  reflexion_budget (default 2) caps the automated critique->re-research
+  passes - raise it to let the run churn until the critique passes;
+  autonomy=autonomous skips the human approval gate (REQUIRED when
+  another agent spawns this one).
 
   This is the canonical Coyote graph-agent reference: it exercises every
   node type (script, llm, rag, map, agent, input, approval, end) and
@@ -27,8 +32,16 @@ conversation_starters:
   - 'How does HTTP/3 differ from HTTP/2?'
   - 'Summarize recent advances in solid-state battery chemistry'
 
+variables:
+  - name: reflexion_budget
+    description: "Max automated critique->re-research passes before synthesis proceeds anyway. Default 2; raise it (4-8) for exhaustive runs that should churn until the critique passes."
+    default: '2'
+  - name: autonomy
+    description: "supervised (default): the finished report stops at the human approval gate. autonomous: skip the gate and return the report directly - REQUIRED when another agent spawns this one (a human gate would block the run)."
+    default: 'supervised'
+
 settings:
-  max_loop_iterations: 40
+  max_loop_iterations: 120
   log_state_snapshots: false
   validate_before_run: true
   max_concurrency: 4
@@ -142,23 +155,11 @@ nodes:
 
   research_one_question:
     id: research_one_question
-    type: llm
-    instructions: |
-      You are a web research assistant. Investigate the SINGLE question
-      given to you using your tools: search the web, fetch and read
-      pages, and search arXiv for academic sources.
-
-      Rules:
-        - Every factual claim must be backed by a real source you
-          actually retrieved. Never fabricate URLs, page titles,
-          authors, or DOIs.
-        - Prefer primary and authoritative sources over aggregators.
-        - Where sources disagree, report the disagreement rather than
-          papering over it.
-        - Put the URL (or DOI) inline next to each claim it supports.
-
-      Return organized findings in plain text. Do not include
-      meta-commentary about the process.
+    type: agent
+    agent: librarian
+    # The leaf research primitive: triage -> doc+OSS search -> cited
+    # synthesis. `inputs:` overlays the librarian graph's initial state,
+    # pinning research_depth to deep for every sub-question lane.
     prompt: |
       Research question: {{question}}
 
@@ -166,16 +167,13 @@ nodes:
       {{local_context}}
 
       {{research_feedback}}
-    tools:
-      - web_search_coyote
-      - fetch_url_via_curl
-      - search_arxiv
-      - mcp:ddg-search
-    max_iterations: 10
+    inputs:
+      research_depth: deep
+    timeout: 1900
     max_attempts: 2
     state_updates:
       finding: '{{output}}'
-    # On failure the engine writes "LLM node ...failed: <chain>" into `finding`
+    # On failure the engine writes "Agent node failed: <chain>" into `finding`
     # via the state_updates above BEFORE routing to the fallback. The marker
     # is branch-local: a map branch's fallback must stay inside the branch.
     fallback: question_fault
@@ -312,6 +310,13 @@ nodes:
     id: verify_sources
     type: script
     script: scripts/verify_sources.py
+    next: approval_gate
+
+  approval_gate:
+    id: approval_gate
+    type: script
+    description: Autonomy router - autonomous runs (spawned by another agent) skip the human approval gate and end accepted with the source check + pipeline notes attached; supervised (default) proceeds to the approval node. A crash fails forward via `next` to approve, the safe human-gated direction.
+    script: scripts/approval_gate.py
     next: approve
 
   approve:
@@ -348,6 +353,15 @@ nodes:
     type: end
     output: '{{report}}{{pipeline_notes}}'
 
+  end_autonomous:
+    id: end_autonomous
+    type: end
+    output: |
+      {{report}}
+
+      ----
+      {{source_check}}{{pipeline_notes}}
+
   end_rejected:
     id: end_rejected
     type: end

assets/agents/deep-research/scripts/approval_gate.py

@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+"""Autonomy router for deep-research's approval gate.
+
+Runs after verify_sources. When the `autonomy` variable is
+'autonomous' (the run was spawned by another agent — architect or
+sisyphus research tiers — with no human at the terminal), the report
+skips the human approval node and ends accepted: the reflexion loop
+already served as the quality gate, and end_autonomous carries the
+source check + pipeline notes with the report so the caller sees them.
+Any other value (default 'supervised') proceeds to the human approval
+gate unchanged.
+
+Routing (`_next`):
+  - autonomy == 'autonomous' -> end_autonomous
+  - anything else            -> approve
+
+Fail-safe: a crash here fails FORWARD via the node's `next` to
+`approve` — the safe, human-gated direction.
+"""
+import json
+import os
+
+
+def load_state():
+    if path := os.environ.get("GRAPH_STATE_FILE"):
+        with open(path) as f:
+            return json.load(f)
+    return json.loads(os.environ.get("GRAPH_STATE", "{}"))
+
+
+def main():
+    state = load_state()
+    autonomy = str(state.get("autonomy") or "supervised").strip().lower()
+    target = "end_autonomous" if autonomy == "autonomous" else "approve"
+    print(json.dumps({"_next": target}))
+
+
+main()

assets/agents/deep-research/scripts/question_fault.py

@@ -4,10 +4,11 @@
 Reached ONLY via research_one_question's `fallback:` route inside the
 `research_each_question` map branch (branch-local — a map branch's fallback
 must stay inside the branch subgraph). Grounded in engine behavior: on
-llm-node failure the engine writes "LLM node …failed: <chain>" into
-`finding` via the node's state_updates BEFORE routing to the fallback (on
-success the same key holds the researched findings, which never start with
-the "LLM node" prefix). This node normalizes the failure text into a
+branch failure the engine writes its failure text into `finding` via the
+node's state_updates BEFORE routing to the fallback — "Agent node failed:
+<chain>" now that the branch spawns a librarian agent, "LLM node …failed:
+<chain>" for an llm branch (on success the same key holds the researched
+findings, which never start with either prefix). This node normalizes the failure text into a
 PIPELINE-FAULT finding and the chain ends here — the map collects the fault
 as the lane's `finding`, so one dead research lane never sinks the whole
 map. combine_findings later lifts PIPELINE-FAULT findings into
@@ -53,7 +54,7 @@ def main():
     state = load_state()
     raw = state.get("finding")
     detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
-    if not detail.startswith("LLM node"):
+    if not detail.startswith(("LLM node", "Agent node")):
         detail = "the research lane died without recording a failure detail"
     if len(detail) > MAX_DETAIL_CHARS:
         detail = detail[: MAX_DETAIL_CHARS - 1] + "…"

assets/agents/deep-research/scripts/reflexion_gate.py

@@ -2,7 +2,8 @@
 """Reflexion gate for deep-research.
 
 Runs after `critique` has reviewed the current research findings. If the
-critique's verdict is REVISE and the reflexion budget is not spent,
+critique's verdict is REVISE and the reflexion budget (the graph's
+`reflexion_budget` variable, default 2) is not spent,
 loops back to `research` with the critique attached as
 `research_feedback`, so the retry is informed rather than a blind
 re-run. Otherwise it proceeds to `synthesize`.
@@ -26,9 +27,11 @@ import json
 import os
 import re
 
-# Automated revision passes allowed. `research` runs at most
-# MAX_REFLEXION_REVISIONS + 1 times per user pass. Bump to allow more.
-MAX_REFLEXION_REVISIONS = 2
+# Default automated revision passes when the `reflexion_budget` variable
+# is absent or malformed. `research` runs at most budget + 1 times per
+# user pass. Callers raise it via --variables {"reflexion_budget": "6"}
+# to let the run churn until the critique passes.
+DEFAULT_REFLEXION_REVISIONS = 2
 
 
 def load_state():
@@ -69,8 +72,9 @@ def main():
     critique = state.get("critique") or ""
     verdict = parse_verdict(critique)
     attempts = as_int(state.get("research_attempts"))
+    budget = as_int(state.get("reflexion_budget"), DEFAULT_REFLEXION_REVISIONS)
 
-    if verdict == "REVISE" and attempts < MAX_REFLEXION_REVISIONS:
+    if verdict == "REVISE" and attempts < budget:
         feedback = (
             "A reviewer judged the previous research pass incomplete. "
             "Address every point in the critique below:\n\n" + critique

assets/agents/librarian/graph.yaml

@@ -6,13 +6,16 @@ description: |
   preamble. The "external grep" sibling of explore (which handles
   internal/codebase grep). Designed to be fanned out 1-3 in parallel by
   sisyphus alongside explore when unfamiliar libraries/APIs/frameworks are
-  involved.
+  involved. Supports research_depth=deep (via --variables) for broad
+  design-time topics: ~3x the search/OSS/fetch budgets of the standard
+  quick-lookup mode.
 
 version: '1.0'
 
 global_tools:
   - web_search_coyote.sh
   - fetch_url_via_curl.sh
+  - search_arxiv.sh
 
 mcp_servers:
   - ddg-search
@@ -26,11 +29,14 @@ variables:
   - name: project_dir
     description: Project directory for context (unused in MVP but reserved for future iterations).
     default: '.'
+  - name: research_depth
+    description: "standard (default) = quick targeted lookup budgets. deep = design-time research: ~3x search/OSS/fetch budgets for broad topics (best-practice checklists, architecture surveys)."
+    default: 'standard'
 
 settings:
   max_loop_iterations: 12
   log_state_snapshots: true
-  timeout: 600
+  timeout: 1800
 
 initial_state:
   language_ecosystem: 'general'
@@ -117,7 +123,7 @@ nodes:
   search:
     id: search
     type: llm
-    description: Identify 3-5 authoritative documentation sources via ddg-search.
+    description: Identify authoritative documentation sources via ddg-search (3-5 standard depth, 6-12 deep).
     skills_enabled: true
     enabled_skills:
       - ai-slop-remover
@@ -129,7 +135,10 @@ nodes:
       Priority order:
         1. Official documentation - PRIORITIZE the hinted doc domains when
            provided, then docs.X.org / readthedocs.io / MDN / vendor docs
-        2. Specifications (RFCs, W3C, ECMA, IEEE)
+        2. Specifications (RFCs, W3C, ECMA, IEEE) and academic sources -
+           the search_arxiv tool is available; use it when the topic is
+           academic/research-flavored (arXiv/DOI links count as
+           authoritative)
         3. Credible secondary references (PEPs, official blog posts) - only
            if 1-2 are sparse
 
@@ -142,14 +151,19 @@ nodes:
 
       ## Search budget and fail-fast rules
 
-      You have a HARD BUDGET of 3 search calls total. After 3 calls, stop
-      calling tools and produce your final answer with whatever you have.
+      Your budget is set by the `Research depth` line in the prompt:
+        - standard -> HARD BUDGET of 3 search calls total.
+        - deep     -> HARD BUDGET of 10 search calls total; spend the extra
+          calls covering DISTINCT subtopics of the research topic, never
+          re-running near-duplicate queries.
+      After the budget is spent, stop calling tools and produce your final
+      answer with whatever you have.
 
       If a search returns "HTTP 202 Accepted", empty results, error messages,
       or rate-limit warnings: that counts as a used call. Do not retry the
       same query - either rephrase OR give up.
 
-      If after 3 calls you have NO usable URLs, output exactly:
+      If after spending the budget you have NO usable URLs, output exactly:
 
         NO_AUTHORITATIVE_SOURCES_FOUND
         Reason: <one line>
@@ -168,9 +182,11 @@ nodes:
         URL: <full url>
         ...
 
-      Output 3-5 source blocks. No prose intro, no closing summary.
+      Output 3-5 source blocks (standard depth) or 6-12 (deep). No prose
+      intro, no closing summary.
     prompt: |
       Research topic: {{initial_prompt}}
+      Research depth: {{research_depth}}
 
       Triage hints:
         - Language/ecosystem: {{language_ecosystem}}
@@ -183,8 +199,9 @@ nodes:
       pathlib` style queries).
     tools:
       - mcp:ddg-search
+      - search_arxiv
       - web_search_coyote
-    max_iterations: 15
+    max_iterations: 30
     state_updates:
       search_output: '{{output}}'
     fallback: synthesize
@@ -193,7 +210,7 @@ nodes:
   search_oss:
     id: search_oss
     type: llm
-    description: Find 2-3 production OSS examples relevant to the topic via the personal-github MCP.
+    description: Find production OSS examples relevant to the topic via the personal-github MCP (2-3 standard depth, 4-8 deep).
     skills_enabled: true
     enabled_skills:
       - ai-slop-remover
@@ -219,8 +236,12 @@ nodes:
 
       ## Search budget and fail-fast rules
 
-      HARD BUDGET: 8 tool calls total. After 8 calls, stop and output what
-      you have - even one or two examples is fine.
+      Your budget is set by the `Research depth` line in the prompt:
+        - standard -> HARD BUDGET of 8 tool calls total; output 2-3 blocks.
+        - deep     -> HARD BUDGET of 20 tool calls total; output 4-8 blocks
+          across DIFFERENT repos/approaches, not near-duplicates.
+      After the budget is spent, stop and output what you have - even one or
+      two examples is fine.
 
       If you find no production examples, output exactly:
 
@@ -240,10 +261,12 @@ nodes:
 
         REPO: ...
 
-      Output 2-3 blocks. The URL should point to a specific file that
+      Output 2-3 blocks (standard depth) or 4-8 (deep). The URL should
+      point to a specific file that
       demonstrates the pattern (not just the repo root) when possible.
     prompt: |
       Research topic: {{initial_prompt}}
+      Research depth: {{research_depth}}
 
       Triage hints:
         - Language/ecosystem: {{language_ecosystem}}
@@ -254,7 +277,7 @@ nodes:
       Filter to {{language_ecosystem}} repositories when the API allows.
     tools:
       - mcp:personal-github
-    max_iterations: 15
+    max_iterations: 30
     state_updates:
       oss_output: '{{output}}'
     fallback: synthesize
@@ -308,8 +331,10 @@ nodes:
         5. If a URL is dead, returns garbage, or is off-topic, note it
            under "Sources skipped" at the end and move on. Do not retry.
 
-      Budget: max 8 fetches total (across both source lists). Skip
-      aggressively.
+      Fetch budget, set by the `Research depth` line in the prompt:
+        - standard -> max 8 fetches total (across both source lists).
+        - deep     -> max 25 fetches total. Depth buys MORE SOURCES, never
+          re-fetching dead or off-topic URLs - skip aggressively either way.
 
       ## Output format
 
@@ -332,6 +357,7 @@ nodes:
       No flattery, no preamble. Start with `## Findings`.
     prompt: |
       Research topic: {{initial_prompt}}
+      Research depth: {{research_depth}}
 
       Documentation sources (from doc search branch):
       {{search_output}}
@@ -340,7 +366,7 @@ nodes:
       {{oss_output}}
     tools:
       - fetch_url_via_curl
-    max_iterations: 20
+    max_iterations: 45
     state_updates:
       findings: '{{output}}'
     fallback: final_format

assets/agents/sisyphus/config.yaml

@@ -1,6 +1,6 @@
 name: sisyphus
 description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline
-version: 3.15.0
+version: 3.17.0
 
 agent_session: temp
 auto_continue: true
@@ -11,6 +11,7 @@ can_spawn_agents: true
 spawnable_agents:
   - explore
   - librarian
+  - deep-research
   - coder
   - oracle
   - code-reviewer
@@ -157,7 +158,8 @@ instructions: |
   | Agent | Use For | Characteristics |
   |-------|---------|-----------------|
   | `explore` | Find patterns in THIS codebase, understand local code | Read-only, returns findings, fan out as many as the task warrants — one per distinct search angle, module, or concern. Large codebases or cross-cutting tasks should spawn 5–15+. |
-  | `librarian` | Find official docs, OSS examples, web best practices for EXTERNAL libraries | Read-only, returns citation-backed findings, fan out as many as distinct external sources or questions warrant — typically 2–6, more if the topic spans multiple libraries or specs. |
+  | `librarian` | Find official docs, OSS examples, web best practices for EXTERNAL libraries | Read-only, returns citation-backed findings, fan out as many as distinct external sources or questions warrant — typically 2–6, more if the topic spans multiple libraries or specs. Default depth = quick-lookup budgets; for a BROAD topic (design research, best-practice surveys, architecture comparisons) pass `--variables {"research_depth": "deep"}` (~3× search/fetch budgets). Still prefer several narrow librarians over one giant one — deep is for questions that are inherently broad, not a substitute for fan-out. |
+  | `deep-research` | Open-ended investigation that must CHURN until solid — plans sub-questions, fans out a deep librarian per question, vets source credibility, runs a critique→re-research loop | Spawn with `--variables {"autonomy": "autonomous"}` — MANDATORY when spawned as a subagent (the supervised default blocks on a human approval gate). Raise `reflexion_budget` (e.g. "6", default 2) to let it churn longer. Expect hours, not minutes; one per investigation — its internal map is the fan-out. |
   | `coder` | Write/edit files, implement features | Graph agent: plan → approval → implement → verify build+tests → self_review → bounded fix-loop |
   | `oracle` | Architecture, complex debugging, review, plan review | Advisory, blocking — never answer the user before collecting Oracle results |
   | `step-runner` | Execute ONE step of a phased plan repo (Phase 8) | Graph agent: orient → staleness check → coder → verify → handoff → user approval gate |