loki/docs/QA-CHECKLIST.md

Loki QA Checklist

Behavioral verification checklist for the REST API refactor. Run after each step or phase to confirm existing functionality is preserved.

How to use

• = not yet verified for current step
• = verified working
• SKIP = not applicable to current step

Check each item manually in the REPL and/or CLI. If a check fails, stop and investigate before proceeding.

---

1. Build & Test Baseline

• cargo check — zero warnings, zero errors
• cargo clippy — zero warnings
• cargo test — all tests pass (63 as of Step 8g)

2. CLI — Basic Operations

• loki "hello" — single-shot chat works, response printed
• loki --role <name> "hello" — role applied, response uses role context
• loki --session <name> "hello" — session created/resumed, response saved
• loki --model <model_id> "hello" — specified model used
• loki --prompt "you are a pirate" "hello" — temp role applied
• loki --info — system info printed, exits cleanly
• loki --list-models — model list printed
• loki --list-roles — role list printed (no hidden files)
• loki --list-sessions — session list printed
• loki --list-agents — agent list printed (no .shared directory)
• loki --dry-run "hello" — no API call, input echoed
• loki --no-stream "hello" — non-streaming response

3. CLI — File Input

• loki --file /tmp/test.txt "summarize" — file content included
• loki --file /tmp/test.txt — file content sent without extra text

4. CLI — Agent (non-interactive)

• loki --agent <name> "do something" — agent starts, tools available, response returned
• Agent MCP servers start (if configured)
• Agent tool calls execute correctly (e.g., execute_command)

5. CLI — Shell Execute

• loki -e "list files in /tmp" — shell command generated
• Shell command explanation shown (describe mode)
• Shell command execution works when confirmed

6. CLI — Macro

• loki --macro <name> "input" — macro executes

7. REPL — Startup & Exit

• loki — REPL starts, welcome message shown
• .exit — REPL exits cleanly
• Ctrl+D — REPL exits cleanly
• Ctrl+C — prints exit hint, does not exit

8. REPL — Chat

• Type a message — response printed
• .continue — continues previous response
• .regenerate — regenerates last response
• .copy — copies last response to clipboard

9. REPL — Roles

• .role <name> — switches to role, prompt changes
• .role <name> <text> — one-shot role message
• .info role — shows role info
• .edit role — opens editor for current role
• .save role <name> — saves current role
• .exit role — exits role, prompt resets
• Role with MCP servers — servers start on .role <name>
• Role with MCP servers — MCP tools available in chat
• .exit role with MCP — servers stop, MCP tools removed

10. REPL — Sessions

• .session — starts temp session
• .session <name> — starts/resumes named session
• .info session — shows session info
• .edit session — opens editor
• .save session <name> — saves session
• .empty session — clears messages
• .compress session — compresses session
• .exit session — exits session
• Session with MCP servers — servers start
• Session carry-over prompt — "incorporate last Q&A?" appears when applicable

11. REPL — Agents

• .agent <name> — agent starts, tools compiled, prompt changes
• .agent <name> <session> — agent starts with specific session
• .agent <name> key=value — agent starts with variables
• .info agent — shows agent info
• .starter — shows conversation starters
• .starter <n> — executes starter
• .edit agent-config — opens agent config editor
• .exit agent — exits agent cleanly
• Agent with MCP servers — servers start
• Agent tool calls work (execute_command, fs_read, etc.)
• Agent global tools work (tools listed in global_tools)
• Agent tool file changes picked up on restart (delete .ts, .sh used instead)
• Auto-continuation works (todo list drives continuation)
• .clear todo — clears todo list

12. REPL — Sub-Agent Escalation

• Parent agent spawns sub-agent via tool call
• Sub-agent runs at depth > 0
• Sub-agent escalation: sub-agent calls user__ask → parent gets notification
• Parent calls agent__reply_escalation → sub-agent unblocked, resumes
• Multiple pending escalations shown in notification
• Max depth enforcement — sub-agent spawn rejected beyond max_agent_depth

13. REPL — RAG

• .rag <name> — initializes/loads RAG
• .info rag — shows RAG info
• .sources rag — shows citation sources
• .edit rag-docs — modify RAG documents
• .rebuild rag — rebuilds RAG index
• .exit rag — exits RAG
• RAG embeddings used in chat (search results included)

14. REPL — MCP Servers

• MCP servers start at REPL init (if globally enabled)
• .set enabled_mcp_servers <name> — changes active servers
• .set mcp_server_support true/false — toggles support
• MCP tool invocation works (mcp__invoke_)
• MCP tool search works (mcp__search_)
• MCP tool describe works (mcp__describe_)

15. REPL — Settings

• .set temperature 0.5 — changes temperature
• .set top_p 0.9 — changes top_p
• .set model <name> — changes model
• .set dry_run true — enables dry run
• .set stream false — disables streaming
• .set save true/false — toggles save
• .set highlight true/false — toggles highlighting
• .set save_session true/false/null — changes session save behavior
• .set compression_threshold <n> — changes threshold

16. REPL — Tab Completion

• .role<TAB> — shows role names (no hidden files)
• .agent<TAB> — shows agent names (no .shared directory)
• .session<TAB> — shows session names
• .rag<TAB> — shows RAG names
• .macro<TAB> — shows macro names
• .model<TAB> — shows model names with descriptions
• .set <TAB> — shows setting names
• .set temperature <TAB> — shows current value
• .set enabled_tools <TAB> — shows tool names
• .set enabled_mcp_servers <TAB> — shows server names

17. REPL — Delete

• .delete role <name> — deletes role
• .delete session <name> — deletes session
• .delete rag <name> — deletes RAG
• .delete macro <name> — deletes macro
• .delete agent-data <name> — deletes agent data

18. REPL — Vault

• .vault list — lists secrets
• .vault add <name> — adds secret
• .vault get <name> — retrieves secret
• .vault update <name> — updates secret
• .vault delete <name> — deletes secret

19. REPL — Prelude

• repl_prelude: "role:coder" — auto-loads role on REPL start
• repl_prelude: "session:mysession" — auto-loads session
• repl_prelude: "mysession:coder" — auto-loads session with role

20. REPL — Miscellaneous

• .help — shows help text
• .info — shows system info
• .authenticate — OAuth flow (if configured)
• .file <path> — includes file in next message
• .file <url> — fetches URL content
• Unknown command — shows error message
• Multi-line input (:::) — works correctly
• Ctrl+O — opens editor for input buffer

21. Session Compression & Autoname

• Session auto-compression triggers when threshold exceeded
• Compression message shown ("Compressing the session.")
• Session auto-naming triggers for new sessions
• Auto-continuation after compression works (agent resumes)

22. Error Handling

• Invalid role name — error shown, REPL continues
• Invalid model name — error shown, REPL continues
• Network error during chat — error shown, REPL continues
• MCP server crash — error shown, REPL continues
• Tool execution failure — error returned to LLM as tool result

---

Phase-specific notes

Phase 1 (Steps 3-10): Config split into AppState + RequestContext

Known bridge-window limitations (acceptable until Steps 9-10):

• ReplCompleter/ReplPrompt still hold GlobalConfig
• Input still holds GlobalConfig internally
• eval_tool_calls still takes &GlobalConfig
• Dual sync (sync_ctx_to_config/sync_config_to_ctx) required

Post-Phase 1 verification focus:

• All items above should work identically to pre-refactor behavior
• No new warnings or errors in build
• Performance should be equivalent (no observable slowdown)