refactor: Extracted common Python parser logic into a common.rs module

CHANGELOG.md

@@ -1,3 +1,74 @@
+## v0.3.0 (2026-04-02)
+
+### Feat
+
+- Added `todo__clear` function to the todo system and updated REPL commands to have a .clear todo as well for significant changes in agent direction
+- Added available tools to prompts for sisyphus and code-reviewer agent families
+- Added available tools to coder prompt
+- Improved token efficiency when delegating from sisyphus -> coder
+- modified sisyphus agents to use the new ddg-search MCP server for web searches instead of built-in model searches
+- Added support for specifying a custom response to multiple-choice prompts when nothing suits the user's needs
+- Supported theming in the inquire prompts in the REPL
+- Added the duckduckgo-search MCP server for searching the web (in addition to the built-in tools for web searches)
+- Support for Gemini OAuth
+- Support authenticating or refreshing OAuth for supported clients from within the REPL
+- Allow first-runs to select OAuth for supported providers
+- Support OAuth authentication flows for Claude
+- Improved MCP server spinup and spindown when switching contexts or settings in the REPL: Modify existing config rather than stopping all servers always and re-initializing if unnecessary
+- Allow the explore agent to run search queries for understanding docs or API specs
+- Allow the oracle to perform web searches for deeper research
+- Added web search support to the main sisyphus agent to answer user queries
+- Created a CodeRabbit-style code-reviewer agent
+- Added configuration option in agents to indicate the timeout for user input before proceeding (defaults to 5 minutes)
+- Added support for sub-agents to escalate user interaction requests from any depth to the parent agents for user interactions
+- built-in user interaction tools to remove the need for the list/confirm/etc prompts in prompt tools and to enhance user interactions in Loki
+- Experimental update to sisyphus to use the new parallel agent spawning system
+- Added an agent configuration property that allows auto-injecting sub-agent spawning instructions (when using the built-in sub-agent spawning system)
+- Auto-dispatch support of sub-agents and support for the teammate pattern between subagents
+- Full passive task queue integration for parallelization of subagents
+- Implemented initial scaffolding for built-in sub-agent spawning tool call operations
+- Initial models for agent parallelization
+- Added interactive prompting between the LLM and the user in Sisyphus using the built-in Bash utils scripts
+
+### Fix
+
+- Clarified user text input interaction
+- recursion bug with similarly named Bash search functions in the explore agent
+- updated the error for unauthenticated oauth to include the REPL .authenticated command
+- Corrected a bug in the coder agent that wasn't outputting a summary of the changes made, so the parent Sisyphus agent has no idea if the agent worked or not
+- Claude code system prompt injected into claude requests to make them valid once again
+- Do not inject tools when models don't support them; detect this conflict before API calls happen
+- The REPL .authenticate command works from within sessions, agents, and roles with pre-configured models
+- Implemented the path normalization fix for the oracle and explore agents
+- Updated the atlassian MCP server endpoint to account for future deprecation
+- Fixed a bug in the coder agent that was causing the agent to create absolute paths from the current directory
+- the updated regex for secrets injection broke MCP server secrets interpolation because the regex greedily matched on new lines, replacing too much content. This fix just ignores commented out lines in YAML files by skipping commented out lines.
+- Don't try to inject secrets into commented-out lines in the config
+- Removed top_p parameter from some agents so they can work across model providers
+- Improved sub-agent stdout and stderr output for users to follow
+- Inject agent variables into environment variables for global tool calls when invoked from agents to modify global tool behavior
+- Removed the unnecessary execute_commands tool from the oracle agent
+- Added auto_confirm to the coder agent so sub-agent spawning doesn't freeze
+- Fixed a bug in the new supervisor and todo built-ins that was causing errors with OpenAI models
+- Added condition to sisyphus to always output a summary to clearly indicate completion
+- Updated the sisyphus prompt to explicitly tell it to delegate to the coder agent when it wants to write any code at all except for trivial changes
+- Added back in the auto_confirm variable into sisyphus
+- Removed the now unnecessary is_stale_response that was breaking auto-continuing with parallel agents
+- Bypassed enabled_tools for user interaction tools so if function calling is enabled at all, the LLM has access to the user interaction tools when in REPL mode
+- When parallel agents run, only write to stdout from the parent and only display the parent's throbber
+- Forgot to implement support for failing a task and keep all dependents blocked
+- Clean up orphaned sub-agents when the parent agent
+- Fixed the bash prompt utils so that they correctly show output when being run by a tool invocation
+- Forgot to automatically add the bidirectional communication back up to parent agents from sub-agents (i.e. need to be able to check inbox and send messages)
+- Agent delegation tools were not being passed into the {{__tools__}} placeholder so agents weren't delegating to subagents
+
+### Refactor
+
+- Made the oauth module more generic so it can support loopback OAuth (not just manual)
+- Changed the default session name for Sisyphus to temp (to require users to explicitly name sessions they wish to save)
+- Updated the sisyphus agent to use the built-in user interaction tools instead of custom bash-based tools
+- Cleaned up some left-over implementation stubs
+
 ## v0.2.0 (2026-02-14)
 
 ### Feat

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "loki-ai"
-version = "0.2.0"
+version = "0.3.0"
 edition = "2024"
 authors = ["Alex Clarke <alex.j.tusa@gmail.com>"]
 description = "An all-in-one, batteries included LLM CLI Tool"
@@ -18,10 +18,11 @@ anyhow = "1.0.69"
 bytes = "1.4.0"
 clap = { version = "4.5.40", features = ["cargo", "derive", "wrap_help"] }
 dirs = "6.0.0"
+dunce = "1.0.5"
 futures-util = "0.3.29"
 inquire = "0.9.4"
 is-terminal = "0.4.9"
-reedline = "0.40.0"
+reedline = "0.46.0"
 serde = { version = "1.0.152", features = ["derive"] }
 serde_json = { version = "1.0.93", features = ["preserve_order"] }
 serde_yaml = "0.9.17"
@@ -37,7 +38,7 @@ tokio-graceful = "0.2.2"
 tokio-stream = { version = "0.1.15", default-features = false, features = [
   "sync",
 ] }
-crossterm = "0.28.1"
+crossterm = "0.29.0"
 chrono = "0.4.23"
 bincode = { version = "2.0.0", features = [
   "serde",
@@ -90,8 +91,8 @@ strum_macros = "0.27.2"
 indoc = "2.0.6"
 rmcp = { version = "0.16.0", features = ["client", "transport-child-process"] }
 num_cpus = "1.17.0"
-rustpython-parser = "0.4.0"
+tree-sitter = "0.24"
-rustpython-ast = "0.4.0"
+tree-sitter-python = "0.23"
 colored = "3.0.0"
 clap_complete = { version = "4.5.58", features = ["unstable-dynamic"] }
 gman = "0.3.0"

assets/functions/scripts/run-agent.py

@@ -50,6 +50,12 @@ def parse_raw_data(data):
 
 def parse_argv():
     agent_func = sys.argv[1]
+
+    tool_data_file = os.environ.get("LLM_TOOL_DATA_FILE")
+    if tool_data_file and os.path.isfile(tool_data_file):
+        with open(tool_data_file, "r", encoding="utf-8") as f:
+            agent_data = f.read()
+    else:
         agent_data = sys.argv[2]
 
     if (not agent_data) or (not agent_func):

assets/functions/scripts/run-agent.sh

@@ -14,7 +14,11 @@ main() {
 
 parse_argv() {
 		agent_func="$1"
+    if [[ -n "$LLM_TOOL_DATA_FILE" ]] && [[ -f "$LLM_TOOL_DATA_FILE" ]]; then
+      	agent_data="$(cat "$LLM_TOOL_DATA_FILE")"
+    else
 				agent_data="$2"
+    fi
     if [[ -z "$agent_data" ]] || [[ -z "$agent_func" ]]; then
         die "usage: ./{agent_name}.sh <agent-func> <agent-data>"
     fi
@@ -57,7 +61,6 @@ run() {
     if [[ "$OS" == "Windows_NT" ]]; then
         set -o igncr
         tools_path="$(cygpath -w "$tools_path")"
-        tool_data="$(echo "$tool_data" | sed 's/\\/\\\\/g')"
     fi
 
     jq_script="$(cat <<-'EOF'

assets/functions/scripts/run-tool.py

@@ -49,6 +49,11 @@ def parse_raw_data(data):
 
 
 def parse_argv():
+    tool_data_file = os.environ.get("LLM_TOOL_DATA_FILE")
+    if tool_data_file and os.path.isfile(tool_data_file):
+        with open(tool_data_file, "r", encoding="utf-8") as f:
+            return f.read()
+
     argv = sys.argv[:] + [None] * max(0, 2 - len(sys.argv))
 
     tool_data = argv[1]

assets/functions/scripts/run-tool.sh

@@ -13,7 +13,11 @@ main() {
 }
 
 parse_argv() {
+    if [[ -n "$LLM_TOOL_DATA_FILE" ]] && [[ -f "$LLM_TOOL_DATA_FILE" ]]; then
+        tool_data="$(cat "$LLM_TOOL_DATA_FILE")"
+    else
 				tool_data="$1"
+    fi
     if [[ -z "$tool_data" ]]; then
         die "usage: ./{function_name}.sh <tool-data>"
     fi
@@ -54,7 +58,6 @@ run() {
     if [[ "$OS" == "Windows_NT" ]]; then
         set -o igncr
         tool_path="$(cygpath -w "$tool_path")"
-        tool_data="$(echo "$tool_data" | sed 's/\\/\\\\/g')"
     fi
 
     jq_script="$(cat <<-'EOF'

docs/ENVIRONMENT-VARIABLES.md

@@ -108,5 +108,6 @@ can also pass the `--disable-log-colors` flag as well.
 
 ## Miscellaneous Variables
 | Environment Variable | Description                                                                                                                                                                                                                                                                                                                    | Default Value |
-|----------------------|--------------------------------------------------------------------------------------------------|---------------|
+|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
 | `AUTO_CONFIRM`       | Bypass all `guard_*` checks in the bash prompt helpers; useful for agent composition and routing                                                                                                                                                                                                                               |               |
+| `LLM_TOOL_DATA_FILE` | Set automatically by Loki on Windows. Points to a temporary file containing the JSON tool call data. <br>Tool scripts (`run-tool.sh`, `run-agent.sh`, etc.) read from this file instead of command-line args <br>to avoid JSON escaping issues when data passes through `cmd.exe` → bash. **Not intended to be set by users.** |               |

src/function/mod.rs

@@ -613,6 +613,7 @@ impl Functions {
             )
         })?;
         let content_template = unsafe { std::str::from_utf8_unchecked(&embedded_file.data) };
+        let to_script_path = |p: &str| -> String { p.replace('\\', "/") };
         let content = match binary_type {
             BinaryType::Tool(None) => {
                 let root_dir = Config::functions_dir();
@@ -622,8 +623,8 @@ impl Functions {
                 );
                 content_template
                     .replace("{function_name}", binary_name)
-                    .replace("{root_dir}", &root_dir.to_string_lossy())
+                    .replace("{root_dir}", &to_script_path(&root_dir.to_string_lossy()))
-                    .replace("{tool_path}", &tool_path)
+                    .replace("{tool_path}", &to_script_path(&tool_path))
             }
             BinaryType::Tool(Some(agent_name)) => {
                 let root_dir = Config::agent_data_dir(agent_name);
@@ -633,16 +634,19 @@ impl Functions {
                 );
                 content_template
                     .replace("{function_name}", binary_name)
-                    .replace("{root_dir}", &root_dir.to_string_lossy())
+                    .replace("{root_dir}", &to_script_path(&root_dir.to_string_lossy()))
-                    .replace("{tool_path}", &tool_path)
+                    .replace("{tool_path}", &to_script_path(&tool_path))
             }
             BinaryType::Agent => content_template
                 .replace("{agent_name}", binary_name)
-                .replace("{config_dir}", &Config::config_dir().to_string_lossy()),
+                .replace(
+                    "{config_dir}",
+                    &to_script_path(&Config::config_dir().to_string_lossy()),
+                ),
         }
         .replace(
             "{prompt_utils_file}",
-            &Config::bash_prompt_utils_file().to_string_lossy(),
+            &to_script_path(&Config::bash_prompt_utils_file().to_string_lossy()),
         );
         if binary_script_file.exists() {
             fs::remove_file(&binary_script_file)?;
@@ -666,7 +670,7 @@ impl Functions {
                     .join(".venv")
                     .join("Scripts")
                     .join("activate.bat");
-                let canonicalized_path = fs::canonicalize(&executable_path)?;
+                let canonicalized_path = dunce::canonicalize(&executable_path)?;
                 format!(
                     "call \"{}\" && {}",
                     canonicalized_path.to_string_lossy(),
@@ -677,19 +681,16 @@ impl Functions {
                 let executable_path = which::which("python")
                     .or_else(|_| which::which("python3"))
                     .map_err(|_| anyhow!("Python executable not found in PATH"))?;
-                let canonicalized_path = fs::canonicalize(&executable_path)?;
+                let canonicalized_path = dunce::canonicalize(&executable_path)?;
                 canonicalized_path.to_string_lossy().into_owned()
             }
             _ => bail!("Unsupported language: {}", language.as_ref()),
         };
         let bin_dir = binary_file
             .parent()
-            .expect("Failed to get parent directory of binary file")
+            .expect("Failed to get parent directory of binary file");
-            .canonicalize()?
+        let canonical_bin_dir = dunce::canonicalize(bin_dir)?.to_string_lossy().into_owned();
-            .to_string_lossy()
+        let wrapper_binary = dunce::canonicalize(&binary_script_file)?
-            .into_owned();
-        let wrapper_binary = binary_script_file
-            .canonicalize()?
             .to_string_lossy()
             .into_owned();
         let content = formatdoc!(
@@ -697,7 +698,7 @@ impl Functions {
 						@echo off
 						setlocal
 
-						set "bin_dir={bin_dir}"
+						set "bin_dir={canonical_bin_dir}"
 
 						{run} "{wrapper_binary}" %*"#,
         );
@@ -1117,6 +1118,20 @@ pub fn run_llm_function(
     #[cfg(windows)]
     let cmd_name = polyfill_cmd_name(&cmd_name, &bin_dirs);
 
+    #[cfg(windows)]
+    let cmd_args = {
+        let mut args = cmd_args;
+        if let Some(json_data) = args.pop() {
+            let tool_data_file = temp_file("-tool-data-", ".json");
+            fs::write(&tool_data_file, &json_data)?;
+            envs.insert(
+                "LLM_TOOL_DATA_FILE".into(),
+                tool_data_file.display().to_string(),
+            );
+        }
+        args
+    };
+
     envs.insert("CLICOLOR_FORCE".into(), "1".into());
     envs.insert("FORCE_COLOR".into(), "1".into());
 

src/parsers/common.rs

@@ -0,0 +1,269 @@
+use crate::function::{FunctionDeclaration, JsonSchema};
+use anyhow::{Context, Result, anyhow, bail};
+use indexmap::IndexMap;
+use serde_json::Value;
+use tree_sitter::Node;
+
+#[derive(Debug)]
+pub(crate) struct Param {
+    pub name: String,
+    pub ty_hint: String,
+    pub required: bool,
+    pub default: Option<Value>,
+    pub doc_type: Option<String>,
+    pub doc_desc: Option<String>,
+}
+
+pub(crate) trait ScriptedLanguage {
+    fn ts_language(&self) -> tree_sitter::Language;
+
+    fn default_runtime(&self) -> &str;
+
+    fn lang_name(&self) -> &str;
+
+    fn find_functions<'a>(
+        &self,
+        root: Node<'a>,
+        src: &str,
+    ) -> Vec<(Node<'a>, Node<'a>)>;
+
+    fn function_name<'a>(&self, func_node: Node<'a>, src: &'a str) -> Result<&'a str>;
+
+    fn extract_description(
+        &self,
+        wrapper_node: Node<'_>,
+        func_node: Node<'_>,
+        src: &str,
+    ) -> Option<String>;
+
+    fn extract_params(
+        &self,
+        func_node: Node<'_>,
+        src: &str,
+        description: &str,
+    ) -> Result<Vec<Param>>;
+}
+
+pub(crate) fn build_param(
+    name: &str,
+    mut ty: String,
+    mut required: bool,
+    default: Option<Value>,
+) -> Param {
+    if ty.ends_with('?') {
+        ty.pop();
+        required = false;
+    }
+
+    Param {
+        name: name.to_string(),
+        ty_hint: ty,
+        required,
+        default,
+        doc_type: None,
+        doc_desc: None,
+    }
+}
+
+pub(crate) fn build_parameters_schema(params: &[Param], _description: &str) -> JsonSchema {
+    let mut props: IndexMap<String, JsonSchema> = IndexMap::new();
+    let mut req: Vec<String> = Vec::new();
+
+    for p in params {
+        let name = p.name.replace('-', "_");
+        let mut schema = JsonSchema::default();
+
+        let ty = if !p.ty_hint.is_empty() {
+            p.ty_hint.as_str()
+        } else if let Some(t) = &p.doc_type {
+            t.as_str()
+        } else {
+            "str"
+        };
+
+        if let Some(d) = &p.doc_desc
+            && !d.is_empty()
+        {
+            schema.description = Some(d.clone());
+        }
+
+        apply_type_to_schema(ty, &mut schema);
+
+        if p.default.is_none() && p.required {
+            req.push(name.clone());
+        }
+
+        props.insert(name, schema);
+    }
+
+    JsonSchema {
+        type_value: Some("object".into()),
+        description: None,
+        properties: Some(props),
+        items: None,
+        any_of: None,
+        enum_value: None,
+        default: None,
+        required: if req.is_empty() { None } else { Some(req) },
+    }
+}
+
+pub(crate) fn apply_type_to_schema(ty: &str, s: &mut JsonSchema) {
+    let t = ty.trim_end_matches('?');
+    if let Some(rest) = t.strip_prefix("list[") {
+        s.type_value = Some("array".into());
+        let inner = rest.trim_end_matches(']');
+        let mut item = JsonSchema::default();
+
+        apply_type_to_schema(inner, &mut item);
+
+        if item.type_value.is_none() {
+            item.type_value = Some("string".into());
+        }
+        s.items = Some(Box::new(item));
+
+        return;
+    }
+
+    if let Some(rest) = t.strip_prefix("literal:") {
+        s.type_value = Some("string".into());
+        let vals = rest
+            .split('|')
+            .map(|x| x.trim().trim_matches('"').trim_matches('\'').to_string())
+            .collect::<Vec<_>>();
+        if !vals.is_empty() {
+            s.enum_value = Some(vals);
+        }
+        return;
+    }
+
+    s.type_value = Some(
+        match t {
+            "bool" => "boolean",
+            "int" => "integer",
+            "float" => "number",
+            "str" | "any" | "" => "string",
+            _ => "string",
+        }
+        .into(),
+    );
+}
+
+pub(crate) fn underscore(s: &str) -> String {
+    s.chars()
+        .map(|c| {
+            if c.is_ascii_alphanumeric() {
+                c.to_ascii_lowercase()
+            } else {
+                '_'
+            }
+        })
+        .collect::<String>()
+        .split('_')
+        .filter(|t| !t.is_empty())
+        .collect::<Vec<_>>()
+        .join("_")
+}
+
+pub(crate) fn node_text<'a>(node: Node<'_>, src: &'a str) -> Result<&'a str> {
+    node.utf8_text(src.as_bytes())
+        .map_err(|err| anyhow!("invalid utf-8 in source: {err}"))
+}
+
+pub(crate) fn named_child(node: Node<'_>, index: usize) -> Option<Node<'_>> {
+    let mut cursor = node.walk();
+    node.named_children(&mut cursor).nth(index)
+}
+
+pub(crate) fn extract_runtime(tree: &tree_sitter::Tree, src: &str, default: &str) -> String {
+    let root = tree.root_node();
+    let mut cursor = root.walk();
+    for child in root.named_children(&mut cursor) {
+        let text = match child.kind() {
+            "hash_bang_line" | "comment" => match child.utf8_text(src.as_bytes()) {
+                Ok(t) => t,
+                Err(_) => continue,
+            },
+            _ => break,
+        };
+
+        if let Some(cmd) = text.strip_prefix("#!") {
+            let cmd = cmd.trim();
+            if let Some(after_env) = cmd.strip_prefix("/usr/bin/env ") {
+                return after_env.trim().to_string();
+            }
+            return cmd.to_string();
+        }
+
+        break;
+    }
+    default.to_string()
+}
+
+pub(crate) fn generate_declarations<L: ScriptedLanguage>(
+    lang: &L,
+    src: &str,
+    file_name: &str,
+    is_tool: bool,
+) -> Result<Vec<FunctionDeclaration>> {
+    let mut parser = tree_sitter::Parser::new();
+    let language = lang.ts_language();
+    parser.set_language(&language).with_context(|| {
+        format!(
+            "failed to initialize {} tree-sitter parser",
+            lang.lang_name()
+        )
+    })?;
+
+    let tree = parser
+        .parse(src.as_bytes(), None)
+        .ok_or_else(|| anyhow!("failed to parse {}: {file_name}", lang.lang_name()))?;
+
+    if tree.root_node().has_error() {
+        bail!(
+            "failed to parse {}: syntax error in {file_name}",
+            lang.lang_name()
+        );
+    }
+
+    let _runtime = extract_runtime(&tree, src, lang.default_runtime());
+
+    let mut out = Vec::new();
+    for (wrapper, func) in lang.find_functions(tree.root_node(), src) {
+        let func_name = lang.function_name(func, src)?;
+
+        if func_name.starts_with('_') && func_name != "_instructions" {
+            continue;
+        }
+        if is_tool && func_name != "run" {
+            continue;
+        }
+
+        let description = lang
+            .extract_description(wrapper, func, src)
+            .unwrap_or_default();
+        let params = lang
+            .extract_params(func, src, &description)
+            .with_context(|| format!("in function '{func_name}' in {file_name}"))?;
+        let schema = build_parameters_schema(&params, &description);
+
+        let name = if is_tool && func_name == "run" {
+            underscore(file_name)
+        } else {
+            underscore(func_name)
+        };
+
+        let desc_trim = description.trim().to_string();
+        if desc_trim.is_empty() {
+            bail!("Missing or empty description on function: {func_name}");
+        }
+
+        out.push(FunctionDeclaration {
+            name,
+            description: desc_trim,
+            parameters: schema,
+            agent: !is_tool,
+        });
+    }
+    Ok(out)
+}

src/parsers/mod.rs

@@ -1,2 +1,3 @@
 pub(crate) mod bash;
+pub(crate) mod common;
 pub(crate) mod python;

src/repl/completer.rs

@@ -111,12 +111,14 @@ fn create_suggestion(value: &str, description: &str, span: Span) -> Suggestion {
         Some(description.to_string())
     };
     Suggestion {
+        display_override: None,
         value: value.to_string(),
         description,
         style: None,
         extra: None,
         span,
         append_whitespace: false,
+        match_indices: None,
     }
 }