diff --git a/Home.md b/Home.md index a9f273b..4d33e27 100644 --- a/Home.md +++ b/Home.md @@ -13,42 +13,43 @@ Agents, and More. It is designed to include a number of useful agents, roles, macros, and more so users can get up and running with Loki in as little time as possible. -![Agent example](https://raw.githubusercontent.com/Dark-Alex-17/loki/main/docs/images/agents/sql.gif) +![Agent example](./docs/images/agents/sql.gif) -Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration guide](AICHAT-MIGRATION) to get started. +Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration guide](./docs/AIChat-Migration) to get started. --- +# Quick Links -* [AIChat Migration Guide](AICHAT-MIGRATION): Coming from AIChat? Follow the migration guide to get started. -* [Installation](docs/Installation.md): Install Loki -* [Getting Started](docs/Getting-Started.md): Get started with Loki by doing first-run setup steps, and learn the basics. -* [REPL](REPL): Interactive Read-Eval-Print Loop for conversational interactions with LLMs and Loki. - * [Custom REPL Prompt](REPL-PROMPT): Customize the REPL prompt to provide useful contextual information. -* [Vault](docs/VAULT.md): Securely store and manage sensitive information such as API keys and credentials. -* [Shell Integrations](SHELL-INTEGRATIONS): Seamlessly integrate Loki with your shell environment for enhanced command-line assistance. -* [Function Calling](TOOLS#Tools): Leverage function calling capabilities to extend Loki's functionality with custom tools - * [Creating Custom Tools](CUSTOM-TOOLS): You can create your own custom tools to enhance Loki's capabilities. - * [Create Custom Python Tools](CUSTOM-TOOLS#custom-python-based-tools) - * [Create Custom TypeScript Tools](CUSTOM-TOOLS#custom-typescript-based-tools) - * [Create Custom Bash Tools](CUSTOM-BASH-TOOLS) - * [Bash Prompt Utilities](BASH-PROMPT-HELPERS) -* [First-Class MCP Server Support](MCP-SERVERS): Easily connect and interact with MCP servers for advanced functionality. -* [Macros](MACROS): Automate repetitive tasks and workflows with Loki "scripts" (macros). -* [RAG](RAG): Retrieval-Augmented Generation for enhanced information retrieval and generation. -* [Sessions](SESSIONS): Manage and persist conversational contexts and settings across multiple interactions. -* [Roles](ROLES): Customize model behavior for specific tasks or domains. -* [Agents](AGENTS): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools. - * [Todo System](TODO-SYSTEM): Built-in task tracking for improved agent reliability with smaller models. -* [Environment Variables](ENVIRONMENT-VARIABLES): Override and customize your Loki configuration at runtime with environment variables. -* [Client Configurations](CLIENTS): Configuration instructions for various LLM providers. - * [Authentication (API Key & OAuth)](CLIENTS#authentication): Authenticate with API keys or OAuth for subscription-based access. - * [Patching API Requests](PATCHES): Learn how to patch API requests for advanced customization. -* [Custom Themes](THEMES): Change the look and feel of Loki to your preferences with custom themes. +* [AIChat Migration Guide](./docs/AIChat-Migration.md): Coming from AIChat? Follow the migration guide to get started. +* [Installation](./docs/Installation.md): Install Loki +* [Getting Started](./docs/Getting-Started.md): Get started with Loki by doing first-run setup steps, and learn the basics. +* [REPL](./docs/REPL.md): Interactive Read-Eval-Print Loop for conversational interactions with LLMs and Loki. + * [Custom REPL Prompt](./docs/REPL-Prompt.md): Customize the REPL prompt to provide useful contextual information. +* [Vault](./docs/VAULT.md): Securely store and manage sensitive information such as API keys and credentials. +* [Shell Integrations](./docs/Shell-Integrations.md): Seamlessly integrate Loki with your shell environment for enhanced command-line assistance. +* [Function Calling](./docs/function-calling/Tools.md#Tools): Leverage function calling capabilities to extend Loki's functionality with custom tools + * [Creating Custom Tools](./docs/function-calling/Custom-Tools.md): You can create your own custom tools to enhance Loki's capabilities. + * [Create Custom Python Tools](./docs/function-calling/Custom-Tools.md#custom-python-based-tools) + * [Create Custom TypeScript Tools](./docs/function-calling/Custom-Tools.md#custom-typescript-based-tools) + * [Create Custom Bash Tools](./docs/function-calling/Custom-Bash-Tools.md) + * [Bash Prompt Utilities](./docs/function-calling/Bash-Prompt-Helpers.md) +* [First-Class MCP Server Support](./docs/function-calling/MCP-Servers.md): Easily connect and interact with MCP servers for advanced functionality. +* [Macros](./docs/Macros.md): Automate repetitive tasks and workflows with Loki "scripts" (macros). +* [RAG](./docs/RAG.md): Retrieval-Augmented Generation for enhanced information retrieval and generation. +* [Sessions](/docs/Sessions.md): Manage and persist conversational contexts and settings across multiple interactions. +* [Roles](./docs/Roles.md): Customize model behavior for specific tasks or domains. +* [Agents](/docs/Agents.md): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools. + * [Todo System](./docs/TODO-System.md): Built-in task tracking for improved agent reliability with smaller models. +* [Environment Variables](./docs/Environment-Variables.md): Override and customize your Loki configuration at runtime with environment variables. +* [Client Configurations](./docs/clients/Clients.md): Configuration instructions for various LLM providers. + * [Authentication (API Key & OAuth)](./docs/clients/Clients.md#authentication): Authenticate with API keys or OAuth for subscription-based access. + * [Patching API Requests](./docs/clients/Patches.md): Learn how to patch API requests for advanced customization. +* [Custom Themes](./docs/Themes.md): Change the look and feel of Loki to your preferences with custom themes. * [History](#history): A history of how Loki came to be. --- -## History +# History Loki began as a fork of [AIChat CLI](https://github.com/sigoden/aichat) and has since evolved into an independent project. @@ -56,5 +57,5 @@ See [CREDITS.md](https://github.com/Dark-Alex-17/loki/CREDITS.md) for full attri --- -## Creator +# Creator * [Alex Clarke](https://github.com/Dark-Alex-17) diff --git a/docs/AIChat-Migration.md b/docs/AIChat-Migration.md new file mode 100644 index 0000000..0fd26d5 --- /dev/null +++ b/docs/AIChat-Migration.md @@ -0,0 +1,210 @@ +Loki originally started as a fork of AIChat but has since evolved into its own separate project with separate goals. + +As a result, there's some changes you'll need to make to your AIChat configuration to be able to use Loki. + +Be sure you've run `loki` at least once so that the Loki configuration directory and subdirectories exist and is +populated with the built-in defaults. + +# Global Configuration File +You should be able to copy/paste your AIChat configuration file into your Loki configuration directory. Since the +location of the Loki configuration directory varies between systems, you can use the following command to locate your +config directory: + +```shell +loki --info | grep 'config_dir' | awk '{print $2}' +``` + +Then, you'll need to make the following changes: + +* `function_calling` -> `function_calling_support` +* `use_tools` -> `enabled_tools` +* `agent_prelude` -> `agent_session` +* `compress_threshold` -> `compression_threshold` +* `summarize_prompt` -> `summarization_prompt` +* `summary_prompt` -> `summary_context_prompt` + +# Roles +Locate your `roles` directory using the following command: + +```shell +loki --info | grep 'roles_dir' | awk '{print $2}' +``` + +Update any roles that have `use_tools` to `enabled_tools`. + +# Sessions +Locate your `sessions` directory using the following command: + +```shell +loki --info | grep 'sessions_dir' | awk '{print $2}' +``` + +Update the following settings: +* `use_tools` -> `enabled_tools` +* `compress_threshold` -> `compression_threshold` +* `summarize_prompt` -> `summarization_prompt` +* `summary_prompt` -> `summary_context_prompt` + +--- + +# LLM Functions Changes +Probably the most significant difference between AIChat and Loki is how tools are handled. So if you cloned the +[llm-functions](https://github.com/sigoden/llm-functions) repo, you'll need to make the following changes. + +**Note: JavaScript functions are not supported in Loki.** + +The following guide assumes you're using the `llm-functions` repository as your base for custom functions, and thus +follows that directory structure. + +## Agents +Agents are now all handled in one place: the `agents` directory (`/agents`): + +```shell +loki --info | grep 'agents_dir' | awk '{print $2}' +``` + +And instead of separate `index.yaml` and `config.yaml` files, they're now both in a single `config.yaml` file. + +So now for all of your agents, copy all the contents of those directories to the corresponding directory in the Loki +`agents` directory. Then make the following changes: + +* Copy the contents of your `/functions/agents` directory into `/agents//tools.txt` +* No `/agents//index.yaml` + +## Functions +Loki consolidates much of the `llm-functions` repo functionality into one binary. So this means + +* There's no need to have `argc` installed anymore +* No separate repository to manage +* No `tools.txt` +* No `functions.json` +* No `functions/mcp` directory at all +* No `functions/scripts` + +Here's how to migrate your functions over to Loki from the `llm-functions` repository. + +* Copy your AIChat `/functions` directory into your Loki config directory +* Delete the following files and directories from your `/functions` directory: + * `scripts/` + * `agents.txt` + * `functions.json` + * `Argcfile.sh` + * `README.md` (irrelevant now) + * `LICENSE` (irrelevant now) + * `utils/guard_operation.sh` + * `utils/guard_path.sh` + * `utils/patch.awk` +* Everything in `tools.txt` now lives in the global config file under the `visible_tools` setting: + ```text + get_current_weather.sh + execute_command.sh + web_search.sh + #execute_py_code.py + query_jira_issues.sh + ``` + becomes the following in your `/config.yaml` + ```yaml + visible_tools: + - get_current_weather.sh + - execute_command.sh + - web_search.sh + # - web_search.sh + - query_jira_issues.sh + ``` +* If you've defined a `functions/mcp.json` file, you can leave it alone. +* Similarly to agents, if you have any bash `tools.sh` that depend on the utility scripts in the `llm-functions` + repository, they've been replaced by built-in utility scripts. So use the following to replace any matching lines in + your `tools.sh` files: + ```bash + ################## + ## Scripts file ## + ################## + ROOT_DIR="${LLM_ROOT_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" + # replace with + source "$LLM_PROMPT_UTILS_FILE" + + ####################### + ## guard_path script ## + ####################### + "$ROOT_DIR/utils/guard_path.sh" + # replace with + guard_path + + ############################ + ## guard_operation script ## + ############################ + "$ROOT_DIR/utils/guard_operation.sh" + # replace with + guard_operation + + ###################### + ## patch.awk script ## + ###################### + awk -f "$ROOT_DIR/utils/patch.awk" + # replace with + patch_file + ``` + +Refer to the [custom bash tools docs](./function-calling/Custom-Bash-Tools.md) to learn how to compile and test bash +tools in Loki without needing to use `argc`. diff --git a/docs/Agents.md b/docs/Agents.md new file mode 100644 index 0000000..fb74be5 --- /dev/null +++ b/docs/Agents.md @@ -0,0 +1,739 @@ +Agents in Loki follow the same style as OpenAI's GPTs. They consist of 3 parts: + +* [Role](./Roles.md) - Tell the LLM how to behave +* [RAG](./RAG.md) - Pre-built knowledge bases specifically for the agent +* [Function Calling](./function-calling/Tools.md#tools) ([#2](./function-calling/MCP-Servers.md)) - Extends the functionality of the LLM through custom functions it can call + +![Agent example](./images/agents/sql.gif) + +Agent configuration files are stored in the `agents` subdirectory of your Loki configuration directory. The location of +this directory varies between systems so you can use the following command to locate yours: + +```shell +loki --info | grep 'agents_dir' | awk '{print $2}' +``` + +If you're looking for more example agents, refer to the [built-in agents](https://github.com/Dark-Alex-17/loki/blob/main/assets/agents). + +--- + +# Directory Structure +Agent configurations often have the following directory structure: + +``` +/agents + └── my-agent + ├── config.yaml + ├── tools.sh + or + ├── tools.py + or + ├── tools.ts +``` + +This means that agent configurations often are only two files: the agent configuration file (`config.yaml`), and the +tool definitions (`agents/my-agent/tools.sh`, `tools.py`, or `tools.ts`). + +To see a full example configuration file, refer to the [example agent config file](https://github.com/Dark-Alex-17/loki/blob/main/config.agent.example.yaml). + +The best way to understand how an agent is built is to go step by step in the following manner: + +--- + +# 1. Metadata +Agent configurations have the following settings available to customize each agent: + +```yaml +# Model Configuration +model: openai:gpt-4o # Specify the LLM to use +temperature: null # Set default temperature parameter, range (0, 1) +top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2), depending on the model +# Agent Metadata Configuration +agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session +# Agent Configuration +name: # Name of the agent, used in the UI and logs +description: # Description of the agent, used in the UI +version: 1 # Version of the agent +# Function Calling Configuration +mcp_servers: # Optional list of MCP servers that the agent utilizes + - github # Corresponds to the name of an MCP server in the `/functions/mcp.json` file +global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent + - web_search + - fs + - python +# Todo System & Auto-Continuation (see "Todo System & Auto-Continuation" section below) +auto_continue: false # Enable automatic continuation when incomplete todos remain +max_auto_continues: 10 # Maximum continuation attempts before stopping +inject_todo_instructions: true # Inject todo tool instructions into system prompt +continuation_prompt: null # Custom prompt for continuations (optional) +# Sub-Agent Spawning (see "Sub-Agent Spawning System" section below) +can_spawn_agents: false # Enable spawning child agents +max_concurrent_agents: 4 # Max simultaneous child agents +max_agent_depth: 3 # Max nesting depth (prevents runaway) +inject_spawn_instructions: true # Inject spawning instructions into system prompt +summarization_model: null # Model for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini') +summarization_threshold: 4000 # Char count above which sub-agent output is summarized +escalation_timeout: 300 # Seconds sub-agents wait for escalated user input (default: 5 min) +``` + +As mentioned previously: Agents utilize function calling to extend a model's capabilities. However, agents operate in +isolated environment, so in order for an agent to use a tool or MCP server that you have defined globally, you must +explicitly state which tools and/or MCP servers the agent uses. Otherwise, it is assumed that the agent doesn't use any +tools outside its own custom defined tools. + +And if you don't define a `agents/my-agent/tools.sh`, `agents/my-agent/tools.py`, or `agents/my-agent/tools.ts`, then the agent is really just a +`role`. + +You'll notice there are no settings for agent-specific tooling. This is because they are handled separately and +automatically. See the [Building Tools for Agents](#4-building-tools-for-agents) section below for more information. + +To see a full example configuration file, refer to the [example agent config file](https://github.com/Dark-Alex-17/loki/blob/main/config.agent.example.yaml). + +# 2. Define the Instructions +At their heart, agents function similarly to roles in that they tell the model how to behave. Agent configuration files +have the following settings for the instruction definitions: + +```yaml +dynamic_instructions: # Whether to use dynamically generated instructions for the agent; if false, static instructions are used. False by default. +instructions: # Static instructions for the LLM; These are ignored if dynamic instructions are used +variables: # An array of optional variables that the agent expects and uses +``` + +## Static Instructions +By default, Loki agents use statically defined instructions. Think of them as being identical to the instructions for a +[role](./Roles.md#instructions), because they virtually are. + +**Example:** +```yaml +instructions: | + You are an AI agent designed to demonstrate agentic capabilities +``` + +Just like roles, agents support variable interpolation at runtime. There's two types of variables that can be +interpolated into the instructions at runtime: special variables (like roles have), and user-defined variables. Just +like roles, variables are interpolated into your instructions anywhere Loki sees the `{{variable}}` syntax. + +### Special Variables +The following special variables are provided by Loki at runtime and can be injected into your agent's instructions: + +| Name | Description | Example | +|-----------------|---------------------------------------------------------------------|----------------------------| +| `__os__` | Operating system name | `linux` | +| `__os_family__` | Operating system family | `unix` | +| `__arch__` | System architecture | `x86_64` | +| `__shell__` | The current user's default shell | `bash` | +| `__locale__` | The current user's preferred language and region settings | `en-US` | +| `__now__` | Current timestamp in ISO 8601 format | `2025-11-07T10:15:44.268Z` | +| `__cwd__` | The current working directory | `/tmp` | +| `__tools__` | A list of the enabled tools (global + mcp servers + agent-specific) | | + +### User-Defined Variables +Agents also support user-defined variables that can be interpolated into the instructions, and are made available to any +agent-specific tools you define (see [Building Tools for Agents](#4-building-tools-for-agents) for more details on how to +create agent-specific tooling). + +The `variables` setting in an agent's config has the following fields: + +| Field | Required | Description | +|---------------|----------|----------------------------------------------------------------------------------------------------| +| `name` | * | The name of the variable | +| `description` | * | The description of the field | +| `default` | | A default value for the field. If left undefined, the user will be prompted for a value at runtime | + +These variables can be referenced in both the agent's instructions, and in the tool definitions via `LLM_AGENT_VAR_`. + +**Example:** +```yaml +instructions: | + You are an agent who answers questions about a user's system. + + + {{__tools__}} + + + + os: {{__os__}} + os_family: {{__os_family__}} + arch: {{__arch__}} + shell: {{__shell__}} + locale: {{__locale__}} + now: {{__now__}} + cwd: {{__cwd__}} + + + + username: {{username}} + +variables: + - name: username # Accessible from the tool definitions via the `LLM_AGENT_VAR_USERNAME` environment variable + description: Your user name +``` + +## Dynamic Instructions +Sometimes you may find it useful to dynamically generate instructions on startup. Whether that be via a call to Loki +itself to generate them, or by some other means. Loki supports this type of behavior using a special function defined +in your `agents/my-agent/tools.py`, `agents/my-agent/tools.sh`, or `agents/my-agent/tools.ts`. + +**Example: Instructions for a JSON-reader agent that specializes on each JSON input it receives** +`agents/json-reader/tools.py`: +```python +import json +from pathlib import Path +from genson import SchemaBuilder + +def _instructions(): + """Generates instructions for the agent dynamically""" + value = input("Enter a JSON file path OR paste raw JSON: ").strip() + if not value: + raise SystemExit("A file path or JSON string is required.") + + p = Path(value) + if p.exists() and p.is_file(): + json_file_path = str(p.resolve()) + json_text = p.read_text(encoding="utf-8") + else: + try: + json.loads(value) + except json.JSONDecodeError as e: + raise SystemExit(f"Input is neither a file nor valid JSON.\n{e}") + json_file_path = "" + json_text = value + + try: + data = json.loads(json_text) + except json.JSONDecodeError as e: + raise SystemExit(f"Provided content is not valid JSON.\n{e}") + + builder = SchemaBuilder() + builder.add_object(data) + json_schema = builder.to_schema() + return f""" + You are an AI agent that can view and filter JSON data with jq. + + ## Context + json_file_path: {json_file_path} + json_schema: {json.dumps(json_schema, indent=2)} + """ +``` + +or + +`agents/json-reader/tools.sh`: +```bash +#!/usr/bin/env bash +set -e + +# @meta require-tools jq,genson +# @env LLM_OUTPUT=/dev/stdout The output path + +# @cmd Generates instructions for the agent dynamically +_instructions() { + read -r -p "Enter a JSON file path OR paste raw JSON: " value + + if [[ -z "${value}" ]]; then + echo "A file path or JSON string is required" >&2 + exit 1 + fi + json_file_path="" + inline_temp="" + cleanup() { + [[ -n "${inline_temp:-}" && -f "${inline_temp}" ]] && rm -f "${inline_temp}" + } + trap cleanup EXIT + + if [[ -f "${value}" ]]; then + json_file_path="$(realpath "${value}")" + if ! jq empty "${json_file_path}" >/dev/null 2>&1; then + echo "Error: File does not contain valid JSON: ${json_file_path}" >&2 + exit 1 + fi + else + inline_temp="$(mktemp)" + printf "%s" "${value}" > "${inline_temp}" + if ! jq empty "${inline_temp}" >/dev/null 2>&1; then + echo "Error: Input is neither a file nor valid JSON." >&2 + exit 1 + fi + json_file_path="" + fi + + source_file="${json_file_path}" + if [[ "${json_file_path}" == "" ]]; then + source_file="${inline_temp}" + fi + + json_schema="$(genson < "${source_file}" | jq -c '.')" + cat <> "$LLM_OUTPUT" +You are an AI agent that can view and filter JSON data with jq. + +## Context +json_file_path: ${json_file_path} +json_schema: ${json_schema} +EOF +} +``` + +For more information on how to create custom tools for your agent and the structure of the `agent/my-agent/tools.sh`, +`agent/my-agent/tools.py`, or `agent/my-agent/tools.ts` files, refer to the [Building Tools for Agents](#4-building-tools-for-agents) section below. + +### Variables +All the same variable interpolations supported by static instructions is also supported by dynamic instructions. For +more information on what variables are available and how to use them, refer to the [Special Variables](#special-variables) +and [User-Defined Variables](#user-defined-variables) sections above. + +# 3. Initializing RAG +Each agent you create also has a dedicated knowledge base that adds additional context to your queries and helps the LLM +answer queries effectively. The documents to load into RAG are defined in the `documents` array of your agent +configuration file: + +```yaml +documents: + - https://www.ohdsi.org/data-standardization/ + - https://github.com/OHDSI/Vocabulary-v5.0/wiki/** + - OMOPCDM_ddl.sql # Relative path to agent (i.e. file lives at '/agents/my-agent/OMOPCDM_ddl.sql') +``` + +These documents use the same syntax as those you'd define when constructing RAG normally. To see all the available types +of documents that Loki supports and how to use custom document loaders, refer to the [RAG documentation](./RAG.md#supported-document-sources). + +Anytime your agent starts up, it will automatically be using the RAG you've defined here. + +# 4. Building Tools for Agents +Building tools for agents is virtually identical to building custom tools, with one slight difference: instead of +defining a single function that gets executed at runtime (e.g. `main` for bash tools and `run` for Python tools), agent +tools define a number of *subcommands*. + +## Limitations +You can only utilize one of: a bash-based `/agents/my-agent/tools.sh`, a Python-based +`/agents/my-agent/tools.py`, or a TypeScript-based `/agents/my-agent/tools.ts`. +However, if it's easier to achieve a task in one language vs the other, +you're free to define other scripts in your agent's configuration directory and reference them from the main +tools file. **Any scripts *not* named `tools.{py,sh,ts}` will not be picked up by Loki's compiler**, meaning they +can be used like any other set of scripts. + +It's important to keep in mind the following: + +* **Do not give agents the same name as an executable**. Loki compiles the tools for each agent into a binary that it + temporarily places on your path during execution. If you have a binary with the same name as your agent, then your + shell may execute the existing binary instead of your agent's tools +* **`LLM_ROOT_DIR` points to the agent's configuration directory**. This is where agents differ slightly from normal + tools: The `LLM_ROOT_DIR` environment variable does *not* point to the `functions/tools` directory like it does in + global tools. Instead, it points to the agent's configuration directory, making it easier to source scripts and other + miscellaneous files + +## .env File Support +When Loki loads an agent, it will also search the agent's configuration directory for a `.env` file. If found, all +environment variables defined in the file will be made available to the agent's tools. + +## Python-Based Agent Tools +Python-based tools are defined exactly the same as they are for custom tool definitions. The only difference is that +instead of a single `run` function, you define as many as you like with whatever arguments you like. + +**Example:** +`agents/my-agent/tools.py` +```python +import urllib.request + +def get_ip_info(): + """ + Get your IP information + """ + with urllib.request.urlopen("https://httpbin.org/ip") as response: + data = response.read() + return data.decode('utf-8') + +def get_ip_address_from_aws(): + """ + Find your public IP address using AWS + """ + with urllib.request.urlopen("https://checkip.amazonaws.com") as response: + data = response.read() + return data.decode('utf-8') +``` + +Loki automatically compiles these as separate functions for the LLM to call. No extra work is needed. Just make sure you +follow all the same steps to define each function as you would when creating custom Python tools. + +For more information on how to build tools in Python, refer to the [custom Python tools documentation](./function-calling/Custom-Tools.md#custom-python-based-tools) + +## Bash-Based Agent Tools +Bash-based agent tools are virtually identical to custom bash tools, with only one difference. Instead of defining a +single entrypoint via the `main` function, you actually define as many subcommands as you like. + +**Example:** +`agents/my-agent/tools.sh` +```bash +#!/usr/bin/env bash + +# @env LLM_OUTPUT=/dev/stdout The output path +# @describe Discover network information about your computer and its place in the internet + +# Use the `@cmd` annotation to define subcommands for your script. +# @cmd Get your IP information +get_ip_info() { + curl -fsSL https://httpbin.org/ip >> "$LLM_OUTPUT" +} + +# @cmd Find your public IP address using AWS +get_ip_address_from_aws() { + curl -fsSL https://checkip.amazonaws.com >> "$LLM_OUTPUT" +} +``` +To compile the script so it's executable and testable: +```bash +$ loki --build-tools +``` + +Then you can execute your script (assuming your current working directory is `agents/my-agent`): +```bash +$ ./tools.sh get_ip_info +$ ./tools.sh get_ip_address_from_aws +``` + +All other special annotations (`@env`, `@arg`, `@option` `@flags`) apply to subcommands as well, so be sure to follow +the same syntax ad formatting as is used to create custom bash tools globally. + +For more information on how to write, [build and test](function-calling/Custom-Bash-Tools.md#execute-and-test-your-bash-tools) tools in bash, refer to the +[custom bash tools documentation](function-calling/Custom-Bash-Tools.md). + +## TypeScript-Based Agent Tools +TypeScript-based agent tools work exactly the same as TypeScript global tools. Instead of a single `run` function, +you define as many exported functions as you like. Non-exported functions are private helpers and are invisible to the +LLM. + +**Example:** +`agents/my-agent/tools.ts` +```typescript +/** + * Get your IP information + */ +export async function get_ip_info(): Promise { + const resp = await fetch("https://httpbin.org/ip"); + return await resp.text(); +} + +/** + * Find your public IP address using AWS + */ +export async function get_ip_address_from_aws(): Promise { + const resp = await fetch("https://checkip.amazonaws.com"); + return await resp.text(); +} + +// Non-exported helper — invisible to the LLM +function formatResponse(data: string): string { + return data.trim(); +} +``` + +Loki automatically compiles each exported function as a separate tool for the LLM to call. Just make sure you +follow the same JSDoc and parameter conventions as you would when creating custom TypeScript tools. + +TypeScript agent tools also support dynamic instructions via an exported `_instructions()` function: + +```typescript +import { readFileSync } from "fs"; + +/** + * Generates instructions for the agent dynamically + */ +export function _instructions(): string { + const schema = readFileSync("schema.json", "utf-8"); + return `You are an AI agent that works with the following schema:\n${schema}`; +} +``` + +For more information on how to build tools in TypeScript, refer to the [custom TypeScript tools documentation](function-calling/Custom-Tools.md#custom-typescript-based-tools). + +# 5. Conversation Starters +It's often helpful to also have some conversation starters so users know what kinds of things the agent is capable of +doing. These are available in the REPL via the `.starter` command and are selectable. + +They are defined using the `conversation_starters` setting in your agent's configuration file: + +**Example:** +`agents/my-agent/config.yaml`: +```yaml +conversation_starters: + - What is my username? + - What is my current shell? + - What is my ip? + - How much disk space is left on my PC?? + - How to create an agent? +``` + +![Example Conversation Starters](./images/agents/conversation-starters.gif) + +# 6. Todo System & Auto-Continuation + +Loki includes a built-in task tracking system designed to improve the reliability of agents, especially when using +smaller language models. The Todo System helps models: + +- Break complex tasks into manageable steps +- Track progress through multi-step workflows +- Automatically continue work until all tasks are complete + +## Quick Configuration + +```yaml +# agents/my-agent/config.yaml +auto_continue: true # Enable auto-continuation +max_auto_continues: 10 # Max continuation attempts +inject_todo_instructions: true # Include the default todo instructions into prompt +``` + +## How It Works + +1. When `inject_todo_instructions` is enabled, agents receive instructions on using five built-in tools: + - `todo__init`: Initialize a todo list with a goal + - `todo__add`: Add a task to the list + - `todo__done`: Mark a task complete + - `todo__list`: View current todo state + - `todo__clear`: Clear the entire todo list and reset the goal + + These instructions are a reasonable default that detail how to use Loki's To-Do System. If you wish, + you can disable the injection of the default instructions and specify your own instructions for how + to use the To-Do System into your main `instructions` for the agent. + +2. When `auto_continue` is enabled and the model stops with incomplete tasks, Loki automatically sends a + continuation prompt with the current todo state, nudging the model to continue working. + +3. This continues until all tasks are done or `max_auto_continues` is reached. + +## When to Use + +- Multistep tasks where the model might lose track +- Smaller models that need more structure +- Workflows requiring guaranteed completion of all steps + +For complete documentation including all configuration options, tool details, and best practices, see the +[Todo System Guide](./TODO-System.md). + +# 7. Sub-Agent Spawning System + +Loki agents can spawn and manage child agents that run **in parallel** as background tasks inside the same process. +This enables orchestrator-style agents that delegate specialized work to other agents, similar to how tools like +Claude Code or OpenCode handle complex multi-step tasks. + +For a working example of an orchestrator agent that uses sub-agent spawning, see the built-in +[sisyphus](https://github.com/Dark-Alex-17/loki/blob/main/assets/agents/sisyphus) agent. For an example of the teammate messaging pattern with parallel sub-agents, +see the [code-reviewer](https://github.com/Dark-Alex-17/loki/blob/main/assets/agents/code-reviewer) agent. + +## Spawning Configuration + +| Setting | Type | Default | Description | +|-----------------------------|---------|---------------|--------------------------------------------------------------------------------| +| `can_spawn_agents` | boolean | `false` | Enable this agent to spawn child agents | +| `max_concurrent_agents` | integer | `4` | Maximum number of child agents that can run simultaneously | +| `max_agent_depth` | integer | `3` | Maximum nesting depth for sub-agents (prevents runaway spawning chains) | +| `inject_spawn_instructions` | boolean | `true` | Inject the default spawning instructions into the agent's system prompt | +| `summarization_model` | string | current model | Model to use for summarizing long sub-agent output (e.g. `openai:gpt-4o-mini`) | +| `summarization_threshold` | integer | `4000` | Character count above which sub-agent output is summarized before returning | +| `escalation_timeout` | integer | `300` | Seconds a sub-agent waits for an escalated user interaction response | + +**Example configuration:** +```yaml +# agents/my-orchestrator/config.yaml +can_spawn_agents: true +max_concurrent_agents: 6 +max_agent_depth: 2 +inject_spawn_instructions: true +summarization_model: openai:gpt-4o-mini +summarization_threshold: 3000 +escalation_timeout: 600 +``` + +## Spawning & Collecting Agents + +When `can_spawn_agents` is enabled, the agent receives tools for spawning and managing child agents: + +| Tool | Description | +|------------------|-------------------------------------------------------------------------| +| `agent__spawn` | Spawn a child agent in the background. Returns an agent ID immediately. | +| `agent__check` | Non-blocking check: is the agent done? Returns `PENDING` or the result. | +| `agent__collect` | Blocking wait: wait for an agent to finish, return its output. | +| `agent__list` | List all spawned agents and their status. | +| `agent__cancel` | Cancel a running agent by ID. | + +The core pattern is **Spawn -> Continue -> Collect**: + +``` +# 1. Spawn agents in parallel (returns IDs immediately) +agent__spawn --agent explore --prompt "Find auth middleware patterns in src/" +agent__spawn --agent explore --prompt "Find error handling patterns in src/" + +# 2. Continue your own work while they run + +# 3. Check if done (non-blocking) +agent__check --id agent_explore_a1b2c3d4 + +# 4. Collect results when ready (blocking) +agent__collect --id agent_explore_a1b2c3d4 +agent__collect --id agent_explore_e5f6g7h8 +``` + +Any agent defined in your `/agents/` directory can be spawned as a child. Child agents: +- Run in a fully isolated environment (separate session, config, and tools) +- Have their output suppressed from the terminal (no spinner, no tool call logging) +- Return their accumulated output to the parent when collected + +## Task Queue with Dependencies + +For complex workflows where tasks have ordering requirements, the spawning system includes a dependency-aware +task queue: + +| Tool | Description | +|------------------------|-----------------------------------------------------------------------------| +| `agent__task_create` | Create a task with optional dependencies and auto-dispatch agent. | +| `agent__task_list` | List all tasks with their status, dependencies, and assignments. | +| `agent__task_complete` | Mark a task done. Returns newly unblocked tasks and auto-dispatches agents. | +| `agent__task_fail` | Mark a task as failed. Dependents remain blocked. | + +``` +# Create tasks with dependency ordering +agent__task_create --subject "Explore existing patterns" +agent__task_create --subject "Implement feature" --blocked_by ["task_1"] +agent__task_create --subject "Write tests" --blocked_by ["task_2"] + +# Mark tasks complete to unblock dependents +agent__task_complete --task_id task_1 +``` + +## Active Task Dispatch + +Tasks can optionally specify an agent to auto-spawn when the task becomes runnable: + +``` +agent__task_create \ + --subject "Implement the auth module" \ + --blocked_by ["task_1"] \ + --agent coder \ + --prompt "Implement auth module based on patterns found in task_1" +``` + +When `task_1` completes and the dependent task becomes unblocked, an agent is automatically spawned with the +specified prompt. No manual intervention needed. This enables fully automated multi-step pipelines. + +## Output Summarization + +When a child agent produces long output, it can be automatically summarized before returning to the parent. +This keeps parent context windows manageable. + +- If the output exceeds `summarization_threshold` characters (default: 4000), it is sent through an LLM + summarization pass +- The `summarization_model` setting lets you use a cheaper/faster model for summarization (e.g. `gpt-4o-mini`) +- If `summarization_model` is not set, the parent's current model is used +- The summarization preserves all actionable information: code snippets, file paths, error messages, and + concrete recommendations + +## Teammate Messaging + +All agents (including children) automatically receive tools for **direct sibling-to-sibling messaging**: + +| Tool | Description | +|-----------------------|-----------------------------------------------------| +| `agent__send_message` | Send a text message to another agent's inbox by ID. | +| `agent__check_inbox` | Drain all pending messages from your inbox. | + +This enables coordination patterns where child agents share cross-cutting findings: + +``` +# Agent A discovers something relevant to Agent B +agent__send_message --id agent_reviewer_b1c2d3e4 --message "Found a security issue in auth.rs line 42" + +# Agent B checks inbox before finalizing +agent__check_inbox +``` + +Messages are routed through the parent's supervisor. A parent can message its children, and children can message +their siblings. For a working example of the teammate pattern, see the built-in +[code-reviewer](https://github.com/Dark-Alex-17/loki/blob/main/assets/agents/code-reviewer) agent, which spawns file-specific reviewers that share +cross-cutting findings with each other. + +## Runaway Safeguards + +The spawning system includes built-in safeguards to prevent runaway agent chains: + +- **`max_concurrent_agents`:** Caps how many agents can run at once (default: 4). Spawn attempts beyond this + limit return an error asking the agent to wait or cancel existing agents. +- **`max_agent_depth`:** Caps nesting depth (default: 3). A child agent spawning its own child increments the + depth counter. Attempts beyond the limit are rejected. +- **`can_spawn_agents`:** Only agents with this flag set to `true` can spawn children. By default, spawning is + disabled. This means child agents cannot spawn their own children unless you explicitly create them with + `can_spawn_agents: true` in their config. + +# 8. User Interaction Tools + +Loki includes built-in tools for agents (and the REPL) to interactively prompt the user for input. These tools +are **always available**. No configuration needed. They are automatically injected into every agent and into +REPL mode when function calling is enabled. + +## User Interaction Available Tools + +| Tool | Description | Returns | +|------------------|-----------------------------------------|----------------------------------| +| `user__ask` | Present a single-select list of options | The selected option string | +| `user__confirm` | Ask a yes/no question | `"yes"` or `"no"` | +| `user__input` | Request free-form text input | The text entered by the user | +| `user__checkbox` | Present a multi-select checkbox list | Array of selected option strings | + +**Parameters:** + +- `user__ask`: `--question "..." --options ["Option A", "Option B", "Option C"]` +- `user__confirm`: `--question "..."` +- `user__input`: `--question "..."` +- `user__checkbox`: `--question "..." --options ["Option A", "Option B", "Option C"]` + +At the top level (depth 0), these tools render interactive terminal prompts directly using arrow-key navigation, +checkboxes, and text input fields. + +## Escalation (Sub-Agent to User) + +When a **child agent** (depth > 0) calls a `user__*` tool, it cannot prompt the terminal directly. Instead, +the request is **automatically escalated** to the root agent: + +1. The child agent calls `user__ask(...)` and **blocks**, waiting for a reply +2. The root agent sees a `pending_escalations` notification in its next tool results +3. The root agent either answers from context or prompts the user itself, then calls + `agent__reply_escalation` to unblock the child +4. The child receives the reply and continues + +The escalation timeout is configurable via `escalation_timeout` in the agent's `config.yaml` (default: 300 +seconds / 5 minutes). If the timeout expires, the child receives a fallback message asking it to use its +best judgment. + +| Tool | Description | +|---------------------------|--------------------------------------------------------------------------| +| `agent__reply_escalation` | Reply to a pending child escalation, unblocking the waiting child agent. | + +This tool is automatically available to any agent with `can_spawn_agents: true`. + +# 9. Auto-Injected Prompts + +Loki automatically appends usage instructions to your agent's system prompt for each enabled built-in system. +These instructions are injected into both **static and dynamic instructions** after your own instructions, +ensuring agents always know how to use their available tools. + +| System | Injected When | Toggle | +|--------------------|----------------------------------------------------------------|-----------------------------| +| Todo tools | `auto_continue: true` AND `inject_todo_instructions: true` | `inject_todo_instructions` | +| Spawning tools | `can_spawn_agents: true` AND `inject_spawn_instructions: true` | `inject_spawn_instructions` | +| Teammate messaging | Always (all agents) | None (always injected) | +| User interaction | Always (all agents) | None (always injected) | + +If you prefer to write your own instructions for a system, set the corresponding `inject_*` flag to `false` +and include your custom instructions in the agent's `instructions` field. The built-in tools will still be +available; only the auto-injected prompt text is suppressed. + +# Built-In Agents +Loki comes packaged with some useful built-in agents: + +* `coder`: An agent to assist you with all your coding tasks +* `code-reviewer`: A [CodeRabbit](https://coderabbit.ai)-style code reviewer that spawns per-file reviewers using the teammate messaging pattern +* `demo`: An example agent to use for reference when learning to create your own agents +* `explore`: An agent designed to help you explore and understand your codebase +* `file-reviewer`: An agent designed to perform code-review on a single file (used by the `code-reviewer` agent) +* `jira-helper`: An agent that assists you with all your Jira-related tasks +* `oracle`: An agent for high-level architecture, design decisions, and complex debugging +* `sisyphus`: A powerhouse orchestrator agent for writing complex code and acting as a natural language interface for your codebase (similar to ClaudeCode, Gemini CLI, Codex, or OpenCode). Uses sub-agent spawning to delegate to `explore`, `coder`, and `oracle`. +* `sql`: A universal SQL agent that enables you to talk to any relational database in natural language diff --git a/docs/Environment-Variables.md b/docs/Environment-Variables.md new file mode 100644 index 0000000..5a225be --- /dev/null +++ b/docs/Environment-Variables.md @@ -0,0 +1,101 @@ +Loki is designed to be highly dynamic and customizable. As a result, Loki utilizes a number of environment variables +that can be used to modify its behavior at runtime without needing to modify the existing configuration files. + +Loki also supports defining environment variables via a `.env` file in the Loki configuration directory. This directory +varies between systems, so you can find the location of your configuration directory using the following command: + +```shell +loki --info | grep 'config_dir' | awk '{print $2}' +``` + +--- + +# Global Configuration Related Variables +All configuration items in the global config file have environment variables that can be overridden at runtime. To see +all configuration options and more thorough descriptions, refer to the [example config file](https://github.com/Dark-Alex-17/loki/blob/main/config.example.yaml). + +Below are the most commonly used configuration settings and their corresponding environment variables: + +| Setting | Environment Variable | +|----------------------------|---------------------------------| +| `model` | `LOKI_MODEL` | +| `temperature` | `LOKI_TEMPERATURE` | +| `top_p` | `LOKI_TOP_P` | +| `stream` | `LOKI_STREAM` | +| `save` | `LOKI_SAVE` | +| `editor` | `LOKI_EDITOR` | +| `wrap` | `LOKI_WRAP` | +| `wrap_code` | `LOKI_WRAP_CODE` | +| `save_session` | `LOKI_SAVE_SESSION` | +| `compression_threshold` | `LOKI_COMPRESSION_THRESHOLD` | +| `function_calling_support` | `LOKI_FUNCTION_CALLING_SUPPORT` | +| `enabled_tools` | `LOKI_ENABLED_TOOLS` | +| `mcp_server_support` | `LOKI_MCP_SERVER_SUPPORT` | +| `enabled_mcp_servers` | `LOKI_ENABLED_MCP_SERVERS` | +| `rag_embedding_model` | `LOKI_RAG_EMBEDDING_MODEL` | +| `rag_reranker_model` | `LOKI_RAG_RERANKER_MODEL` | +| `rag_top_k` | `LOKI_RAG_TOP_K` | +| `rag_chunk_size` | `LOKI_RAG_CHUNK_SIZE` | +| `rag_chunk_overlap` | `LOKI_RAG_CHUNK_OVERLAP` | +| `highlight` | `LOKI_HIGHLIGHT` | +| `theme` | `LOKI_THEME` | +| `serve_addr` | `LOKI_SERVE_ADDR` | +| `user_agent` | `LOKI_USER_AGENT` | +| `save_shell_history` | `LOKI_SAVE_SHELL_HISTORY` | +| `sync_models_url` | `LOKI_SYNC_MODELS_URL` | + + +# Client Related Variables +The following environment variables are available for clients in Loki: + +| Environment Variable | Description | +|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `{client}_API_KEY` | For clients that require an API key, you can define the keys either through environment variables or
using the [vault](./Vault.md). The variables are named after the client to which they apply;
e.g. `OPENAI_API_KEY`, `GEMINI_API_KEY`, etc. | +| `LOKI_PLATFORM` | Combine with `{client}_API_KEY` to run Loki without a configuration file.
This variable is ignored if a configuration file exists. | +| `LOKI_PATCH_{client}_CHAT_COMPLETIONS` | Patch chat completion requests to models on the corresponding client; Can modify the URL, body,
or headers. | +| `LOKI_SHELL` | Specify the shell that Loki should be using when executing commands | + +# Files and Directory Related Variables +You can also customize the files and directories that Loki loads its configuration files from: + +| Environment Variable | Description | Default Value | +|----------------------|------------------------------------------------------------------------|---------------------------------| +| `LOKI_CONFIG_DIR` | Customize the location of the Loki configuration directory. | `/loki` | +| `LOKI_ENV_FILE` | Customize the location of the `.env` file to load at startup. | `/.env` | +| `LOKI_CONFIG_FILE` | Customize the location of the global `config.yaml` configuration file. | `/config.yaml` | +| `LOKI_ROLES_DIR` | Customize the location of the `roles` directory. | `/roles` | +| `LOKI_SESSIONS_DIR` | Customize the location of the `sessions` directory. | `/sessions` | +| `LOKI_RAGS_DIR` | Customize the location of the `rags` directory. | `/rags` | +| `LOKI_FUNCTIONS_DIR` | Customize the location of the `functions` directory. | `/functions` | + +# Agent Related Variables +You can also customize the location of full agent configurations using the following environment variables: + +| Environment Variable | Description | +|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| +| `_CONFIG_FILE` | Customize the location of the agent's configuration file; e.g. `SQL_CONFIG_FILE` | +| `_MODEL` | Customize the `model` used for the agent; e.g `SQL_MODEL` | +| `_TEMPERATURE` | Customize the `temperature` used for the agent; e.g. `SQL_TEMPERATURE` | +| `_TOP_P` | Customize the `top_p` used for the agent; e.g. `SQL_TOP_P` | +| `_GLOBAL_TOOLS` | Customize the `global_tools` that are enabled for the agent (a JSON string array); e.g. `SQL_GLOBAL_TOOLS` | +| `_MCP_SERVERS` | Customize the `mcp_servers` that are enabled for the agent (a JSON string array); e.g. `SQL_MCP_SERVERS` | +| `_AGENT_SESSION` | Customize the `agent_session` used with the agent; e.g. `SQL_SESSION` | +| `_INSTRUCTIONS` | Customize the `instructions` for the agent; e.g. `SQL_INSTRUCTIONS` | +| `_VARIABLES` | Customize the `variables` used for the agent (in JSON format of `[{"key1": "value1", "key2": "value2"}]`);
e.g. `SQL_VARIABLES` | + +# Logging Related Variables +The following variables can be used to change the log level of Loki or the location of the log file: + +| Environment Variable | Description | Default Value | +|----------------------|---------------------------------------------|----------------------------------| +| `LOKI_LOG_LEVEL` | Customize the log level of Loki | `INFO` | +| `LOKI_LOG_FILE` | Customize the location of the Loki log file | `/loki/loki.log` | + +**Pro-Tip:** You can always tail the Loki logs using the `--tail-logs` flag. If you need to disable color output, you +can also pass the `--disable-log-colors` flag as well. + +# Miscellaneous Variables +| Environment Variable | Description | Default Value | +|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------| +| `AUTO_CONFIRM` | Bypass all `guard_*` checks in the bash prompt helpers; useful for agent composition and routing | | +| `LLM_TOOL_DATA_FILE` | Set automatically by Loki on Windows. Points to a temporary file containing the JSON tool call data.
Tool scripts (`run-tool.sh`, `run-agent.sh`, etc.) read from this file instead of command-line args
to avoid JSON escaping issues when data passes through `cmd.exe` → bash. **Not intended to be set by users.** | | diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index be79602..80965c3 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -13,7 +13,7 @@ loki --list-secrets # Authentication Each client in your configuration needs authentication (with a few exceptions; e.g. ollama). Most clients use an API key -(set via `api_key` in the config or through the [vault](./docs/VAULT.md)). For providers that support OAuth (e.g. Claude Pro/Max +(set via `api_key` in the config or through the [vault](./Vault.md)). For providers that support OAuth (e.g. Claude Pro/Max subscribers, Google Gemini), you can authenticate with your existing subscription instead: ```yaml @@ -29,7 +29,7 @@ loki --authenticate my-claude-oauth # Or via the REPL: .authenticate ``` -For full details, see the [authentication documentation](./docs/clients/CLIENTS.md#authentication). +For full details, see the [authentication documentation](./clients/Clients.md#authentication). # Configuration The location of the global Loki configuration varies between systems, so you can use the following command to find your @@ -40,7 +40,7 @@ loki --info | grep 'config_file' | awk '{print $2}' ``` The configuration file consists of a number of settings. To see a full example configuration file with every setting -defined, refer to the [example configuration file](./config.example.yaml). +defined, refer to the [example configuration file](https://github.com/Dark-Alex-17/loki/blob/main/config.example.yaml). ## Default LLM The following settings are available to configure the default LLM that is used when you start Loki, and its @@ -70,7 +70,7 @@ shown below: | Setting | Description | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `repl_prelude` | This setting lets you specify a default `session` or `role` to use when starting Loki in [REPL](./docs/REPL.md) mode.
Values can be
  • `role:` to define a role
  • `session:` to define a session
  • `:` to define both a session and a role to use
| +| `repl_prelude` | This setting lets you specify a default `session` or `role` to use when starting Loki in [REPL](./REPL.md) mode.
Values can be
  • `role:` to define a role
  • `session:` to define a session
  • `:` to define both a session and a role to use
| | `cmd_prelude` | This setting lets you specify a default `session` or `role` to use when running one-off queries in Loki via the CLI.
Values can be
  • `role:` to define a role
  • `session:` to define a session
  • `:` to define both a session and a role to use
| | `agent_session` | This setting is used to specify a default session that all agents should start into, unless otherwise specified in the agent configuration. (e.g. `temp`, `default`) | diff --git a/docs/Macros.md b/docs/Macros.md new file mode 100644 index 0000000..73d69d8 --- /dev/null +++ b/docs/Macros.md @@ -0,0 +1,94 @@ +Macros are essentially Loki "scripts"; that is, a predefined sequence of REPL commands that automate repetitive tasks or +workflows. Macros run in isolated environments, ensuring that the macros don't inherit any pre-existing role, session, +RAG, or agent state, and they will not affect your current context. + +This isolation ensures that your workspace remains clean and unaffected by macro operations. + +![Macro Example](./images/macros/macros-example.gif) + +For more information on Loki's REPL, refer to the [REPL](./REPL.md) documentation. + +--- + +# Macro Definition +Macros are defined as YAML files in the `macros` subdirectory of your Loki configuration directory. The Loki configuration +directory can vary between systems, so to find the location of your macros config directory, you can use the following +command: + +```shell +loki --info | grep 'macros_dir' | awk '{print $2}' +``` + +Macro definitions are broken into two parts: the `steps` of the macro, and an optional `variables` section that lets +users pass in variables to alter the behavior of the macro at runtime. + +## Step Definitions +The step definitions for a macro are straightforward: They are simply the exact commands you would otherwise type in the +REPL. + +**Example: Macro to generate a git commit message** +`macros/generate-commit-message.yaml` +```yaml +steps: + - .file `git diff` -- generate git commit message +``` +Usage: +```shell +$ loki --macro generate-commit-message +>> .file `git diff` -- generate a git commit message +Add documentation on macros +``` + +For a full example configuration, refer to the [example macro configuration file](https://github.com/Dark-Alex-17/loki/blob/main/config.macro.example.yaml) in the root of this project. + +## Macro Variables +Sometimes it's useful to be able to modify the behavior of a macro at runtime. This is achieved with the `variables` +array of the macro definition. + +To pass variables to a macro, since they are just Loki scripts, the syntax is the same as it is for any other scripting +language: You just pass them alongside your invocation. + +**Example:** +```shell +$ loki --macro example-variable-macro first_argument second_argument +``` + +Each variable in the `variables` array has the following properties: +* `name` (Required): the name of the variable, which can be referenced in the actual steps of the macro using the + `{{name}}` syntax. +* `default` (Optional): A default value for the variable if no value is specified. If no default value is defined, and + no value is provided for the variable at runtime, Loki will error out. +* `rest` (Optional, Boolean): When set to `true`, this variable will collect all remaining arguments passed to the + macro. This behavior is only applicable when the variable is the last variable in the list. By default, this is + `false`. + +The `variables` array is order-dependent; that is to say that all arguments passed to the macro are positional. So be +careful about the ordering if that is important to your macro's invocation. + +**Example: Simple variable example to invoke an agent** +`macros/invoke-agent.yaml` +```yaml +variables: + - name: agent # No default value means this must be defined at runtime + - name: args + rest: true # All remaining arguments to the macro are collected into this variable + default: What can you do? # This is used if no value is passed at runtime +steps: + - .agent {{agent}} + - '{{args}}' +``` +Usage: +```shell +$ loki --macro invoke-agent sql +# or +$ loki --macro invoke-agent sql What tables are available? +``` + +For a full example configuration, refer to the [example macro configuration file](https://github.com/Dark-Alex-17/loki/blob/main/config.macro.example.yaml) in the root of this project. + +# Built-In Macros +Loki comes packaged with some useful built-in macros. These are also good examples if you're looking for more examples +on how to make your own macros, so be sure to check out the [built-in macro definitions](https://github.com/Dark-Alex-17/loki/blob/main/assets/macros) if you're +looking for more examples. + +* `generate-commit-message` - Generate a Git commit message based on the staged changes in the current directory diff --git a/docs/RAG.md b/docs/RAG.md new file mode 100644 index 0000000..811b953 --- /dev/null +++ b/docs/RAG.md @@ -0,0 +1,282 @@ +Retrieval Augmented Generation (RAG) is a method of minimizing LLM hallucinations and extending the model's context +without consuming a significant portion of the context length. It uses documents and other additional resources that you +provide to give the model more context for all of your prompts. + +Loki has a built-in vector database and full-text search engine to support RAG knowledge bases for your queries. + +The generated knowledge bases are stored in the `rag` subdirectory of your Loki configuration directory. The location of +this directory varies by system, so you can use the following command to find your RAG directory: + +```shell +loki --info | grep 'rags_dir' | awk '{print $2}' +``` + +--- + +# Usage +There's two ways to use RAG in Loki: A persistent RAG that can be loaded on-demand for queries, and an ephemeral one for +adding RAG to a single specific query. + +## Persistent RAG +In the REPL, persistent RAG is initialized via the `.rag` command: + +![Persistent RAG example](./images/rag/persistent-rag.gif) + +The generated RAG is then saved to the `rag` subdirectory of the Loki configuration, and can then be loaded whenever you +want that knowledge base via either `.rag ` or `loki --rag `. + +## Ephemeral RAG +Short-lived RAG that is only used for a single session or query is loaded using `.file`/`--file`. + +You can use it to either execute a prompt from a file, or for temporary RAG. The difference is the usage of the `--` +separator. If you only specify a filename and no `--` separator, Loki will know to read the file contents and pass them +as a query to the model. Otherwise, the `--` separator is read to indicate that this is the end of the list of documents +to load into the ephemeral RAG, and what follows is the query to pass to the model. + +```shell +.file prompt.md # Read the file as a prompt +.file %% -- translate the last reply to italian +.file `git diff` -- generate a commit message +``` + +![Ephemeral RAG Example](./images/rag/ephemeral-rag.gif) + +Once the session ends, this RAG will no longer be accessible and is only visible to the current session. + +### The `%%` Document Type +In addition to the usual documents that can be specified for persistent RAG, ephemeral RAG has a special `%%` value. +This value references the content of the last reply. So you can use it like this: + +```shell +.file %% -- translate the last reply to italian +``` + +The `--` indicates that this is the end of your documents and the beginning of your prompt. + +### The `cmd` Document Type +Loki also lets you use command outputs for ephemeral RAG input. Simply enclose the command in backticks: + +```shell +.file `git diff` -- generate a commit message +``` + +The `--` indicates that this is the end of your documents and the beginning of your prompt. + +# How It Works +### 1. Build +When you define RAG, Loki will first "build" the RAG. This means that Loki will consume the documents you specified and +generate [embeddings](https://huggingface.co/spaces/hesamation/primer-llm-embedding) for that text. This essentially just means that Loki translates the document into a language +the LLM can understand. + +These embeddings are then stored in an in-memory vector database. + +### 2. Lookup +Loki sits between you and the model. So when you submit a prompt to the model, before Loki ever sends it, it will first +convert your prompt into embeddings (LLM language), and look for relevant snippets of text in the vector database. + +Loki then passes the top `n`-snippets of text that it finds in the vector database as additional context to the model +before your prompt. + +### 2a. Reranking (Optional) +The lookup for relevant snippets of texts uses embeddings to find text that is semantically similar to your prompt, and +returns the top `n`-results. This often works fairly well, however these top results aren't always the most relevant for +answering the specific query. + +Reranking improves these initial results (say, the top 20-100 text snippets) and re-scores them using a more +sophisticated model. The reranker model will rank documents by their actual usefulness for answering the query to ensure +the most relevant context is passed to the model alongside your query. + +This reranking model can be customized for each RAG you build in Loki. See the [Custom Reranker](#reranker) section +below for more details on how to customize this. + +### 3. Prompt +Finally, the text snippets that were looked up in RAG are passed to the model as additional context to your prompt, +giving the model query-specific context to answer your question. + +# Supported Document Sources +Loki supports a number of document sources that can be used for RAG: + +| Source | Example | Comments | +|--------------------------|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| +| Files | `/tmp/dir1/file1;/tmp/dir1/file2` | | +| Directory | `/tmp/dir` | Picks up all files in a directory and all its subdirectories | +| Directory (extensions} | `/tmp/dir2/**/*.{md,txt}` | Finds all files in all subdirectories with the specified extensions | +| Recursive Filename | `/tmp/*/LOKI.md` | The following files will be picked up:
  • `/tmp/dir1/LOKI.md`
  • `/tmp/dir2/subdir1/LOKI.md`
  • `/tmp/dir2/subdir2/LOKI.md`
| +| URL | `https://www.ohdsi.org/data-standardization/` | Downloads and loads the specified webpage into the
knowledge base | +| Recursive URL (Websites) | `https://github.com/OHDSI/Vocabulary-v5.0/wiki/**` | Crawls all pages under the given URL and loads them
into the knowledge base | +| Document Loader (custom) | `jina:https://cloud.google.com/bigquery/docs/reference/standard-sql/` | Use a custom document loader to parse the given document | + +# Document Loaders +Loki only has built-in support for loading text files. But that functionality can be extended to read all kinds of files +into your knowledge bases. These custom loaders are used by both RAG and for documents specified using the +`.file`/`--file` flags. + +In the global configuration file, you can specify loaders for specific document types using the `document_loaders` +setting. Each loader is defined by specifying a name and then a command that Loki will execute to load the document. + +The following variables are interpolated at runtime by Loki and can be used as placeholders in your command definitions: +* `$1` (Required) - The input file +* `$2` (Optional) - The output file. If omitted, `stdout` is used as the output destination + +**Note:** It is your responsibility to ensure that any tools used to parse documents into text that Loki can read are +installed on your system and are available on your `$PATH`. Loki does not have any built-in way of installing +dependencies for document loaders for you. + +The following are some example loaders: +```yaml +document_loaders: + pdf: 'pdftotext $1 -' # Use pdftotext to convert a PDF file to text + # (see https://poppler.freedesktop.org for details on how to install pdftotext) + docx: 'pandoc --to plain $1' # Use pandoc to convert a .docx file to text + # (see https://pandoc.org for details on how to install pandoc) + jina: 'curl -fsSL https://r.jina.ai/$1 -H "Authorization: Bearer {{JINA_API_KEY}}' # Use Jina to translate a website into text; + # Requires a Jina API key to be added to the Loki vault + git: > # Use yek to load a git repository into the knowledgebase (https://github.com/bodo-run/yek) + sh -c "yek $1 --json | jq 'map({ path: .filename, contents: .content })'" +``` + +## Document Loader Usage +Once you have your loaders defined, you can specify when Loki should use them by prefixing any RAG file/directory/URI +with the name of the loader. + +**Example: Load a git repo into RAG** +![Git Repo Loader Example](./images/rag/git-loader.png) + +**Example: Use pdf loader for ephemeral RAG** +```shell +$ loki --file pdf:some-file.pdf +``` + +# Advanced Customizations +For those familiar with RAG, Loki exposes a handful of advanced global settings that can be used to tweak your default +RAG configurations. + +## Embedding Model +When Loki queries your RAG knowledge bases, it needs to first convert your query into embeddings. By default, Loki uses +the same embedding model that was used to create the knowledge base in the first place. + +This can be customized to any other embedding model available in your configured clients by setting the +`rag_embedding_model` setting in your global Loki configuration file: + +```yaml +rag_embedding_model: null # Specifies the embedding model used for context retrieval +``` + +## Reranker +By default, Loki uses [Reciprocal Rank Fusion (RRF)](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion) to merge vector and keyword search results. + +You can change the default reranker model to any other reranking model in your configured clients. To change the default +reranker model, simply change the value of the `rag_reranker_model` setting in your global configuration file: + +```yaml +rag_reranker_model: null # By default, +``` + +## Chunk Size +In the context of RAG, the chunk size is the maximum length of each text chunk (measured in characters) that is created +when splitting documents. In Loki, this defaults to `2000` characters. + +You can specify a different global default by setting the `rag_chunk_size` property in your global configuration file: + +```yaml +rag_chunk_size: null # Defines the size of chunks for document processing in characters +``` + +### Chunk Size Trade-Offs +Keep in mind the following trade-offs when changing the chunk size: + +* **Smaller chunks (e.g. 256 characters):** More precise retrieval, better semantic focus, but may lack context or split + important information +* **Larger chunks (e.g. 1024 characters):** More context preserved, fewer chunks to manage, but less precise matching + and more noise in retrieved document + +## Chunk Overlap +Chunk overlap in RAG is the number of characters that overlap between consecutive chunks to maintain continuity. + +--- + +**Example:** If the following sentence is cut off at the end of one chunk + +`I was doing fine until someone brought up` + +You'll ideally want that full sentence to be picked up at the beginning of the next chunk to make sure the full meaning +is captured. So in this example, if your chunk overlap is 42 characters, then the start of the next chunk would look +like this: + +`I was doing fine until someone brought up the game. ` + +--- + +Often, this value is 10%-20% of the chunk size. + +By default, in Loki, this value is 5% the chunk size. You can override this and specify the default chunk overlap (in +characters) that Loki should use as a global default by setting the `rag_chunk_overlap` property in the global Loki +configuration file: + +```yaml +rag_chunk_overlap: null # Defines the overlap between chunks +``` + +## Top K +In RAG, `top_k` represents the top `k`-chunks to return from the vector database query. Think of it like if you search +something on Google and only care about the top 10 results, that's what you'll use for your context. + +In Loki, the default value for this is `5`. You can customize this global default by setting the `rag_top_k` property in +your global configuration file: + +```yaml +rag_top_k: 5 # Specifies the number of documents to retrieve for answering queries +``` + +### Top K Trade-Offs +When customizing this value, keep in mind the following trade-offs so you get the best performance: + +* **Lower top_k (e.g. 3):** Faster, more focused context, lower cost, but risks missing relevant information +* **Higher top_k (e.g. 10):** More comprehensive coverage, but more noise, higher latency, increased token costs, and + potential context window constraints + +## RAG Template +When you use RAG in Loki, after Loki performs the lookup for relevant chunks of text to add as context to your query, it +will add the retrieved text chunks as context to your query before sending it to the model. The format of this context +is determined by the `rag_template` setting in your global Loki configuration file. + +This template utilizes three placeholders: +* `__INPUT__`: The user's actual query +* `__CONTEXT__`: The context retrieved from RAG +* `__SOURCES__`: A numbered list of the source file paths or URLs that the retrieved context came from + +These placeholders are replaced with the corresponding values into the template and make up what's actually passed to +the model at query-time. The `__SOURCES__` placeholder enables the model to cite which documents its answer is based on, +which is especially useful when building knowledge-base assistants that need to provide verifiable references. + +The default template that Loki uses is the following: + +```text +Answer the query based on the context while respecting the rules. (user query, some textual context and rules, all inside xml tags) + + +__CONTEXT__ + + + +__SOURCES__ + + + +- If you don't know, just say so. +- If you are not sure, ask for clarification. +- Answer in the same language as the user query. +- If the context appears unreadable or of poor quality, tell the user then answer as best as you can. +- If the answer is not in the context but you think you know the answer, explain that to the user then answer with your own knowledge. +- Answer directly and without using xml tags. +- When using information from the context, cite the relevant source from the section. + + + +__INPUT__ + +``` + +You can customize this template by specifying the `rag_template` setting in your global Loki configuration file. Your +template *must* include both the `__INPUT__` and `__CONTEXT__` placeholders in order for it to be valid. The +`__SOURCES__` placeholder is optional. If it is omitted, source references will not be included in the prompt. diff --git a/docs/REPL-Prompt.md b/docs/REPL-Prompt.md new file mode 100644 index 0000000..c8edbd8 --- /dev/null +++ b/docs/REPL-Prompt.md @@ -0,0 +1,109 @@ +The prompt you see when you start the Loki REPL can be customized to your liking. This is achieved via the `left_prompt` +and `right_prompt` settings in the global Loki configuration file: + +```yaml +left_prompt: '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} ' +right_prompt: '{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}' +``` + +The location of the global configuration file differs between systems, so you can use the following command to find your +global configuration file's location: + +```shell +loki --info | grep 'config_file' | awk '{print $2}' +``` + +# Syntax +The syntax for the prompts consists of plain text and templates contained in `{...}`. The plain text is +printed exactly as given. + +The syntax for the templates `{...}` is as follows: + +* `{variable}` - Replaced with the value of `variable` +* `{?variable