Dark-Alex-17/coyote
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
Files
e82e5ab8e41bb7743669a452ab7d96fc0558d9bf
coyote/.sisyphus/plans/rich-markdown-renderer.md
T

18 KiB
Raw Blame History

Rich Markdown Renderer for the REPL

Status: Planning complete, awaiting Momus review before implementation. Owner: Coyote maintainer Estimated effort: Phase 1 = 4-5 days, Phase 2 (tables) = +1-2 days Related flag: raw_markdown (already plumbed; see commit history for the plumbing PR)

Goal

Replace Coyote's current syntect-only markdown rendering with a rich renderer that transforms markdown syntax into styled terminal output (headings become colored + bold text, **bold** becomes actual bold, backticks strip and stylize, blockquotes get a │ prefix, etc.), matching glamour's structural output while preserving the user's existing syntect .tmTheme colors.

The current renderer just applies syntect's markdown grammar for syntax highlighting — the markdown syntax characters (#, **, `) stay in the output, just colored. Users get raw markdown with color, not rendered markdown. The new renderer actually transforms the markdown into styled output like glamour (github.com/charmbracelet/glamour) does.

Non-Goals

  • Not replacing the renderer's public API. MarkdownRender::init, render, render_line, and RenderOptions all keep their existing signatures. Callers (stream.rs, session.rs, app_config.rs::print_markdown, request_context.rs::session_info) do not change.
  • Not changing streaming architecture. stream.rs still calls render() on complete lines and render_line() on the incomplete tail. New renderer must fit this line-by-line contract.
  • Not touching code block rendering. Fenced code blocks (```lang ... ```) continue to route to syntect language-specific highlighting via find_syntax_by_token. The new renderer only affects markdown syntax rendering, never code content.
  • Not adding new dependencies. All work uses existing syntect, fancy-regex, crossterm, textwrap.
  • Not shipping tables in Phase 1. Tables require multi-line buffering, which conflicts with the stateless streaming model. Table rows render as raw | col | col | until Phase 2.
  • Not implementing OSC 8 hyperlink fallback logic. Emit OSC 8 codes unconditionally + always show URL visibly. Terminals that don't support OSC 8 strip the codes and see plain "text URL" text.

Design Principles

  1. Colors from user theme, layout from glamour. Every construct extracts its color from the user's syntect theme via scope lookup with fallback chains. The structural layout (prefixes, indents, borders, box-drawing) matches glamour's default dark style.
  2. raw_markdown: true = current behavior byte-identical. The existing syntect-on-markdown-grammar path is preserved as the "raw" branch and reachable via config/CLI/REPL. Zero regression risk for users who want the old behavior.
  3. Preserve line-by-line rendering. No state beyond the existing LineType code-block tracker. Stateless per-line rendering means the streaming's render_line for partial buffer works identically to the mutating render_line_mut for complete lines.
  4. Regex-based inline parsing, not pulldown-cmark. A full markdown parser needs the complete document to disambiguate. Regexes match balanced spans and gracefully leave unclosed spans as raw text — exactly right for streaming's mid-token partial-line rendering.
  5. Only src/render/markdown.rs changes. Scope containment: the entire implementation lives in one file. No touches to stream.rs, mod.rs, session.rs, app_config.rs, request_context.rs.

Resolved Design Decisions

Recorded here so future sessions don't re-litigate them:

  1. Tables: deferred to Phase 2. Phase 1 leaves table rows as raw markdown.
  2. H2-H6 hash prefixes: matched to glamour — keep ##, ###, ####, #####, ###### visible in the heading color as a level indicator. H1 gets padded text treatment.
  3. Link rendering: OSC 8 hyperlink codes wrapping visible {text} {url} — modern terminals show a clickable link, older terminals show plain styled text. Matches glamour exactly. Users on broken terminals can fall back to .set raw_markdown true.

Architecture

Data structures (added to MarkdownRender)

struct MarkdownStyles {
    heading: (Color, bool /* force_bold */),
    bold: Color,
    italic: Color,
    inline_code_fg: Color,
    inline_code_bg: Option<Color>,
    blockquote: Color,
    list_bullet: Color,
    link_text: Color,
    link_url: Color,
    strikethrough: Color,
    hrule: Color,
}

Populated once in MarkdownRender::init via a new resolve_scope_style(theme, primary_scope, fallbacks) helper that generalizes the existing get_code_color() pattern (markdown.rs:299).

When options.theme.is_none(), all styles collapse to defaults (raw text output with no colors — matches current behavior).

Line-type detection

Extended check_line returns a new LineKind enum (only for non-code lines):

Regex LineKind
^\s*(#{1,6}) +.+ Heading(level)
^\s*> ?.* Blockquote
^(\s*)- \[[ xX]\] +.+ TaskItem(checked)
^(\s*)[-*+] +.+ BulletItem
^(\s*)\d+\. +.+ NumberedItem
`^\s*(-{3,} _{3,}
^\s*|.*|\s*$ (Phase 2: TableRow) — treated as paragraph for now
default Paragraph

Stateless: line-type detection carries no state beyond the existing prev_line_type/code_syntax fields for code block tracking. Streaming's partial-line render_line works identically to complete-line render_line_mut.

Block-level rendering

Each LineKind triggers a block transformation that strips syntax markers and applies structural styling. All block types then run their remaining text content through the inline pipeline.

LineKind Transformation
Heading(1) Prefix , suffix (single spaces), apply bold + heading color to entire line
Heading(2..=6) Keep visible ##/###/etc. prefix, apply bold + heading color
Blockquote Replace > with │ (styled blockquote color); apply blockquote color to remaining content
BulletItem Replace -/*/+ with • (styled list_bullet color); preserve leading whitespace for nesting
NumberedItem Preserve number, style the . in list_bullet color
TaskItem(false) Replace [ ] with [ ] styled in list_bullet color
TaskItem(true) Replace [x] with [✓] styled
HorizontalRule Emit ──────── (8-char box-drawing) styled with hrule color (typically dim/gray)
Paragraph No block transform, inline pass only

Inline rendering (regex pipeline, applied in order)

Order matters — inline code first prevents re-parsing code content as bold/italic:

  1. Inline code (`text`) — regex `([^`\n]+)`, strip backticks, apply inline_code_fg + optional inline_code_bg.
  2. Images (![alt](url)) — regex !\[([^\]]*)\]\(([^)]+)\), emit Image: {alt} → {url} styled with link_url. Wrap in OSC 8 hyperlink codes.
  3. Links ([text](url)) — regex \[([^\]]+)\]\(([^)]+)\), emit {text} {url} with link_text on the label and link_url on the URL. Wrap in OSC 8 hyperlink codes.
  4. Bold (**text** or __text__) — regex \*\*([^*\n]+)\*\* and __([^_\n]+)__, strip markers, apply bold ANSI + bold color.
  5. Italic (*text* or _text_) — regex (?<![*\w])\*([^*\n]+)\*(?!\*) and (?<![_\w])_([^_\n]+)_(?!_) — lookbehind/lookahead prevents word-internal _ from matching (e.g., some_var_name). fancy-regex supports lookbehind.
  6. Strikethrough (~~text~~) — regex ~~([^~\n]+)~~, strip markers, apply ANSI strikethrough (\x1b[9m).

Partial-span handling for streaming: regexes only match balanced spans. Unclosed spans (**bold with no closing) stay raw. When the closing marker arrives on the next token, the complete-line pass renders the full span correctly.

OSC 8 hyperlinks

\x1b]8;;{url}\x1b\\{visible_text}\x1b]8;;\x1b\\

Emit unconditionally around links and images. Unsupported terminals strip the codes and see plain visible text. Zero degradation.

Branching in highlight_line

fn highlight_line(&self, line: &str, syntax: &SyntaxReference, is_code: bool) -> String {
    if is_code {
        // unchanged — code block content via language-specific syntect
        self.highlight_code_syntect(line, syntax)
    } else if self.options.raw_markdown {
        // preserved current behavior: syntect on markdown grammar
        self.highlight_markdown_syntect(line, &self.md_syntax)
    } else {
        // new rich rendering path
        self.render_markdown_line(line)
    }
}

Code blocks route to syntect regardless of raw_markdown — the flag only affects markdown syntax rendering.

Consumers Verified

Complete map of MarkdownRender consumers (from explore agent research). All continue to work without modification because the public API is unchanged:

  1. src/render/mod.rs:16-33 — render_stream() (streaming path via markdown_stream())
  2. src/render/stream.rs:67-171 — markdown_stream_inner() calls render.render(head) and render.render_line(&buffer)
  3. src/config/app_config.rs:420-429 — print_markdown() (CLI one-shot)
  4. src/config/request_context.rs:1706-1723 — session_info() (.info REPL command)
  5. src/config/session.rs:278-396 — Session::render() (per assistant message)
  6. src/render/markdown.rs:311-397 — existing tests

Phase 1 Implementation

Phase 1.1 — Scope lookup helper + precomputed styles

  • Add resolve_scope_style(theme, primary, fallbacks) helper (generalizes get_code_color())
  • Add MarkdownStyles struct + populate in MarkdownRender::init for all 10 constructs
  • Handle theme.is_none() gracefully (all styles = defaults)
  • Test: verify each style resolves correctly with the built-in dark theme
  • Test: verify each style falls back correctly with a minimal theme that only defines root scopes

Commit: feat(render): precompute markdown scope styles for rich rendering

Phase 1.2 — Line-type detection

  • Add LineKind enum + detect_line_kind() function
  • Wire into check_line — return LineKind alongside existing LineType
  • Test each pattern in isolation (heading, blockquote, bullets, numbered, task, hrule, paragraph)
  • Test edge cases: ## vs ##text (no space, not a heading), indented list items, empty blockquote

Commit: feat(render): detect markdown block-level line types

Phase 1.3 — Inline rendering pipeline

  • Add regex constants (LazyLock) for each inline construct
  • Add apply_inline(text: &str, styles: &MarkdownStyles) -> String that runs the pipeline in order
  • Test each construct in isolation
  • Test order-dependence: **foo bar baz** — bold wraps inline code correctly
  • Test partial spans stay raw: **unclosed → **unclosed
  • Test italic doesn't false-positive: some_var_name, a * b * c (math-like expression)
  • Test OSC 8 emission for links and images

Commit: feat(render): rich inline markdown rendering (bold, italic, code, links)

Phase 1.4 — Block-level rendering

  • Add render_markdown_line(line) that dispatches on LineKind
  • Implement each block transform (heading, blockquote, bullet, numbered, task, hrule, paragraph)
  • After block transform, always run apply_inline on the content
  • Test each block type with inline styling nested inside (bold in heading, code in list item, link in blockquote)

Commit: feat(render): rich block-level markdown rendering (headings, quotes, lists, hr)

Phase 1.5 — Wire into highlight_line with raw_markdown branch

  • Refactor highlight_line to branch on options.raw_markdown
  • Remove #[allow(dead_code)] from RenderOptions::raw_markdown
  • Verify all existing tests pass with raw_markdown: true (byte-identical output)
  • Manual REPL test: send a message with a mix of constructs, verify output matches expectations
  • Manual streaming test: verify no flashing, partial spans render smoothly

Commit: feat(render): activate rich markdown renderer as default

Phase 1.6 — Test coverage

  • Heading levels 1-6 (transforms + styling)
  • Bold, italic, inline code, strikethrough
  • Inline code strips backticks
  • some_var_name NOT italicized
  • a * b * c math not italicized
  • Blockquote │ prefix
  • Bullet • transformation
  • Numbered list preservation
  • Task items [ ] / [✓]
  • Horizontal rule
  • Links: styled text + URL, OSC 8 codes present
  • Images: Image: {alt} → {url} format, OSC 8 codes present
  • Nested inline in blocks (bold in heading, code in list)
  • Partial spans in render_line
  • theme=None degrades to raw stripped text (no colors, but syntax stripped)
  • raw_markdown=true matches current behavior byte-for-byte

Commit: test(render): comprehensive coverage for rich markdown renderer

Phase 2 (Follow-up PR) — Tables

Deferred scope. Rough sketch:

  • Add Option<TableBuffer> field to MarkdownRender
  • On table row detection, accumulate rows in buffer (emit raw markdown for now to keep streaming visible)
  • On non-table line (or blank), flush the buffer: compute column widths, render with box-drawing chars, emit
  • Handle streaming: use cursor-erase to replace raw rows with rendered table when buffer flushes
  • Test coverage: single-column, multi-column, alignment specifiers (:---, ---:, :---:), empty cells, long content wrapping

Success Criteria (Phase 1)

  • All existing tests pass with raw_markdown: true
  • All new tests pass with raw_markdown: false
  • cargo check clean
  • cargo test all pass
  • Manual REPL test: streaming looks smooth (no flashing, no visible partial spans getting re-rendered)
  • Manual REPL test: .set raw_markdown true reverts to current behavior
  • Manual test: user's custom theme colors apply to headings/bold/etc. (not just default)

Progress Log

Append-only. One entry per commit or session.

2026-07-22 — Planning complete

  • Scoped implementation via research (glamour source, syntect scope conventions, current renderer consumers)
  • Resolved 3 open design questions (tables deferred, glamour hash prefixes matched, OSC 8 with fallback)
  • Wrote this plan file
  • Next: hand to Momus for review before starting Phase 1.1

2026-07-22 — Phase 1.1 complete (d2940a8)

  • Added resolve_scope_style helper + MarkdownStyles struct with 10 constructs, precomputed once in MarkdownRender::init; new struct is #[allow(dead_code)] until Phase 1.5 wires it in. 6 new tests cover primary/fallback/default paths, theme.is_none(), built-in dark theme, and a minimal-root-scopes theme.

2026-07-22 — Phase 1.2 complete (f40ba4c)

  • Added LineKind enum (Heading/Blockquote/TaskItem/BulletItem/NumberedItem/HorizontalRule/Paragraph) and detect_line_kind() using fancy_regex for the 6 block patterns. Wired into check_line — signature now returns (LineType, LineKind, Option<SyntaxReference>, bool); callers ignore LineKind with _ until Phase 1.4. 8 new tests cover each pattern plus edge cases (##notheading, -nospace, --, indented items, empty blockquote).

2026-07-22 — Phase 1.3 complete (89db5b3)

  • Added inline regexes (INLINE_CODE, IMAGE, LINK, BOLD_AST, BOLD_US, ITALIC_AST, ITALIC_US, STRIKETHROUGH, CODE_PLACEHOLDER) and apply_inline() running the plan's 6-step pipeline. Refined italic regexes with (?!\s) opener + (?<!\s) closer to prevent a * b * c false-positives while still requiring the word-boundary lookbehind for some_var_name. Inline code is masked with \x00C{idx}\x00 placeholders before other transforms so its content is never re-parsed. Links/images wrap in OSC 8 hyperlink codes. 15 new tests cover each construct, order-dependence, partial spans, italic false-positives, OSC 8 emission, and image-before-link ordering.

2026-07-22 — Phase 1.4 complete (9890cf0)

  • Added render_markdown_line(line, kind, styles) dispatcher plus per-LineKind block renderers (render_heading, render_blockquote, render_bullet, render_numbered, render_task, render_hrule). H1 gets space-padded, H2-6 keep their ##... prefix; blockquotes get │ ; bullets → •; numbered items keep the number and style only the .; task items → [ ] / [✓]; hrules render as ────────. All block variants delegate leftover content to apply_inline. 15 new tests cover each block type, indent preservation, and nested inline (code in bullet, link in blockquote).

2026-07-22 — Phase 1.5 complete (d65d63e)

  • Wired the rich renderer into render_line / render_line_mut: code lines still route to syntect; non-code lines branch on options.raw_markdown (true → existing markdown-grammar syntect path, false → render_rich_markdown_line). Removed #[allow(dead_code)] from RenderOptions::raw_markdown, MarkdownStyles, LineKind, detect_line_kind, render_markdown_line, apply_inline, and the styles field. Updated the 3 existing tests (no_theme, no_wrap_code, wrap_all) to set raw_markdown: true — they still produce byte-identical output, proving the raw path is preserved. Manual REPL/streaming tests deferred to user.

2026-07-22 — Phase 1.6 complete (b0eeba1)

  • 6 more tests filling out the coverage checklist: bold nested inside a heading, partial bold/link spans via render_line (streaming path), rich rendering with theme=None still strips syntax and emits block glyphs, rich vs raw paths diverge on the same input, and fenced code blocks still route through syntect. Total: 55 markdown tests, 1207 total tests pass, cargo check clean.

2026-07-22 — Phase 2 complete (Phase 1 successor shipped)

  • Phase 2 (tables + hanging-indent list/blockquote wrapping) is complete on top of this foundation. See rich-markdown-renderer-tables.md for the full plan and per-sub-phase progress log. Final commits: fcc4a1d (2.1) → 7671d28 (2.2) → c062f34 (2.3) → cdfaa0f (2.4) → bf06d5e (2.5) → d790782 (2.6) → 336b374 (2.7). Total markdown tests grew from 55 → 94; total test suite 1207 → 1246, all passing, cargo check clean.
Reference in New Issue View Git Blame Copy Permalink