An embedded Relate graph made our PR writer better and cheaper

What Tracetoon is

Tracetoon is an open-source tool that writes better pull-request descriptions from the work an AI coding agent already did. Agents such as Claude Code, Codex and Antigravity keep session logs on disk: the user's requests, plans, decisions, file edits and the commands they ran. A PR description written from the diff alone loses most of that. It can't say why the change was made, what was tried first, or which tests actually ran on the final code.

Tracetoon runs as an agent skill. When an agent is about to open a PR, Tracetoon takes over:

• Finds the matching sessions. It identifies which sessions produced the current branch by matching agent edits to lines in the diff, and asks when a session is ambiguous.
• Builds a redacted evidence set. It pulls prompts, plans, edits, runs and commits from those logs, with private paths and secrets removed.
• Writes the description. The agent drafts the PR body from that evidence, and Tracetoon checks it: every factual sentence must trace back to evidence, and the Validation section is generated from recorded test and build runs, not typed by the model.
• Draws an optional comic. Tracetoon can turn the story of the change into a comic that goes in the PR. It is optional: Tracetoon leaves it out where a repo's contributing rules forbid images in PRs, and you can turn comics off yourself.
• Creates the PR. The agent has the draft reviewed, creates the PR through the normal workflow, and then reports what it did: the link, whether a comic is included, and any sessions it left out.

Tracetoon installs a hook that redirects the agent's gh pr create through this workflow, respects repository AI policies (for example, repos that ask for author-written PR text), and can be turned off per repo. This page compares two versions of its drafting step: the original multi-agent workflow and a new version built on an evidence graph.

One PR, three ways

The same change described three ways. The PR (pr8 in the benchmark) fixes the preview link, which a user testing in Codex called "not very helpful". It also fixes two digest bugs found while dogfooding: commands prefixed with cd were filed under the wrong repo, and a git log in another repo could claim authorship of a commit. All three drafts were written by Claude Sonnet 5.5 and scored by the same judges against the same fact sheet. They are drafts from round 15, the earlier published comparison, so the graph draft here predates the brief changes in the shipped build.

The no-Tracetoon cost is an estimate, because the agent that opens its own PR carries the whole session's context (173K tokens for this PR) into every call. It uses the method in Token cost. The no-Tracetoon draft makes no mistakes, mostly because it says almost nothing. The original Tracetoon workflow brings back the why ("found while dogfooding", the user's "not very helpful"), but this run cost $1.24 and made two confident statements the evidence contradicts. The graph version keeps the context, adds the precise details (pinned commits, what the new tests check, fallback behavior), generates validation from receipts, and points the reviewer at the riskiest rule. In round 15 it cost less than half as much as the original workflow. For space, some lines are trimmed (marked […]) or merged, and the generated footers are left out. Highlighted text is verbatim.

How quality is judged

Each PR in the benchmark is a real change from Tracetoon's own development, with the original agent session logs. Every description is judged against a hand-checked fact sheet, by two judges, in both orders.

The fact sheet

For each case, a person read the diff and the original work sessions and wrote down four lists. Judges cite these items by ID when they score a draft.

For the pr8 example above, a trap is "saying the preview now opens automatically in the browser" (it only prints a clickable URL). Context X2 is "the commit messages say the two digest fixes were found while dogfooding".

Pairwise judging, both orders, two judges

A judge sees the fact sheet, the commits, the diff, the original session evidence, any fresh validation runs, and two descriptions labeled A and B. It audits every checkable clause in each and marks it supported, contradicted, unsupported (broader than the evidence) or unverifiable. It scores ten reviewer-usefulness items, then says which description a maintainer would rather receive. The prompt states the priority order outright: accuracy first, then usefulness to the reviewer, then the reader's effort. Do not prefer a description for being longer, more formatted or more confident.

flowchart LR
  A["Draft A
(no-graph workflow)"] --> J1
  B["Draft B
(graph workflow)"] --> J1
  F["Fact sheet
diff · commits · sessions"] --> J1
  J1["Judge 1
order A,B and B,A"] --> V{"Consensus?"}
  J2["Judge 2
order A,B and B,A"] --> V
  A --> J2
  B --> J2
  F --> J2
  V -->|"all 4 judgments agree"| W["Win for one side"]
  V -->|"any disagreement"| T["Tie"]

Each pair of drafts gets four judgments: two judges, each reading both orders. Claude runs use Opus 5.5 and Sonnet 5.5 as judges; Codex runs use Sol 6.1 and Terra 6.1. A consensus win needs all four to pick the same draft, so position bias and judge noise turn into ties rather than wins. That strict rule is why most pairs are ties. An 18–1 record means eighteen pairs where every judgment, in both orders, preferred the graph draft, and one pair where they all preferred the other one.

The ten rubric items

Each item is scored 0 (missing or misleading), 1 (partial) or 2 (sufficient and accurate), or left null when it doesn't apply. The first seven come from Tracetoon's own description rubric; the last three come from PR-description research (Xiao et al., FSE 2024; Pirouzkhah et al., MSR 2026).

The composite score

The composite rolls the judgment into one 0–100 number per draft, averaged over its four judgments:

composite = 35·claims + 15·context + 15·uncertainties + 35·rubric − 15·errors − 7·unsupported − 10·traps

Coverage terms are fractions of the fact sheet (0–1). Rubric is the mean item score divided by 2. Errors, unsupported claims and traps are counts per judgment. A single contradicted claim costs 15 points, as much as missing all of the context. The head-to-head consensus result and the composite are reported separately because they can disagree: the composite rewards coverage, while the head-to-head reflects which draft a maintainer would rather get.

The judges have an answer key, and the win counts are deliberately hard to earn. Two different judges each read the pair twice, once in each order, and only a unanimous verdict counts as a win.

Where the quality gain comes from

The composite gain splits cleanly into its terms. On Claude, almost all of the gain is better-written descriptions (rubric), plus smaller gains from context and fewer unsupported claims. Fewer stated uncertainties take part of it back. On Codex, it's mostly coverage and accuracy: the graph surfaces facts the no-graph workflow leaves out, and its drafts had no factual errors.

Accuracy: what the graph is particularly good at

On Codex, the judges found no factual errors in any of the 25 graph drafts, against 0.07 per judgment for the no-graph workflow. On Claude, factual errors are level with the no-graph workflow (0.08 against 0.09 per judgment), and unsupported claims fell by about three quarters (0.03 against 0.11). Neither arm hit a trap on either provider. These are the measures a maintainer cares about most: a confident sentence that's wrong costs more trust than a missing one.

Most writer errors in this benchmark are misattribution: the behavior is right, but the place or the scope is wrong. Examples are crediting a change to a nearby helper, calling a pre-existing function "new", or writing "always" for code with a fallback branch. The graph answers those questions before the writer starts:

• Sentence-to-code links. Each commit-message sentence is linked to the hunks that implement it (implementedBy, → H#). Where structure leaves a gap, search links from Relate's fulltext index fill it. The writer doesn't have to guess which file owns a behavior.
• Code lines with a direction. Under each commit sentence the brief prints the code lines it refers to, marked as removed or added, with the change spelled out as was → now. The writer can't describe a change backwards.
• Owners rows. For each behavior, the brief names the function that implements it, so the writer doesn't credit a nearby helper.
• Commit-vs-code conflicts. When a commit message says something the code does not show, the brief lists it as a conflict row, and the writer follows the code.
• New versus changed. A symbol is "new" only if the patch defines it on added lines and on no removed line. Everything else is "changed", and the brief says which is which.
• Branch conditions. Hunks that add conditionals are marked, so the brief tells the writer where "always" or "only" would be wrong.
• A do-not-say list. Existing names, conditional code, results only a commit message asserts, and the exact file list are put in front of the writer before it writes.
• Validation from receipts. The Validation section is written by the tool from recorded runs: the broadest run of each check on the final code, plus any earlier failure. The writer never types test results. In the pr8 example, this is exactly what removed the wrong type-check claim.

The reviewer-usefulness items

Read the chart from the biggest gaps:

• Feedback request (+1.27 Claude, +0.75 Codex) and entry point (+1.11, +0.47). The brief's entry row names where to start and what is tied to it, and the drafting rules keep a closing ask for the kind of review wanted.
• Validation (+0.60 on Claude). Tracetoon writes this section from receipts, so it is bound to evidence and honest about limits.
• Economy (+0.27 on Claude). The drafting rules ask for 250–400 words, and Claude's graph drafts now read tighter than its no-graph ones.
• Economy (−0.83) and problem framing (−0.27) on Codex are where the graph workflow still loses. Terra's no-graph descriptions are very tight (economy 2.00), which makes this the clearest next target.

Coverage: more of what a reviewer needs

Context coverage rose by five to seven points on both providers, and this is the result that depends most on the graph. Context lives in the session logs, not the diff: the user's goal, an earlier failed attempt, a plan that was approved. The graph links each surviving code change back to the prompt, plan or decision that produced it. It then hands the writer those facts as rows: "why this work started", process history, and requests scoped to this PR's own turns.

The graph helps in three ways:

• Accuracy: it has already worked out which code implements each claim, what's new versus changed, and where conditionals make "always" wrong.
• Context: it links code back to the conversation that produced it, which the diff alone can't tell you.
• Structure: a ranked set of change units gives the reader an entry point.

It doesn't fix everything yet. On Claude, the graph drafts state fewer open questions (uncertainties respected fell from 0.73 to 0.60). On Codex, they are wordier than the strong baseline.

Token cost

The Claude no-graph drafts were regenerated from round 20 on, after a Claude Code update, with the same Claude Code release (2.1.319) as the graph drafts. They cost $0.690 per draft, against $0.781 for the no-graph drafts in earlier rounds, so the Claude ratio here is measured against the cheaper baseline.

The cost story is about where the expensive model spends its tokens. Without the graph, the expensive model reads source code to check its own claims, two or three times per PR. With the graph, that work is done once, deterministically, before any model runs, and building the brief costs no tokens. What's left for the models is writing, plus a cheap model checking a short packet:

• Review moves to a cheap model, on a small packet. In the no-graph workflow, Sonnet reviewers spend about $0.35 of the $0.69 per PR re-reading roughly 100K characters of source to check each claim. In the graph workflow, two Haiku (or Luna) reviewers get a compact packet: line-numbered body, brief rows, the hunk-to-symbol map and the relevant diff. Haiku's two reviews on Claude cost about four cents per PR, and Luna's on Codex under one cent.
• No forked subagents. On Codex, the no-graph workflow forks a researcher, a drafter and a reviewer, and each fork carries the parent's whole history. The graph workflow forks nothing: Tracetoon runs the two reviews itself as separate small sessions.
• One merged list, one edit pass, no re-review. Findings from both reviewers are deduplicated into one list of at most 12 items, and the writer applies them in a single edit. When this arrived in round 15, it alone cut the Codex writer's input from 3.1M to 1.6M tokens per PR, compared with the earlier review loop.
• Less writer output. Sonnet writes 7K tokens per PR instead of 22K, and Terra 5K instead of 10K, because the brief has already done the analysis the writer would otherwise narrate to itself.

Versus drafting the PR in the main session

Why not have the agent that did the work write the PR before it stops? The catch is that it carries the whole session's context, so every model call re-reads hundreds of thousands of tokens. Prompt caching makes those re-reads cheaper but not free, and a PR still adds new tokens: the diff and git output it reads, and the description it writes. We estimate this case rather than measure it:

• 5 model calls to write a PR in the session: look at the diff and log, draft, open it.
• Context per call: the average end-of-session size across the five cases, about 494K tokens (range 173K–860K).
• New tokens: about 10K of new input (diff and command output) and about 2K of output (commands and the PR body).
• Cached: all 5 calls read the context from cache.
• Not cached: the cache has expired, so the first call writes the whole context to cache and the next 4 read it.
• List prices: Sonnet $3/M input, $3.75/M cache write, $0.30/M cache read, $15/M output; Terra $1/M input, $0.05/M cached input, $5/M output. Any long-context surcharge is not included.

Worked through for Sonnet with the cache warm, 5 × 494K × $0.30/M = $0.74 of cached reads, plus about $0.07 of new input and output. With the cache expired, the first call costs 494K × $3.75/M = $1.85 on its own. The graph workflow avoids both: it starts a fresh session whose context is a compact brief of about 2–3K tokens instead of the whole history.

On Sonnet the graph workflow is cheaper either way. On Terra, where cached input is very cheap, in-session drafting with a warm cache comes out cheaper. Once the cache has expired, it costs about three times more. These are estimates; the quality of in-session drafting hasn't been measured. A fresh agent opening the PR without Tracetoon, the first column of the example above, scores far lower, because it never reads the logs.

Tracetoon without a graph

The original workflow was already a careful multi-agent design. It reads the session logs through a digest, and coordinates three separate roles so that no agent checks its own work.

flowchart LR
  L["Session logs
+ git"] --> D["digest.json"]
  D --> R["Researcher
(subagent)"]
  D --> W["Drafter
(reads digest, patch,
expands refs)"]
  R --> W
  W --> RV["Fresh reviewer
(re-reads source,
checks every clause)"]
  RV -->|"findings"| W
  RV -->|"up to 2 rounds"| RV
  RV --> P["Render + publish"]

Where its tokens go (Sonnet, the 25 no-graph drafts used in the shipped-build comparison): the coordinator about $0.29, reviewers about $0.35 over 1.4 review sessions per PR, and the researcher about $0.05. On Codex the split is coordinator $0.11, drafter fork $0.09, reviewer fork $0.08 and researcher fork $0.04, because each forked role re-reads the parent's history.

The design is sound, and it produces accurate descriptions. Its cost comes from asking an expensive model to rediscover structure from text. It has to work out which commit sentence maps to which hunk, which symbols are new, and why a change was made. Then a second instance does all of it again to check the first, at full price.

What the graph does

The graph moves that rediscovery out of the model and into a deterministic build step. One command, tracetoon prepare, reads the session logs and git, builds an embedded graph in Relate, adds search links from Relate's fulltext index, runs a fixed set of Cypher queries, and prints a compact evidence brief. No model is involved, so this step costs no tokens.

The graph

flowchart LR
  P["Prompt
(user request)"] -->|"IN_TURN"| E["Edit"]
  PL["Plan / Decision"] -->|"EXPLAINED_BY"| CH["Change unit"]
  E -->|"SURVIVES_IN"| H["Hunk"]
  H -->|"OF"| F["File"]
  C["Commit"] -->|"SAYS"| S["Commit sentence"]
  S -.->|"implementedBy / TIED_TO"| H
  P -.->|"TIED_TO (search)"| H
  H -->|"part of"| CH
  R["Run (test/build)"] -->|"after edits?"| F

The graph is small: thirteen node types and sixteen relationship types. The edge that does the most work is SURVIVES_IN, which connects an agent's edit in the logs to a hunk that's still in the final diff. From a hunk you can walk back to the prompt that asked for it and forward to the commit sentence that describes it.

The evidence brief

The queries turn the graph into short tables the writer can cite by reference (CH1, S2, H4…):

Fewer roles, fewer steps

flowchart LR
  L["Session logs
+ git"] --> G["prepare
(graph + search links
+ queries, 0 tokens)"]
  G --> B["Evidence brief"]
  B --> W["Writer
(Sonnet / Terra)"]
  W --> CK["render --check
(deterministic)"]
  CK --> R1["Reviewer: general
(Haiku / Luna, xhigh)"]
  CK --> R2["Reviewer: completeness
(Haiku / Luna, xhigh)"]
  R1 --> M["Merged findings
(≤12 items)"]
  R2 --> M
  M --> W2["One edit pass"]
  W2 --> P["Render + publish"]

Once the map exists, fewer agents are needed. The researcher and the separate drafter go away. The reviewer becomes a cheap model with a checklist and a short packet, and one edit pass replaces several review rounds. On Claude the graph workflow runs two subagent sessions per PR, against 1.6 without the graph, but they run on Haiku and read a short packet instead of the source.

Deterministic checks at render

Because every sentence carries brief refs, Tracetoon can check the draft without a model. It flags:

• change units, commit sentences and tests no sentence covers;
• a sentence that names a file outside the cited change;
• "new" next to a name that already existed;
• "always" or "only" on code with a branch;
• results that only a commit message claims;
• a "housekeeping only" line when logic changed.

Brief refs (CH1, S2, H4…) that leak into the text are stripped before the body is published.

Fixing these costs nothing in model review.

Why a strong writer and cheap reviewers, not the reverse

We first tried using the graph to let a small model write: Haiku as the writer, with strong or high-effort reviewers. In direct head-to-head judging, Haiku with the graph still lost to Sonnet's no-graph workflow, 1–16–8 and 2–14–9 (wins–ties–losses). Coverage and errors on the hardest PRs were the gap. The graph doesn't turn a small model into a mid-size one. It lets the mid-size model skip the expensive rediscovery and hand checking to a small model that only needs to compare a draft with a short packet. Two settings mattered:

• xhigh reasoning effort on the reviewer, which is cheap on Haiku or Luna;
• a completeness lens: one reviewer checks only whether every change unit, test file, doc change and old → new value is covered.

The shipped build makes this setup the default: reviewer model cheap (Haiku 5.5 on Claude Code, Luna 6 on Codex) at xhigh effort, two reviewers (general and completeness), one merged list of findings, no re-review, the compact toon brief layout and fulltext search links.

Why Relate: an embedded graph database for agent systems

Relate is an embedded graph database written in Rust, and it is the engine the shipped Tracetoon build runs on (relatedb/relate, pinned at v0.1.2-5, commit aa0f79e). Embedded means it runs inside the application process, with no server to install, start or connect to. Tracetoon ships Relate compiled to WebAssembly inside its CLI. The same build runs on Linux, macOS and Windows, works offline, and builds a fresh graph for every PR. Tracetoon queries it with Cypher.

That matters for agent tooling, which usually runs on a developer's machine, inside a sandbox, or in CI, where a database server is a non-starter. Relate gives those tools a real graph model and query language with the deployment footprint of a library.

Graph algorithms in a few lines of Cypher

The questions Tracetoon has to answer are relationship questions. Which request led to this code? Did any passing test run come after the last edit to this file? Which commit sentence describes this hunk? In a graph these are short path queries. Here is the core of the query that finds which user request produced a surviving piece of code, and which plans or decisions came with it:

MATCH (h:Hunk {id: $hunk})<-[:SURVIVES_IN]-(e:Edit)-[:IN_SESSION]->(:Session {band: 'included'})
OPTIONAL MATCH (e)-[:IN_TURN]->(p:Prompt)
OPTIONAL MATCH (x)-[:IN_TURN]->(p) WHERE x:Decision OR x:Plan
RETURN p.id AS prompt, p.text AS request, collect(DISTINCT x.id) AS plansAndDecisions

The same pattern carries the rest of the brief:

• Traversal: from a hunk back to the prompt that asked for it, and forward to the commit sentence that describes it.
• Reachability and ordering: whether a passing run follows the last edit of each file, which yields the open "not checked after the last edit" limits.
• Set difference: symbols defined on added lines and on no removed line are new; everything else the patch touches is changed.
• Neighborhoods and ranking: each change unit with its requests, commits, tests and runs, ranked by weight to pick an entry point.

Each of these would take an LLM thousands of tokens to reconstruct from raw text, and the result would vary from run to run. In Relate, each is a deterministic query that costs no tokens.

Search inside the graph

Some links have no structure to follow. A commit sentence about a docs or script change may share no symbol with the code it describes. For those, Tracetoon uses Relate's built-in fulltext index, which ranks hunks with BM25. It indexes each hunk's path, symbol names and changed lines, then searches it from each commit sentence, commit subject, request and plan paragraph:

CREATE FULLTEXT INDEX hunk_text FOR (h:Hunk) ON EACH [h.text]

MATCH (h:Hunk)
SEARCH h IN (FULLTEXT INDEX hunk_text FOR $text LIMIT 256) SCORE AS s
RETURN h.id AS hunk, s

A hit becomes a link only when it agrees with an exact name or path mention, or when BM25 alone is clearly ahead of the next hit. A commit's sentences only see that commit's own hunks. Accepted hits are stored as TIED_TO edges with a score, so the brief queries read them like any other relationship.

GraphRAG: retrieval by relationship, not just similarity

Most retrieval-augmented generation pulls text chunks that look similar to a query. That finds relevant passages, but not the links between them: which plan explains which change, or which test covers which behavior. The evidence brief is GraphRAG, built from fulltext search and graph structure together. Search links from Relate's BM25 index find the hunks a sentence or request talks about. Retrieval then starts from each change unit and follows edges to the facts connected to it, and packs those neighborhoods into a compact context for the writer. The writer gets the request, the commit sentence, the hunk, the test and the run that belong together, already joined. It doesn't have to work out those joins itself, which is where writers make most of their mistakes.

Higher accuracy and explainability

Because the brief comes from queries, every fact has a lineage. A sentence in the PR cites a brief row (CH1, S2, H4). The row comes from a query over specific nodes, and the nodes point to specific hunks in the diff and specific lines in the session logs. That chain makes three things possible:

• Checking without a model. Tracetoon checks cited sentences against the graph at render time: coverage of every change and commit sentence, new versus changed names, scope words on branching code, and results only a commit message asserts.
• Cheaper, more focused review. Reviewers check a draft against the same refs instead of re-reading the source, which is why a small model at high effort can do the job.
• Audit trails. A reader, or another agent, can ask why a sentence is in the PR and follow it back to the evidence. The same graph gives the same brief every time.

The results on this page show the payoff. On Codex, the judges found no factual errors in the graph drafts. On Claude, unsupported claims fell by about three quarters. Context coverage rose by five to seven points on both providers, because the graph links code to the conversation that produced it.

The pattern generalizes beyond PR descriptions. Agent systems produce large, messy traces: logs, tool calls, edits, tests. Turning those into a small embedded graph lets deterministic queries do the structural reasoning. Models are left with what they're good at, writing and judgment, on a context that is smaller, cheaper and traceable. Relate makes that practical anywhere an agent runs, because the graph travels with the tool.

How we got here

The shipped build came out of twenty-three benchmark rounds. Each point below is the graph workflow against the same model's no-graph workflow, on the same harness. Round 15 is the earlier published result. Rounds 18 to 23 refined the brief and moved the graph onto Relate, and round 23 is the build that ships.

The early graph version was cheap and bad: it covered too little. Adding back independent review got it to parity. Rounds 13 to 15 turned parity into a win, mostly through unglamorous fixes: enforcing which model reviews, dropping a template, and collapsing the review loop into one pass. Rounds 18 to 23 made the brief more exact about where code lives and which way it changed, and moved the graph onto Relate. On Codex the gain stayed between +3 and +5 throughout. On Claude it sat inside the noise band from r20 to r22 and cleared it with the shipped build.

Lessons

• Measure head-to-head, not with composites from separate runs. Composites from separate runs put Haiku with the graph level with Sonnet's no-graph workflow. Direct judging showed it losing 2–14–9.
• Instructions don't enforce model choice; code does. Codex's own tool description says "inherited parent model is preferred", so the cheap drafter skipped the reviewer model a third of the time, then reported it anyway. Having Tracetoon run the reviewer, and verify the model from the session log, fixed it.
• Templates cost economy points. A fixed seven-section layout read as "inventory-like" to judges. Flexible structure raised the writing score more than any prompt tweak.
• Rerun before you trust a gap. Two runs of the shipped build on Codex came out at +3.5 and +5.2. One run can move as much as most single-round changes, so we read trends across rounds and the consensus records before any single composite.
• Review loops are where the money goes. On Codex the reviews themselves cost 3% of the total. Most of the remaining cost was the writer re-reading its context between reviews.

Per-case results

Method and caveats

Appendix: the numbers

Sources: results/r23/claude/sonnet-close and results/r23/codex/terra-close (both arms in each run; the Claude no-graph drafts are cached from r20/claude/sonnet-g4b, the Codex ones from r11), plus the same-build rerun r23/codex/terra-close2. Claude token splits come from the writer traces (the no-graph ones from r20/claude/sonnet-g4b); Codex token splits come from each draft's per-model usage, and the Codex no-graph role split from r11/codex/terra-c25 traces. The round history comes from r10–r23 summaries. The example drafts are r15/claude/sonnet-h1 replicate 5 (pr8) and evgraph-dev12/claude replicate 2 (pr8, native).