Code graphs sound like a very good match for coding agents. And there is rarly a day where not another one of these graph frameworks passes my social streams.
The idea is simple. Parse the repository once. Store classes, functions, imports, calls, and relationships in a graph. Then give the agent small pieces of that graph instead of making it search through source files. If the agent wants to know what a method calls, it asks the graph. Less source goes into the context. Fewer files need to be opened. Navigation gets cheaper. That is at least the theory.
That is also the promise behind code-review-graph. It uses Tree-sitter to build a local graph and exposes that graph through MCP. And that was enough to make me curious to at least give one of these
I wanted to know what happens when you connect it to IBM Bob Shell and point the combination at a repository where navigation actually matters. Open Liberty is a good test for this. My pinned checkout contains 120,016 tracked files. 47,668 of those are Java files. There is enough code here that every wrong turn costs time and context.
There was one problem with my assumption: Bob is already pretty good at this.
With its normal filename search, text search, and targeted file reads, Bob found all five required call sites in my super narrow benchmark using four native tool calls. Then it repeated the result even. So that was not only a lucky coincidence. No prebuilt graph. No 6.9 GiB index. Just search and source reads.
The graph integration worked technically because it is nothing more than an mcp server. Bob launched code-review-graph locally and called its tools. The performance result also looked promising at first. Across two paired runs, the graph-assisted condition was 18.8% faster and returned 18.7% fewer tool-result characters.
But when we look at the answers, these impressive results quickly became marketing numbers. Bob without the graph found ten of ten expected call sites across those runs. Bob with the graph found only one.
So the measured improvement was zero. The graph produced a latency improvement but the navigation was worse.
This tutorial builds that experiment from scratch so you can play with it more and draw your own conclusions.
Prerequisites
I captured these results on my Arm Mac. You need enough disk space for both the Open Liberty checkout and the generated graph. The graph alone grew to 6.9 GiB in this experiment.
IBM Bob Shell 2.0.2, commit
a31a75e3(grab a free trial if you like and play with it)code-review-graph2.3.8Python 3.14.7
git,uv, andjqAt least 10 GiB of free disk space
A Bob API key for non-interactive runs
Basic familiarity with MCP and Bob Shell
I use Open Liberty commit 3da6c82529721046a4a6a73f07b34c5c57f8d76e from the integration branch and kept that commit pinned. The grader depends on exact symbols and line numbers, and both change as the repository moves. Also I was just looking for volume here and not an up to date implementation.
The complete scripts and benchmark prompts are in the companion project on my Github.
Project Setup
The lab has four moving parts: the pinned Open Liberty checkout, the local code graph, Bob with and without MCP, and an external grader.
Nothing in the benchmark asks Bob to edit Open Liberty. The runner disables subagents, uses the same prompt and limits for both conditions, captures Bob’s event stream as JSONL, and checks that tracked source files stay unchanged.
Let’s build the repository and graph first.
Pin the Open Liberty checkout
Create a shallow checkout at the exact commit:
OPEN_LIBERTY=/path/to/open-liberty
git init "$OPEN_LIBERTY"
git -C "$OPEN_LIBERTY" remote add origin \
https://github.com/OpenLiberty/open-liberty.git
git -C "$OPEN_LIBERTY" fetch --depth 1 origin \
3da6c82529721046a4a6a73f07b34c5c57f8d76e
git -C "$OPEN_LIBERTY" checkout --detach FETCH_HEADPinning the repository is part of the benchmark design. Without it, a citation to line 1640 can be correct today and wrong next week and you will have trouble reproducing this. The same applies to graph relationships. If the source changes between runs, you are measuring two different systems and I really love your comments on articles but I do not want to argue about the correct commit to use in my examples ;-)
Install code-review-graph
Keep the graph installation isolated:
CRG_VENV=/path/to/crg-venv
uv venv --python 3.14 "$CRG_VENV"
uv pip install --python "$CRG_VENV/bin/python" \
code-review-graph==2.3.8Now build the graph:
time "$CRG_VENV/bin/code-review-graph" build \
--repo "$OPEN_LIBERTY"
"$CRG_VENV/bin/code-review-graph" status \
--repo "$OPEN_LIBERTY"My initial build took almost 8 minutes. The resulting graph looked like this:
Files: 48,547
Nodes: 463,449
Edges: 3,554,089
Languages: bash, csharp, java, javascript, perl, properties, python, sql, yaml
Graph database: 6.9 GiBThe file count needs a little explanation though. Open Liberty contains 120,016 tracked files. The graph indexed 48,547 because it indexes supported source and configuration formats. It does not represent every artifact in the checkout.
While 7 minutes sounds ok-ish, it matteres even more when you think about doing this more regularly.
When you look at the code-review-graph quick start talking about roughly ten seconds for an initial 500-file repository, they surly do not expect this thing to run against something more mature like Open Liberty. That turns out to be a very different workload.
That cost can amortize nicely if you reuse the graph for weeks of code reviews. It does not amortize across two navigation prompts. And depending on how fast your project moves, you will have to rebuild the graph pretty often. Maybe something to also keep in mind when choosing a graph/indexing approach.
Implementation
Now we connect the graph to Bob and create the two benchmark conditions.
Connect the MCP server to Bob
IBM’s Bob Shell MCP documentation supports local stdio servers through a project-level .bob/mcp.json.
code-review-graph supports automatic installation for several MCP clients. Bob is not one of those installation targets, so we wire the server manually.
Run this from the Open Liberty checkout:
cd "$OPEN_LIBERTY"
bob mcp add \
--scope workspace \
--transport stdio \
code-review-graph \
"$CRG_VENV/bin/code-review-graph" \
-- serve \
--repo "$OPEN_LIBERTY" \
--tools get_minimal_context_tool,query_graph_tool,traverse_graph_tool,semantic_search_nodes_tool,list_graph_stats_tool,get_impact_radius_tool,get_review_context_toolThis creates .bob/mcp.json.
I intentionally expose only a subset of the available graph tools. code-review-graph exposes 30 MCP tools by default and supports --tools to reduce that surface. Bob’s own guidance follows the same idea. Every unused tool definition consumes model context, so there is no reason to expose 30 tools when the experiment needs seven.
There is another argument here that looks boring but is important:
--repo "$OPEN_LIBERTY"My first pilot built the graph successfully and then Bob’s MCP process told me that no graph existed.
The problem was the working directory. Bob launched the MCP child process from the harness directory, not from the repository given to bob run --workspace. Something I had to learn too while writing this article. Passing the repository explicitly to serve fixed it.
Smoke-test the connection
Use a cheap Bob run:
BOB_API_KEY=... bob run \
--workspace "$OPEN_LIBERTY" \
--mode agent \
--max-turns 4 \
--max-cost 0.10 \
--disable-subagents \
--trust \
--accept-license \
'Use the code-review-graph stats tool. Return only files, nodes, and edges.'Bob called list_graph_stats_tool and returned the same file, node, and edge counts from the graph status command.
Good. MCP works. Thanks Bob :-)
Define a benchmark an agent can actually fail
The graph project’s published token benchmark compares a small graph response with the complete source corpus. The reproduction notes correctly describe that whole-corpus comparison as an upper bound.
Bob does not load all 120,016 files into context either. It searches. It opens candidates. It changes direction. Sometimes it makes a bad choice and pays for it. That is the behavior we need to measure.
So I record the complete agent run:
Wall-clock duration reported by Bob
Session cost reported by Bob in Bobcoins
Native and MCP tool calls
Characters returned by tools
Whether the task completes
Whether the final answer matches the pinned source
Bob’s non-interactive command documentation defines --max-cost in Bobcoins and exposes the session cost through stats.session_costs in its JSON output.
The character count needs another warning. Tool-result characters are not tokens. I use them because both runs use the same Bob version and harness, and because they give us a tokenizer-independent measure of how much material tools return. Treat this as just a context proxy for this little test and not a token count.
Start with an exact-symbol task
The narrow benchmark starts from a method we know:
Inspect this exact method:
dev/com.ibm.ws.kernel.feature.core/src/com/ibm/ws/kernel/feature/internal/
FeatureManager.java::FeatureManager.updateFeatures
Identify the direct calls inside updateFeatures(...) that perform feature
resolution and the OSGi bundle lifecycle.
Return five relevant direct callees with file:line citations, explain each,
and give a reading list of no more than three files.I intentionally made this strict. There are five required call sites. The answer needs file and line citations and the run gets only four tool calls.
The second benchmark is wider. It starts from EnvCheck.java and asks Bob to follow the four-to-six-step production path to the OSGi framework launch. It also asks why the launcher layers exist, for one focused test, and for a six-file reading list.
That task is deliberately harder. It crosses overloaded methods, launcher abstractions, classloading, reflection, and eventually the concrete Equinox framework creation.
Change only the navigation condition
For solo Bob, the runner disables MCP. For graph-assisted Bob, the runner enables the graph and tells Bob to start with minimal graph context and an exact-symbol query, then verify selected source lines with a native read.
Everything else stays the same.
This is important because I only want to test whether adding the graph improves Bob.
Run the complete benchmark:
./scripts/run-benchmark.sh \
/path/to/open-liberty \
/path/to/bob-key.jsonThe runner stores Bob’s JSONL stream and creates a small summary.json with the metrics we care about.
Keep credentials out of the MCP configuration
Do not put your Bob API key in .bob/mcp.json, a prompt, or the captured transcript.
The companion runner reads an .apikey value from a separate JSON file and passes it through BOB_API_KEY. You can export the json file from the Bob admin console. Make sure it is protected locally:
chmod 600 /path/to/bob-key.jsonIBM’s MCP security guidance points out that a local MCP server runs with Bob’s permissions.
Leave embeddings out of the first experiment
I did not build embeddings. This keeps all source-derived graph data in the local SQLite database and leaves us with the default local installation. Without embeddings, semantic search is based on names, paths, and signatures. That works well for something like EnvCheck. It is much weaker for architectural prose such as:
where does startup launch OSGi?We will see that limitation in the wider benchmark.
If you have to, test embeddings separately. Adding embeddings and changing Bob’s graph workflow at the same time makes it impossible to tell which change fixed the result. And I also wanted to keep this article at a readable size. I have been throwing a lot of experiments and more complex tutorials at y’all lately.
Verification
Now we can look at the results.
Narrow run one
The first exact-symbol pair looked like a clear graph win before grading:
Bob solo
duration 27.404 s
reported cost 0.139082 Bobcoins
tool calls 4 native
tool-result characters 13,883
correct requested sites 5/5
Bob + code-review-graph
duration 20.284 s
reported cost 0.132528 Bobcoins
tool calls 2 MCP + 2 native
tool-result characters 12,381
correct requested sites 0/5The graph-assisted run was 26.0% faster. It cost 4.7% less and returned 10.8% fewer tool-result characters. What a brilliant marketing statement.
But it found absolutely zero of the requested call sites.
The solo Bob answer returned all five.
Narrow run two
I repeated the complete benchmark because it was almost unbelievable:
Bob solo, second pair
duration 23.231 s
reported cost 0.138920 Bobcoins
tool calls 4 native
tool-result characters 14,299
correct requested sites 5/5
Bob + code-review-graph, second pair
duration 20.844 s
reported cost 0.159434 Bobcoins
tool calls 3 MCP + 2 native
tool-result characters 10,542
correct requested sites 1/5The direction stayed surprisingly consistent.
Graph-assisted Bob was 10.3% faster and returned 26.3% fewer tool-result characters. This time it cost 14.8% more and also exceeded the four-call budget.
It found one of the five required sites.
Across both paired runs, solo Bob averaged 25.318 seconds. Graph-assisted Bob averaged 20.564 seconds.
So the graph was 18.8% faster.
It also returned 18.7% fewer tool-result characters and cost about 5.0% more on average.
Tl;dr: There is no usable performance win here. Correctness failed first.
What did the wrong answer look like?
The pinned FeatureManager.updateFeatures implementation gives us the five exact calls:
1582 resolveFeatures(...)
1640 provisioner.installBundles(...)
1666 provisioner.uninstallBundles(...)
1694 provisioner.resolveBundles(...)
1711 provisioner.preStartBundles(...)The first graph-assisted answer cited lines 1087, 1204, 1255, 1260, and 1274. It introduced BundleInstaller.startBundles and an older-looking Provisioner API that the current updateFeatures method does not call.
The second graph-assisted answer improved a little. It found:
1582 resolveFeatures(...)Then it replaced the four bundle lifecycle calls with methods such as setResolvedFeatures, updateServices, checkInstallStatus, and checkBundleStatus.
The result sounded plausible but are not sufficient.
Why minimal graph context hurt this task
The graph database itself matched the pinned Git SHA. The exact-symbol lookup found the correct method. The failure came afterwards.
In code-review-graph 2.3.8, query_graph with detail_level="minimal" caps the visible result list at five. The implementation uses min(max_results, 5) in this mode.
FeatureManager.updateFeatures has 54 distinct callee names.
The minimal response therefore returned five and told Bob:
49 omittedThe interesting bundle lifecycle calls occur much later in the method. They never appeared in the compressed result. The graph documentation tells clients to escalate when minimal output is insufficient. Bob saw the truncation signal and did not escalate to standard detail.
And this is partly my fault too.
I explicitly told Bob to start with minimal detail because this is the advertised token-efficient workflow. The experiment proves that this retrieval policy needs a recovery rule. Compressed context is only useful if the agent knows when to ask for more.
Many Java calls remain bare targets
There is another problem deeper in the graph.
For FeatureManager.updateFeatures, the graph stores 54 unique call targets. Only 18 resolve to qualified nodes. The other 36 remain bare targets.
The four provisioner calls we need look like this:
line target receiver
1640 installBundles provisioner
1666 uninstallBundles provisioner
1694 resolveBundles provisioner
1711 preStartBundles provisionerThe receiver is preserved.
But installBundles does not resolve to something like:
Provisioner.installBundlesThe same applies to the other three calls.
resolveFeatures is the only one of our five required calls that resolves to a qualified graph target. That matches the second graph-assisted result almost perfectly. Bob found resolveFeatures and missed all four unresolved provisioner calls.
You can inspect these graph details without modifying the database:
./scripts/inspect_graph.py /path/to/open-libertyThis is an important limit. Bob’s reasoning contributed to the failure, but the graph does not give Bob four concrete Java method targets to traverse in the first place.
The wider startup trace exposes more boundaries
The second task starts from EnvCheck.java and asks Bob to trace the production path into the OSGi framework.
The initial run used 12 turns and a 0.50-Bobcoin ceiling:
Bob solo
duration 64.087 s
reported cost 0.326164 Bobcoins
tool calls 12 native
tool-result characters 57,085
outcome final answer
Bob + code-review-graph
duration 43.657 s
reported cost 0.524368 Bobcoins
tool calls 9 MCP + 6 native
tool-result characters 46,419
outcome turn limit, no final answerSolo Bob completed, but it still failed my strict quality gate. It reached LauncherDelegateImpl and then claimed that this class instantiated and started Equinox.
The real path continues into FrameworkManager.launchFramework. That method calls initFramework, and the concrete framework creation appears later:
597 FrameworkFactory.newFramework(...)
600 fwk.init()The answer also claimed that LauncherTest.TestLauncher overrides bootstrap creation. In this pinned checkout it only overrides getEnv.
So a final answer is not automatically a correct answer.
More budget did not fix the broad task
I increased both conditions to 20 turns and 0.80 Bobcoins.
The next pair ended without final answers:
Bob solo rerun
duration 77.992 s
reported cost 0.879986 Bobcoins
tool calls 16 native
tool-result characters 113,054
Bob + code-review-graph rerun
duration 73.377 s
reported cost 0.880286 Bobcoins
tool calls 4 MCP + 16 native
tool-result characters 91,874Another paired batch produced:
Bob solo, second higher-budget run
duration 66.975 s
reported cost 0.814708 Bobcoins
tool calls 16 native
tool-result characters 102,643
outcome cost limit, no final answer
Bob + code-review-graph, second higher-budget run
duration 62.905 s
reported cost 0.695030 Bobcoins
tool calls 4 MCP + 16 native
tool-result characters 35,866
outcome turn limit, no final answerNeither condition produced a correct answer on this task. Throwing more budget at a task is absolutely not solving all problems.
Overloads already weaken the graph anchor
EnvCheck defines two main methods. The graph stores one EnvCheck.main node with the two-argument signature and combines call edges from both overloads. So Bob starts from a graph node that already represents more source behavior than the method we intended.
The minimal result then cuts off another useful edge. EnvCheck.main has ten distinct callee names, but launcher.createPlatform(args) appears after the first five.
Again, the compression removes the part of the result the task needs.
Reflection breaks the easy static path
The startup path later reaches KernelBootstrap.go().
This code loads the LauncherDelegate implementation through a nested classloader and reflection. The graph stores the call at line 215 as the bare target:
launchFrameworkIt does not resolve that directly to:
LauncherDelegateImpl.launchFrameworkBob has to return to the source, find the implementation, and continue into FrameworkManager.
At this point the graph is no longer giving us the promised direct traversal. We are back to a hybrid search-and-read workflow. And that is okay, as long as the integration knows when to switch.
This one did not.
Generic minimal context burns budget too
Every graph-assisted run starts with get_minimal_context because the MCP tool description recommends calling it first.
For both tasks, it returned the same global communities:
html-view
servlets-sip
impl-messageAnd the same unrelated flows:
RESTMBeanServerConnection
getObjectInstance
queryNamesNone of that helps with FeatureManager.updateFeatures or the EnvCheck startup path. The task text influenced the suggested next tools, but it did not make these communities and flows relevant to the symbol we were investigating. That means the first MCP call spent context and tool budget before useful retrieval had even started.
MCP compatibility does not install a navigation policy
This is an easy detail to miss.
code-review-graph install does more than expose tools for clients it directly supports. It can install client-specific configuration, hooks, and instructions.
Bob is not one of those installation targets.
Our .bob/mcp.json gives Bob access to the verbs:
query
traverse
search
inspectIt does not teach Bob a full graph navigation workflow. IBM’s MCP usage guidance recommends clear tool descriptions and custom rules when you want a preferred workflow. That matches what I saw in the transcripts.
Bob needs a rule along these lines:
If a result reports omitted entries, expand it.
Use edge line numbers and receiver metadata.
Treat bare Java targets as unresolved.
Verify unresolved targets with a targeted source read.
Stop once the requested evidence is verified.The current manual integration has none of that.
Compatibility gave Bob the tools. It did not give Bob the navigation grammar.
The recovery policy is the real next experiment
The graph is not the only thing responsible for the failure.
Bob received clear truncation signals and failed to escalate. My benchmark preamble pushed it toward minimal output too aggressively. On the broad task, Bob also tried references_to against the EnvCheck class, got no result, guessed a nonexistent module path, and then burned tool calls recovering with globs and full-file reads.
With embeddings disabled, graph semantic search also only covers names, paths, and signatures. A query like:
KernelBootstrap go launch OSGi frameworkreturned no node. That makes sense given what is indexed. It also means architectural prose and exact-symbol navigation are different benchmarks.
I would make the next experiment three-way:
Bob using native search and source reads
Bob using the current minimal-first graph workflow as the control
Bob using a checked-in graph-specific rule with explicit recovery behavior
The third condition should skip get_minimal_context when the prompt already contains an exact symbol. It should start with standard detail and a result count large enough for the method, inspect receiver metadata and edge lines, expand every truncated response, and verify every unresolved Java target with a focused native source read.
Once it has the requested evidence, it should stop.
Then I would run every condition at least ten times and alternate execution order. All three should get identical native tools, budgets, and output contracts.
Only after that I would add an embeddings condition for natural-language architecture queries. That answers a different question.
And I would change the task set too. Code graphs are designed for things such as impact analysis and review, so the benchmark should include tasks closer to that purpose: find tests affected by a pinned change, trace callers of a modified method, and review a pinned diff.
Conclusion
Bob and code-review-graph are compatible. That worked out as expected. On this pinned Open Liberty benchmark, the integration still failed to beat Bob’s native navigation: the graph-assisted narrow runs were 18.8% faster and returned 18.7% less tool output, but they found one of ten required call sites while solo Bob found all ten. The real result is not necessarly that code graphs are bad it is more about the lesson that smaller context is not automatically better context, and an MCP tool needs a retrieval and recovery policy before its raw efficiency numbers mean much. So, if you ever just pull a random MCP connector and believe it is going to significantly improve anything, think again. None of these integrations are worth a lot without the necessary vocabulary in place.