Agentic development is reaching people who never planned to become software engineers. Business analysts, finance teams, and many others are starting to use agents for their daily work. Exciting times to life in. It also shows where our tools still assume too much or aren’t’ the right fit for non developer roles.
An IDE is already unfamiliar territory if your normal tools are Excel, PowerPoint, and a browser. Suddenly there is a project tree, a terminal, several panels, and a chat that can change files. Even finding the file you just downloaded can feel harder than it should. Not speaking of workspaces or conversations that are tied to them.
Then the real data arrives. A spreadsheet may have hundreds of thousands of rows, and a report may contain many screenshots. The natural reaction is to attach everything to the little chat box and ask the agent to sort it out. Large files quickly fill the model’s context, which is its working space for the task, and every image adds processing cost. A screenshot also turns cells into pixels. You can see a number, but its cell address, formula, and surrounding workbook structure are gone.
Software engineers usually handle large systems through small interfaces. A database gets a query. A large log gets a search or filter. We can give Excel the same kind of interface before an agent gets to works with it. Wondering when Microsoft comes up with the first headless Excel after all.
I wanted one example that makes this concrete for people who live in spreadsheets. We use a 44 MB workbook with 1,067,371 real transactions. Unpacked, the file is 317 MB. A plain text dump of its formatted cells has more than 86 million characters and to answer the example question we need four evidence rows from it. This is way too much for every context window out there.
I’ve broken this down into steps. The agent first asks for a map of the workbook. Then it runs one defined calculation and reads a few exact rows to check the result. Images stay in the file until we choose one. This keeps the context small, lowers repeated input cost, and makes the answer easier to verify.
We build the local tool with Apache POI, run it with JBang, and teach IBM Bob when and how to use it. And because I wanted an example for handling very large Excel files, the parser is written in Java. Of course.
What We Build
We give the agent four commands:
inventorymaps sheets, dimensions, formulas, and embedded imagesaudit-retailanswers one documented business question across both year sheetsslicereturns one explicit cell range with formulas and cached valuesimagesandextract-imagelist visual assets, then expand one selected image
An IBM Bob skill explains Bob when each command is appropriate. We also generate a small test workbook with a hidden sheet, a stale formula result, and an embedded PNG. The retail dataset has none of those, and I still want to know that the tool handles them.
What You Need
The commands are ready to run, so prior Apache POI knowledge is optional. Basic terminal use helps, and I explain what the Java code does where it matters. The download is about 44 MB, and the full tutorial takes roughly the time it takes me to drink two ☕️☕️.
Java 21 or newer
JBang 0.138.0 or newer
IBM Bob with a local workspace (Get your free trial here if you like)
curl,unzip, andjqPython 3.11 or newer for the independent verification
Get the Real Workbook
I chose UCI Online Retail II because it feels like a workbook somebody might send you at work. It contains two years of transactions for a UK-based online retailer. There are 1,067,371 rows, missing customer IDs, cancellations, and stock codes that are not merchandise. UCI publishes the dataset under CC BY 4.0.
Clone the example:
git clone https://github.com/myfear/the-main-thread.git
cd the-main-thread/excel-context-xrayDownload the workbook:
./scripts/download-data.shExpected final line:
/path/to/the-main-thread/excel-context-xray/data/online_retail_II.xlsxThe script checks the ZIP and the extracted workbook with a SHA-256 file fingerprint. You should get:
bcbe73b35f5b7babf197fb0cb983a11f5d9ff929078d4aa53d171b1f2df2e980This makes sure we run the audit on the same file.
Put Apache POI Behind a Small Interface
There is no Maven project to build for this tool. scripts/ExcelXray.java is a JBang source file, so its dependency declarations live at the top of the Java file. JBang downloads those JARs once and then runs the source like a command:
jbang scripts/ExcelXray.java --helpThe first run downloads Apache POI, Jackson, Picocli, and Log4j. After that we use --offline. This also makes Bob’s command predictable because it cannot fetch a new dependency during the audit.
Apache POI can load a workbook through its XSSF user model. That API is perfect when you need to edit cells, but it keeps much more workbook state in memory. For this read-only job we use POI’s event model. XSSFReader opens the workbook XML files, and XSSFSheetXMLHandler gives us one row at a time.
The tool does five things before it prints anything:
It checks that the input is an
.xlsxfile and caps the compressed size at 100 MB.It inspects the ZIP structure and caps expanded bytes at 1 GB and entries at 1,000.
It streams worksheet XML and keeps counters, totals, and a fixed number of evidence rows.
It serializes one JSON result.
It fails before printing when the JSON exceeds
--max-output-chars.
The limits are build directly into the example and not parameters so you can see the boundary. A higher quality skill should take the values from a policy and run the parser in a worker with its own memory and time limits. Excel files are ZIP files, and a ZIP file from an unknown source deserves all the suspicion you can imagine.
Inventory before analysis
There is no Maven project to build for this tool. scripts/ExcelXray.java is a JBang source file, so its dependency declarations live at the top of the Java file. JBang downloads those JARs once and then runs the source like a command:
jbang scripts/ExcelXray.java --helpThe first run downloads Apache POI, Jackson, Picocli, and Log4j. After that we use --offline. This also makes Bob’s command predictable because it cannot fetch a new dependency during the audit.
Apache POI can load a workbook through its XSSF user model. That API is easy to use when you need to edit cells, but it keeps much more workbook state in memory. For this read-only job we use POI’s event model. XSSFReader opens the workbook XML files, and XSSFSheetXMLHandler gives us one row at a time.
The tool does five things before it prints anything:
It checks that the input is an
.xlsxfile and caps the compressed size at 100 MB.It inspects the ZIP structure and caps expanded bytes at 1 GB and entries at 1,000.
It streams worksheet XML and keeps counters, totals, and a fixed number of evidence rows.
It serializes one JSON result.
It fails before printing when the JSON exceeds
--max-output-chars.
I keep these limits directly in the example so you can see the boundary. A production service should take the values either from its upload policy or a configuration parameter and run the parser in a worker with its own memory and time limits. Excel files are ZIP files, and a ZIP file from an unknown source deserves some suspicion.
Inventory before analysis
First, ask for a map of the workbook:
jbang --offline scripts/ExcelXray.java inventory \
data/online_retail_II.xlsx \
--sample-rows 0 \
--max-output-chars 12000 |
jq '{
workbook,
sheets: [
.sheets[] |
{
name,
visibility,
rowsIncludingHeader,
dataRows,
nonEmptyCells,
formulaCells,
imageCount: (.images | length)
}
],
summary,
contextBudget
}'Expected output:
{
"workbook": {
"path": "/path/to/excel-context-xray/data/online_retail_II.xlsx",
"fileBytes": 45622278,
"expandedBytes": 317305853,
"zipEntries": 11
},
"sheets": [
{
"name": "Year 2009-2010",
"visibility": "visible",
"rowsIncludingHeader": 525462,
"dataRows": 525461,
"nonEmptyCells": 4092841,
"formulaCells": 0,
"imageCount": 0
},
{
"name": "Year 2010-2011",
"visibility": "visible",
"rowsIncludingHeader": 541911,
"dataRows": 541910,
"nonEmptyCells": 4198754,
"formulaCells": 0,
"imageCount": 0
}
],
"summary": {
"sheetCount": 2,
"cellCount": 8291595,
"formulaCellCount": 0,
"imageCount": 0
},
"contextBudget": {
"serializedCellCharactersIfDumped": 86474183,
"note": "This is a character count for formatted cell text, not a model token count."
}
}This output tells us what we are dealing with. Both year sheets contain data, so both belong in the audit. The workbook has no formulas and no images, which means neither can affect this result. Bob can stop looking for extra tabs and screenshots.
The final number is a character count. I am not using token count here, because that depends on the model and tokenizer that I do not know for Bob.
Define the Business Question
The parser can read a stock code such as M. We still have to tell it that M is a manual adjustment in this dataset.
The audit question is:
Across both workbook years, which customer outside the United Kingdom produced the most net revenue, and which merchandise product produced the largest value of returned goods? Explain why a naive product ranking is wrong.
The skill in .bob/skills/excel-xray/references/retail-audit.md defines the calculation:
Line amount is
Quantity * PriceNegative line amounts reduce customer net revenue
Missing customer IDs remain in workbook totals and stay out of the customer ranking
Returned value uses negative quantities with positive prices
Merchandise codes match
[0-9]{5}[A-Z]?Both year sheets are included
These business rules define the result. UCI describes a normal product code as a five-digit number. The workbook also uses codes for manual entries and other operations. If we rank every code, M wins and we end up calling an adjustment the most returned product.
Run the audit:
jbang --offline scripts/ExcelXray.java audit-retail \
data/online_retail_II.xlsx \
--evidence-lines 3 \
--max-output-chars 12000The command reads the workbook twice. During the first pass it calculates the totals and finds the winning IDs. The second pass keeps only the largest evidence rows for those winners. This gives us exact cell references while memory stays bounded.
The answer is:
{
"topNonUkCustomer": {
"customerId": "14646",
"country": "Netherlands",
"netRevenueGbp": 523342.07
},
"largestReturnedProduct": {
"stockCode": "23843",
"description": "PAPER CRAFT , LITTLE BIRDIE",
"returnedValueGbp": 168469.60
},
"dataQualityTrap": {
"naiveLargestReturnCode": "M",
"description": "Manual",
"returnedValueGbp": 423886.17,
"excludedBecause": "The stock code does not match the five-digit merchandise code shape and represents an adjustment."
}
}Four rows support the answer. Three customer rows come from Year 2010-2011!A421603:H421603, Year 2010-2011!A534954:H534954, and Year 2009-2010!A330926:H330926. The returned-product evidence is Year 2010-2011!A540424:H540424.
Inspect that return directly:
jbang --offline scripts/ExcelXray.java slice \
data/online_retail_II.xlsx \
--sheet "Year 2010-2011" \
--range "A540424:H540424" \
--max-output-chars 12000Expected row:
{
"row": 540424,
"cells": [
{
"cell": "A540424",
"value": "C581484"
},
{
"cell": "B540424",
"value": "23843"
},
{
"cell": "C540424",
"value": "PAPER CRAFT , LITTLE BIRDIE"
},
{
"cell": "D540424",
"value": "-80995"
},
{
"cell": "E540424",
"value": "12/9/11 9:27"
},
{
"cell": "F540424",
"value": "2.08"
},
{
"cell": "G540424",
"value": "16446"
},
{
"cell": "H540424",
"value": "United Kingdom"
}
]
}The row belongs to customer 16446 in the United Kingdom, so it counts toward the product return and stays out of the non-UK customer ranking. We can also reproduce the amount from the row: 80995 * £2.08 = £168469.60.
Dumping every formatted cell would produce 86,474,183 characters. The complete audit response, including its four evidence rows, is 3,311 bytes. The small response is more than 26,000 times smaller by those measurements. Again, this compares characters and bytes. It is not a model token count.
Teach Bob the Narrowing Loop
Repeating this workflow in every chat would defeat the purpose. IBM Bob loads project skills from .bob/skills, so we can keep the rules next to the code and share them with the team. Bob uses the skill description to decide when the workflow applies.
Put these core instructions in .bob/skills/excel-xray/SKILL.md:
---
name: excel-xray
description: >
Inspect large or complex XLSX workbooks with this repository's JBang and
Apache POI analyzer before answering. Use for workbook structure, formulas,
bounded cell evidence, the UCI Online Retail II audit, or embedded images
when the binary must stay outside the agent context.
---
# Excel X-Ray
Keep the workbook on disk. Bring only a bounded inventory, aggregate, cell range, or selected image into context.
## Required Workflow
1. Confirm the input is an `.xlsx` file inside the workspace. Replace `$WORKBOOK` below with that exact path before running the command. Do not modify the input, encode it as text, or dump its ZIP/XML contents into the conversation.
2. From the repository root, inventory it first:
```bash
jbang --offline scripts/ExcelXray.java inventory "$WORKBOOK" \
--sample-rows 2 \
--max-output-chars 12000
```
3. Turn the user's request into one narrow question. State the metric, filters, grouping, and treatment of missing values before calculating.
4. Use the smallest command that can answer it:
- For the tutorial's retail question, read [the retail audit contract](references/retail-audit.md), then run `audit-retail`.
- For exact cell evidence, use `slice` with one sheet and one explicit range.
- For visuals, run `images` first. Use `extract-image` for one selected index only.
5. If a command reaches `--max-output-chars`, reduce the sample or range. Do not remove the cap just to make the command succeed.
6. Cite workbook evidence as `Sheet!A1:H20` ranges. Keep conclusions separate from assumptions and data-quality rules.
## Accuracy Rules
- Treat formula results as cached workbook values unless a spreadsheet engine has recalculated them. Return both formula and cached value when they matter.
- Call a number a character count when the tool reports characters. Do not convert it to tokens without a named tokenizer and model.
- Do not call every stock code a product. Apply documented business rules before ranking.
- Do not hide excluded or missing rows. Report the applied policy.
- Do not claim that a workbook image proves a numeric result unless its visible content agrees with cell-level evidence.
## Final Answer Shape
Return:
1. The direct answer.
2. The calculation scope and business rules.
3. A small set of sheet-and-range evidence.
4. Any formula-cache, missing-data, image, or file-format limitation that could change the conclusion.The retail rules live in a separate reference file. Bob reads them when the workbook matches this audit. A different Excel task can use the same inventory and slice commands without carrying retail-specific rules through the whole conversation. This is also a good composition approach for a command that could easily serve more purposes than just this specific retail calculation. A good reminder that skills should ideally be designed in a reusable way.
Open the excel-context-xray directory in Bob and send this prompt:
Use the excel-xray skill to audit data/online_retail_II.xlsx.
Across both workbook years, which customer outside the United Kingdom
produced the most net revenue, and which merchandise product produced
the largest value of returned goods?
Explain why the naive ranking is wrong. Run inventory first, keep the
workbook outside the conversation, and cite exact sheet ranges.Bob should return customer 14646 in the Netherlands with £523,342.07 net revenue. The returned product should be stock code 23843 with £168,469.60. The explanation also needs the naive M / Manual result and the merchandise rule that excludes it.
Java calculates the result and returns a small evidence package. Bob uses that package to explain the answer, which leaves room in the context for assumptions and follow-up questions.
Add Formulas, Hidden Sheets, and Images
The retail workbook gives us the large-row test, but it has no formulas or images. We need a second workbook for those cases. CreateFixture.java generates it:
jbang --offline scripts/CreateFixture.javaExpected output:
/path/to/excel-context-xray/build/visual-fixture.xlsxThe generated workbook contains:
A visible
Dashboardsheet with one formulaA visible
Transactionssheet with three rowsA hidden
RulessheetA named range
One embedded PNG evidence card
The generator evaluates SUM('Transactions'!D2:D4) with values of £10, £20, and £30. Then it changes the last value to £50 and saves the workbook without recalculating. The current cells add up to £80, while Excel’s stored result still says £60.
Which value should the agent report? Both values matter. £60 explains what the dashboard shows, and £80 explains why that dashboard is stale.
Ask for the formula and cached value:
jbang --offline scripts/ExcelXray.java slice \
build/visual-fixture.xlsx \
--sheet Dashboard \
--range A1:D5 \
--include-formulasExpected output:
{
"sheet": "Dashboard",
"range": "A1:D5",
"formulaCellsInSheet": 1,
"rows": [
{
"row": 1,
"cells": [
{
"cell": "A1",
"value": "Returns investigation"
}
]
},
{
"row": 3,
"cells": [
{
"cell": "A3",
"value": "Cached returned value"
},
{
"cell": "B3",
"value": "£60.00",
"formula": "SUM('Transactions'!D2:D4)"
}
]
}
]
}Excel stores a cached result beside every formula, and Apache POI can read it quickly. Our fixture shows the risk: the cached value can be old. When a decision depends on the current result, recalculate a copy with a compatible spreadsheet engine and leave the source file unchanged.
Now list image metadata:
jbang --offline scripts/ExcelXray.java images \
build/visual-fixture.xlsx |
jq '{
imageCount,
images: [
.images[] |
{index, sheet, anchorRange, contentType, extension}
]
}'Expected result:
{
"imageCount": 1,
"images": [
{
"index": 1,
"sheet": "Dashboard",
"anchorRange": "not-resolved",
"contentType": "image/png",
"extension": "png"
}
]
}The full command also returns the byte size and SHA-256. The hash helps us identify one asset when the workbook contains several similar images. So far Bob has only metadata; the image bytes are still outside the conversation.
Extract the selected image:
jbang --offline scripts/ExcelXray.java extract-image \
build/visual-fixture.xlsx \
--index 1 \
--output build/extracted-evidence.pngNow Bob can inspect build/extracted-evidence.png as one image. The card shows invoice C100003, stock code 23843, and an expected value of £50. It agrees with the transaction row, while the dashboard formula stays stale at £60.
This version does not resolve the image’s exact cell anchor. It maps the image to its sheet, hash, and content type. If your question depends on placement, add drawing-anchor parsing before asking the model to infer where the image belongs.
How Agent Harnesses Parse XLSX Files
An .xlsx file is a ZIP container holding worksheet XML, relationships, styles, shared strings, and media. A model cannot reason over that container directly. Something in the harness has to open it, choose what to keep, and turn the result into model input or tool output.
IBM Bob: direct workbook context
IBM Bob’s current changelog says Bob can read .xlsx files directly. IBM does not document the parser, row-selection policy, formula handling, or image extraction behind that feature.
The extracted content and later tool results consume context. Bob’s context window documentation puts file reads, @ mentions, and tool output in the Messages category, which is processed again on later turns. This project therefore keeps data/ in .bobignore and gives Bob a smaller path: run the Java analyzer, then read its JSON result.
OpenAI API: sampled spreadsheet augmentation
The OpenAI API documents its parsing policy. The input_file spreadsheet flow parses up to the first 1,000 rows of each sheet, then adds generated summary and header metadata. It does not place the complete workbook in model context. Embedded images and charts in non-PDF files are not extracted.
That path is appropriate for a quick overview. It cannot prove a winner stored near row 540,424. For joins, aggregation, charting, and custom calculations, the same guide recommends Hosted Shell, where code can work beside the file.
Codex spreadsheet skill: import, then inspect
The built-in spreadsheet skill available in my Codex uses a harness-provided workbook runtime called @oai/artifact-tool. Its normal read path imports the workbook, then asks for bounded views of sheets, regions, formulas, drawings, or exact cell ranges:
import { FileBlob, SpreadsheetFile } from "@oai/artifact-tool";
const input = await FileBlob.load("workbook.xlsx");
const workbook = await SpreadsheetFile.importXlsx(input);
const map = await workbook.inspect({
kind: "workbook,sheet",
maxChars: 6000,
tableMaxRows: 6,
tableMaxCols: 8
});That package is part of the harness, not a dependency someone has to install, so it is very convenient to use.
This is a real spreadsheet object rather than a text dump. On our 17 KB fixture, the import itself completed in about 0.35 seconds. The runtime found all three sheets, the formula in Dashboard!B3, its stored value of £60, and the embedded image anchor.
I tried the same import-first path on the 44 MB retail workbook. After 210 seconds it had not reached the first sheet inventory, so I stopped it. That does not make the runtime a bad spreadsheet tool. It means whole-workbook import is the wrong first operation for this file.
For this audit, my built-in skill should route around its default parser:
Run the streaming POI
inventorycommand against the original file.Run
audit-retailfor the deterministic calculation.Give the model only the JSON aggregates and exact evidence rows.
Use the spreadsheet runtime later for a small derivative workbook or selected range when editing, formula tracing, or rendering matters.
The built-in skill is still doing its job. A good skill chooses the parser that fits the file instead of insisting that every workbook enter the same runtime.
Claude: files plus sandboxed Python
Claude takes another executable path. The Claude API code execution tool accepts Excel files through the Files API, mounts them into a sandboxed container, and lets Claude write and run Python to parse and analyze them. The model receives the code-execution results rather than every workbook cell by default.
Claude’s consumer file workflow currently has a 30 MB per-file upload limit, so this 44 MB workbook does not fit that direct Claude chat path. The Anthropic Files API accepts files up to 500 MB. An API workflow can upload this workbook once, pass its file ID to the code-execution container, and return only the bounded results. The binary does not have to travel inside a Messages API request, which has its own 32 MB limit.
That solves the transport problem. The parsing code and its output boundary still remain part of the workflow design.
The common design is simple even when the products differ:
Keep the original binary in file or object storage.
Choose a parser that can handle its real size and features.
Create a cheap structural map.
Run deterministic calculations beside the file.
Return a fixed amount of evidence with stable locations.
Expand one image or range when the question needs it.
Upload size, parser memory, tool output, model context, and billed input are separate budgets. A good harness can manage all five. A good skill tells it which path to use for this workbook.
What a good skill does
Bob’s packaged spreadsheet support inside the harness is a useful general-purpose hammer. Attach an .xlsx file, ask a question, and let the harness prepare something the model can read. That is a good way to explore a smaller workbook or start an unfamiliar task.
Our audit needs a more specific tool. We must scan both year sheets, apply one merchandise rule, handle missing customer IDs consistently, and return exact cell ranges without filling the conversation with a million rows. A general file handler cannot know those decisions because they belong to this dataset and a specific business question.
In Agent Skills Need Guardrails, Not Just Prompts, I described a skill as an agent-facing workflow contract. Excel X-Ray makes that definition very clear:
The skill description tells Bob when the workflow applies.
SKILL.mddefines the order: inventory, load the audit contract, run the narrow command, and report a fixed amount of evidence.The reference file owns the retail rules, including the merchandise code pattern and missing-customer policy.
Java and Apache POI own parsing, counting, and arithmetic.
Output caps decide how much can enter the model context.
The fixture and independent Python audit decide whether the result is correct.
This isn’t turning Bob into a better spreadsheet parser. It stops Bob from improvising the parsing and business rules.
What is important is to test your skills. When I did the first naive run it followed the workflow because my prompt repeated it. In the next run, Bob loaded the skill but skipped the inventory, returned a much longer report than requested, and said the workbook was never read. The numbers were correct, but the contract was loose. Apache POI had read the file from disk; the binary simply had not entered the model context.
Only after I made the command order explicit, changed inventory to return zero sample rows, capped the final explanation, and added the accurate context statement to the skill this changed to the better. The next run activated the skill, inventoried the workbook, read the contract, ran the audit, and returned the four evidence rows. This is the same engineering loop I wrote about in Why Quarkus Agent Skills Matter More Than Another Model Upgrade: a bad run should become a missing guardrail or tool fix, not a vague complaint about the model.
For me a practical definition of a good skill is: It has a narrow trigger, an ordered workflow, named sources of truth, deterministic tools for fragile work, explicit context and output limits, coupled with honest stop conditions, and a test that runs through the real harness. Once a skill changes execution, it also needs an owner and code review. The skillsaw walkthrough covers that maintenance side: lint the files, pin the rules, run CI, and keep the installed copy traceable.
Narrow support is part of this design. Excel X-Ray accepts .xlsx and rejects legacy .xls, macro-enabled .xlsm, password-protected workbooks, and arbitrary ZIP files. slice --include-formulas returns a formula together with Excel’s stored value, but only a compatible spreadsheet engine can recalculate a copy reliably. Images start as count, type, size, and hash; we only extract the selected image, then crop or resize it when the question allows. The original stays available for audit. Shared strings, styles, image data, and very wide cells still consume memory, so a production service also needs CPU and wall-clock timeouts, isolated workers, authenticated uploads, and storage quotas.
For another domain, keep the reusable boundary and replace the business command. An invoice skill can reuse inventory, slice, image selection, and the output cap, then add audit-invoices with explicit rules for invoice identity, tax, and duplicates. The packaged harness remains the general-purpose tool. The project skill is the purpose-built tool for an answer we need to reproduce and defend.
Verify the Whole Path
The repository checks the generated fixture and runs a second implementation of the retail audit. That cross-check uses Python’s standard ZIP and XML libraries, so a POI bug or Java aggregation bug is less likely to produce the same answer twice.
Run everything:
./scripts/verify.shExpected output:
Cross-check matched /path/to/excel-context-xray/verification/expected-audit.json.
Fixture and full-workbook checks passed.This covers both parts of the example. The small fixture verifies the hidden sheet, stale formula value, and selected image. The full workbook check verifies the file hash, row counts, totals, winners, business-rule trap, and evidence limit.
Finally, start a fresh Bob task and repeat only the audit prompt. The wording may change, but the customer, product, rule, and evidence ranges should stay the same.
Conclusion
People who work in Excel should be able to use agents without feeding every cell and screenshot into a chat or burning endless token just because they have bad data or no engineering background to prepare it for an agent. We kept the 44 MB workbook on disk, let Java scan it, and gave Bob 3,311 bytes of answer and evidence. The model context now contains the business question, the rules, and the rows that prove the result.