A benchmark is a very good place to start a technical conversation. It is a dangerous place to end a purchasing decision.
Imagine the meeting. An AI vendor shows a chart. Its system finds more of the right documents than the alternatives. The demonstration works. The engineering team likes the architecture. Someone asks how quickly it can be deployed.
Before answering that question, I would ask two others: Are we allowed to use the exact components that produced that score? And what does it cost to turn our archive into something the system can search?
These are not objections to the technology. They are the beginning of a business case.
My earlier essay, The Numbers Were Right. The Explanation Was Invented., examined the gap between a correct result and a trustworthy explanation. This one concerns another gap: between a result you can reproduce in a research setting and a system you can responsibly buy.
The score is not the purchase
In investment work, an attractive return is the start of the analysis. You still ask how the position is financed, how liquid it is, what happens when the trade gets crowded, and what it costs to get out. Those questions do not make the return irrelevant. They determine whether the return is available to you.
I think enterprise AI deserves the same discipline. A ranking measures something important, but it does not sign the deployment contract, operate the infrastructure, or take responsibility when an update changes the answer.
A leaderboard helps you choose what to test. It does not tell you what to buy.
The question behind our preprint, The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks, coauthored with Kosrow Dehnad, is practical: what information must accompany a retrieval score before that score becomes useful to a buyer?
Permission is part of the product
There is an easy mistake to make when evaluating an open-weight model: if you can download it, run it, and inspect the code around it, it feels as though you can build a product with it.
Those are different permissions.
NVIDIA's NV-Embed-v2 model card lists CC-BY-NC-4.0 and explicitly rules out commercial use. A permissively licensed application wrapped around those weights does not, by itself, replace the model's terms. The buyer needs to review the particular components, versions, and permissions proposed for the deployment—not just the repository's front-page license.
That does not make the research illegitimate. It means that reproducing an experiment and shipping a commercial system are not interchangeable activities.
Nor does the answer have to be “do not deploy.” A separately authorized service, another model, or a different architecture may be appropriate. But a component substitution is a new configuration. It deserves a new measurement.
If the commercial proposal uses a different engine, show me the score with that engine. Not the result from the version we cannot use.
What the comparison found
We compared 13 embedders on 1,000 MuSiQue questions and 11,656 passages. NVIDIA's commercial, self-hostable Nemotron-3-Embed-8B reached 69.79 Recall@5 versus 69.55 for NV-Embed-v2. The paired difference was +0.24 points, with a 95% interval of −0.94 to +1.43 and p=0.69: no detected difference, not proof of equivalence. Results and inference.
| Embedder | Recall@5 | 95% interval |
|---|---|---|
| Nemotron-3-Embed-8B | 69.79 | 68.13–71.43 |
| NV-Embed-v2 | 69.55 | 67.89–71.20 |
The distinction matters. An embedder helps locate relevant text; it is not the whole answering system. Finding a clause is not the same as interpreting it correctly, reconciling it with an amendment, or demonstrating that the person asking is entitled to see it.
I would not turn a narrow numerical lead into a victory announcement. I would turn this result into a reason to test the commercially available option on the documents that matter. NVIDIA's Nemotron model card and license are part of that evaluation, not an afterthought.
The study uses exact search, one principal corpus, and uneven formatting sweeps; it does not establish production performance. Its smaller 2Wiki probe did not reproduce the earlier commercial-reference shortfall. Limitations.
For a buyer, that is the useful level of ambition: a credible candidate, a reproducible test, and a reason to run the next experiment. Not a guarantee delivered by decimal places.
The bill before the first answer
When someone asks what an AI system costs, the answer often begins with the model's price per token. I would begin earlier.
Your files may need extracting, cleaning, splitting, embedding, and organizing into structures that can be searched. Relationships may need to be extracted. A changed document may invalidate more than one result. Someone has to keep the index useful as the archive changes.
That work happens before the polished answer appears on the screen.
KET-RAG's GraphRAG reproductions cost $2.30 and $24.94 to index 5.64 MB under different configurations. Our paper's linear 1 TB extrapolations are about $428,000 and $4.6 million—not measured deployments or Microsoft quotes. Configuration and cost provenance.
Those figures are a prompt for diligence, not a substitute for a quote. Linear scaling from a small corpus cannot establish what a particular customer's build will cost. Document structure, extraction choices, caching, hardware utilization, and update policy can all change the calculation.
What I want from a vendor is a cost explanation I can interrogate. If a number changes when the document chunks get smaller, show the change. If the estimate assumes an overnight batch service, say so. If it excludes people, storage, or the machinery used to plan a query, leave those lines visible.
| Budget line | What to establish |
|---|---|
| Prepare and index | What is processed, how long it takes, and what triggers a rebuild. |
| Answer and verify | Cost per completed task, including retrieval, tools, retries, and review. |
| Operate and protect | Hardware or hosting, storage, permissions, backups, and support. |
| Change or leave | Migration work, exportability, re-embedding, and switching constraints. |
A cheap answer is not necessarily a cheap system. And an expensive system is not necessarily a bad purchase. The missing step is to connect the expense to something the customer actually needs.
Rent or own is not a slogan
I am building a company around customer-controlled AI, so my preference is not a mystery. I want firms to have a credible option to run the work on infrastructure they control.
But “own” is not another spelling of “free.”
A self-hosted model may avoid a provider's per-token charge. It still consumes electricity, memory, storage, engineering time, and operational attention. The server does not patch itself because the weights were downloadable. A backup is not a recovery plan until someone has tested restoring it.
A hosted service can be the better choice when it gets a team into production sooner, absorbs operational work the team cannot support, or meets requirements that a local deployment does not. Those benefits have value. So do control over upgrades, the ability to inspect a fixed configuration, and the option to change suppliers.
The sensible comparison is not “cloud bad, local good.” It is: what are we paying for, which responsibilities remain ours, and what happens when the workload changes?
For an occasional, small workload, buying hardware may be difficult to justify. For a continuously changing private archive, a recurring processing bill and the freedom to revise the pipeline deserve serious attention. Neither conclusion follows from a benchmark score alone.
And neither ownership nor a graph structure establishes governance. If access control, deletion, audit trails, or reproducibility are part of the reason for paying more, demonstrate those properties in the deployed system. Do not infer them from the architecture's name.
The test we owe the buyer
I would put five requests in front of any vendor—including Toryx:
- Show me the deployable configuration. Name the models, versions, licenses, and services in the proposal.
- Run my questions on representative documents. Include ambiguity, updates, and the cases where finding one passage is not enough.
- Separate the first build from the next twelve months. Show what changes when the corpus, query volume, or model changes.
- Show the failures and the operating burden. Include review time, unavailable services, permission tests, and recovery work.
- Give me an exit path. Establish what can be exported, what must be rebuilt, and what is tied to a provider.
A pilot should turn those requests into observable work. Agree the corpus and questions. Define what success means before seeing the answers. Measure both the result and the effort needed to produce it. Then decide whether the improvement is worth paying for.
I have a commercial stake in that argument. The paper is not a benchmark of the complete Toryx product or evidence of customer return on investment. It is part of the discipline I want our own product to face.
The score is where the conversation starts. The business case begins when somebody asks what the score left out.
The first essay asked whether the explanation could survive inspection. This one asks whether the deployment proposal can. Both questions matter before an AI system becomes part of consequential work.
Bring the workflow. We will define the test.
Interested in evaluating private document AI through a 12-week Toryx pilot? Start with the workflow, deployment requirements, and questions your team needs answered. Scope and success criteria are agreed together. Please do not send documents in the first inquiry.
Discuss a design-partner pilotFor the research, investment, or press conversation, contact me on LinkedIn or use the inquiry links below.
Sources & scope
- Luis M. Sánchez and Kosrow Dehnad, The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks, arXiv:2608.16096v1, 17 August 2026. Public preprint, not peer reviewed. Table 1 and §4.2: retrieval comparison; §§4.4–4.5: cost provenance and extrapolation; §6: limitations.
- Replication archive: Zenodo DOI 10.5281/zenodo.21972866. Public code and result artifacts. These are the paper's replication resources, not a customer-performance guarantee.
- NVIDIA primary model cards: NV-Embed-v2 and Nemotron-3-Embed-8B-BF16. Review the actual license and service terms for the intended use.
- Companion essay: The Numbers Were Right. The Explanation Was Invented.
Research values refer to the cited version and its measurement conditions, not today's vendor quotes. The purchasing worksheet and recommendations are the author's perspective; they are not measured outcomes or legal advice. This essay was drafted with AI assistance from the cited research.