Services
AI & Software
Solutions
Training
Tools
Insights
References
Contact
Free initial consultation Ask a first question by email
About Deutsch
← AI & Software

Model selection & test catalogue

Test models with your own domain questions instead of spec sheets: quality, cost per task, privacy and failure risk, plus a clear switching plan.

Model choice is often decided by gut feeling: the best known, the most expensive, or the one that appeared in a recent comparison. In operation, the model does not decide the value; the question is whether it solves your tasks reliably and what a single task costs.

How I work: with source · with approval · in your system.

Starting point

A business collects use cases: check proposal texts, classify invoices, answer questions about project documentation, sort emails. Each case has candidates, local or in the cloud. Vendors promise similar abilities. What is missing is a test against the real cases.

What I do

I build a test catalogue with you from real domain questions. Not generic examples, but tasks from daily work, with expected answers and a clear scoring rule. The catalogue stays in-house and grows.

Then I test the candidates against it and measure four things: hit rate, cost per task, privacy and failure risk. For local models, hardware comes on top; for cloud models, the question of where the data sits and what happens when the vendor changes the service.

The outcome is a decision paper with numbers, a prompt library for the main cases, and a plan for a later switch. A model is not a marriage; the switch must be prepared.

Result

You get a decision paper with test results and cost calculation, a prompt library and a switching plan. The documents also show which cases a model does not solve reliably and which therefore stay with people.

Example

I switched models in my own agent when a local model became much faster at the same task; how that switch runs is described in the article on OpenClaw, a local agent without cloud. In a manufacturer engagement, a test catalogue of domain questions was part of the job; the engagement is anonymous, its numbers stay internal and do not appear on this page. How token costs add up and why the vendor’s invoice is not always right is in the article on AI token economics.

Limit

A measurement is a snapshot. Models are updated, prices change, your questions remain. The test catalogue is the lasting asset, not today’s result. I do not publish comparisons with named vendor products; internally it is possible.

Your effort: you provide the domain questions and say which answer counts as correct. That work cannot be shortened, because a catalogue built from my examples says nothing about your business.

How it runs

  1. Collect use cases and rank them by frequency and risk
  2. Build the test catalogue with expected answers and a scoring rule
  3. Measure candidates, calculate cost per task
  4. Hand over the decision paper and prompt library
  5. Define the switching plan and a review routine for later updates

Common questions

How many questions does a test catalogue need?

Enough that each use case area is covered by both common and difficult cases. The number follows from that, not from a quota: an area with few exceptions needs fewer cases than one with many edge cases.

Is a local model worth it?

When data must not leave the building or unit cost must stay low. For complex texts, a cloud model is often stronger. The test shows whether the difference matters for your task.

Who maintains the test catalogue?

Your domain team, with a short briefing. New cases come from operations, not from a marketing list.

Can we switch models later?

Yes, if the interface is properly encapsulated and the test catalogue remains. Both are part of the result.

The next step costs nothing.

30 to 60 minutes, no strings attached: where is the biggest friction, what is the most effective lever, and is a collaboration a fit?