An AI demo is usually built around 5 or 6 well-chosen examples, and with those the answers look excellent. When the same feature has to handle hundreds of real cases, with typos, half-filled records and requests nobody anticipated, the results change, sometimes a little and sometimes a lot. Without a way to measure, nobody can say which of the 2 is happening, and the decision to put it in front of clients ends up being made on gut feeling. This is how we measure our AI features, with examples from real projects. It's meant for whoever has to decide whether a demo goes into production and for whoever has paid for an AI feature and wants to check that it does what was promised.

Before writing any code, we agree with the client on 3 things and write them down.

• A business number. Time saved per task, appointments booked, records completed, cases closed without escalation. It's the number the person paying for the project cares about.
• A technical number. Usually, the share of correct answers on a set of test cases. It tells us whether the model is doing its job before we look at the business.
• The threshold. The value from which the feature is worth it. If a person catalogues 40 records an hour and the AI gets 6 out of 10 right, it may still save time if fixing a suggestion is quicker than writing from scratch, or it may not. You have to work it out for each case.

We also write down the baseline, meaning what happens today without AI or with the simplest possible solution. For a recommender, the baseline is usually a "most popular first" list, and the recommender has to beat it to justify the extra complexity.

A test set (an evaluation set) is a collection of real cases, each with the answer a person considers correct. Every version of the feature is compared against it.

The cases come from the client's real data: documents, records, requests that have already been handled. We include the easy ones, the common ones and the ones that cause trouble, such as poor-quality images, incomplete records or ambiguous questions. A few dozen well-chosen cases are enough to start with, and the set grows every time production turns up an error worth keeping.

The correct answers are reviewed by someone who knows the subject, for example an archivist for poster descriptions or a coordinator for matches between patients and psychologists.

The set is run again every time the model, the instructions given to the model (the prompt) or the data it works with changes. In Carte, the poster catalogue we built, the AI model for each feature can be chosen from the admin panel. That's handy for trying a newer or cheaper model, and every change is a reason to run the test set again before leaving it in place. We keep traces and test sets in Langfuse, which lets us compare 2 versions case by case and see which ones improved and which ones got worse.

Once the feature is live, the most useful information is what people do with each answer. They can accept it as it is, correct it or ignore it.

For that to be measurable, it has to be designed in. In Carte, every catalogue field keeps 3 things separately: the original data from the archive, the AI's suggestion and the data a professional has validated. The AI never overwrites the original record. Because the 3 are stored apart, the share of suggestions accepted without changes comes straight out of the database, and so do the fields that get corrected most often, which are usually the ones to improve first.

We also look at the cases that go to a person. We prefer a feature that hands the case over when it isn't confident enough to one that always returns an answer. The number to watch is how many cases go to review and whether that number goes down as the feature improves. 

A correct answer that takes 20 seconds may be no use to someone on a call with a client, and if each query costs too much, the cost can outweigh the time saved. So we measure, for each version, the average cost per answer and the response time, along with the correct answers, and compare them against the threshold agreed at the start.

This is also where the decision to switch models gets made. New models come out every few months, often cheaper or faster. Before switching, we run the test set with the new model and compare correct answers, cost and response time. If it's equal or better on all 3, we switch. If it gains on cost and loses on quality, the client decides with the numbers in front of them. 

Carte: cataloguing with human review

Carte is a digital catalogue of more than 7,500 historical posters. The AI analyses each poster and suggests descriptions, iconography and themes, and a professional reviews and approves each suggestion before it becomes validated data. Because the accepted, corrected and rejected suggestions are stored separately, they show where the AI helps and where it doesn't. When we test another model, we run the same test cases and look at those same figures.

La Psicóloga en Casa: recommending a psychologist

In La Psicóloga en Casa, the AI suggests which psychologists fit each case based on their specialties and availability, and the coordinator makes the final choice. The clearest signal is how often the coordinator picks someone from the suggestions and how often she looks elsewhere. Every time she goes outside the list, it's worth asking why: perhaps a psychologist's profile is incomplete, or there's a criterion the model isn't taking into account. We explain this kind of measurement in more detail in how to add a recommendation system to your product.

Reading invoices in a campaign

In a campaign for installers, users upload their purchase invoices and the AI reads the amounts. When it can't read an amount with enough confidence, the invoice goes straight to "pending review" and the AI doesn't guess. The figures we watch here are how many invoices are read without problems and how many end up pending, because those are the ones that cost time.

If you're hiring someone to build an AI feature, or buying one off the shelf, ask the vendor these questions before signing.

• How will we know it works? Which business number and which technical number, and from what threshold is it worth it?
• Which test cases will you use, where do they come from and who reviews them?
• What happens when you change the model or the instructions? Do you run the tests again before releasing the change?
• What do you record in production? Can I see which answers were accepted, corrected or ignored?
• What does the AI do when it isn't sure? Does it pass the case to a person?
• How much does each answer cost and how long does it take, and how will you report on it?

If most of the answers are vague, there will be no way to check whether the feature does what was promised. The same principle is behind our corporate RAG projects, and it's how we approach all our custom AI development work.

If you have an AI feature and don't know whether it works, tell us what it does and we'll tell you how we would measure it.