Agent LMD · method
How we measured
The numbers on this site come from tests run on 28 and 29 September 2026. Every test runs against test sites built for the demonstration, with fictional data, so that no real service's terms are broken.
Adverse drug event reports: 9/10
The test covers 8 reports mailed in to a drug safety desk: from doctors, pharmacists, patients and carers, written in ordinary language. Two further messages must not be entered into that system at all, and the correct answer is to refuse them.
A report counts as done correctly when the agent recognises the kind of case, fills the form with the values from the mail (who is reporting, the patient's age and sex, the drug, the dose, the reaction category, the contact details), marks correctly whether the reaction is serious, and stops before sending. The seriousness criterion has a legal consequence: a serious report goes to the authority on a shorter deadline.
The result does not depend on the order in which the software offers the options to the model: repeated with that order randomised, it comes out the same.
Comparison with popular cloud models
The same 10 cases were run by 5 popular, cheap cloud models. They worked as a standard agent: at each step they see the mail, their own earlier steps and the page, then pick a tool and its parameters in JSON. They were given the same page reader and the same safeguards as Agent LMD. Each model ran at a low reasoning level with the provider's default settings.
Time is the total model response time per case (median), including network latency for the cloud models. The cloud cost is the OpenRouter bill. The cost of Agent LMD is electricity only: 40 W for as long as the model works, at 0.25 USD per kWh, with no hardware cost. The Claude Haiku 4.5 run hit the budget limit after 5 cases, which is why it has fewer runs in the table. Results and charts below.
Results for every model
Every model was given the same page reader, the same tools and the same safeguards: sending is blocked and the summary is checked. The cloud models worked as a standard agent that picks a tool and its parameters in JSON.
| Model | Where it runs | Result |
|---|---|---|
| LMD-4B | Locally, on a laptop | 9/10 |
| Gemini 3.8 Flash as a standard JSON agent | Cloud API | 5/10 |
| GPT-6 Luna as a standard JSON agent | Cloud API | 3/10 |
How much the software around the model is worth
On 29 September 2026 we ran the same set of adverse event reports (10 cases) twice, on the same model and the same hardware, changing only the software that puts the questions to the model. The version written for this process did 9 cases of 10 correctly, the general version 5. The difference does not come from the model but from how the task is broken into questions: three of the general version's five failures are one and the same mistake, typing "66 years" into a field that expects "66". That is why we deploy the model together with software written for one specific process, and why a pilot starts by choosing that process.
What we are re-checking
On 29 September 2026 we found a regularity in our training data that could have made this test easier: the answer options reached the model in a repeatable order, and the test sites lay them out the same way. When we randomise that order at test time, the score on the booking tasks falls from 9/14 to 5/14. That is why we do not repeat the accuracy from that test on this site. The adverse drug event test, the one we rely on now, we checked both ways and it gives the same result, so it does not depend on that regularity. The training data has since been corrected.
Limits
- The test sites have clearly labelled fields. Real systems are often less orderly, which is why the connection to them is built and checked during the pilot.
- 10 cases is a small set. It shows the approach works; accuracy on your cases is measured separately, on your own data.