Can a small, local model diagnose network faults?

Network teams in closed environments have a problem: the AI tools that could help them live in the cloud. Many of these networks are air-gapped by policy, so sending configuration to an outside service is not an option. We wanted to know if a model small enough to run on one ordinary machine could still do real diagnosis.

How we built the test

We did not want a quiz. Every question in the exam is a real fault, injected into a live virtual network the model has never seen. The model gets the same read-only tools an engineer would use, investigates, and names the fault class, the device and the root cause.

• 150 questions, kept out of training
• 51 network topologies the model never saw
• 18 fault classes
• Read-only tools: the model can look, never change
• Deterministic scoring: no human or AI judge

The results

The same base model, before our training, solved 11%. After training in our synthetic network lab, Kestrel solved 68%. That is a jump of 57 points on faults it had never seen.

Why we show the cloud model

Claude Sonnet 5 is a much larger cloud model. We include it as a reference point, not as a rival. It needs an internet connection and costs money for every question. Kestrel runs offline on your own hardware. The fair comparison is accuracy, privacy, cost and hardware together.

What it means

A small model trained for one job can do real work where cloud AI is not allowed. 68% is not 100%, and we say so. Kestrel is the model behind Auspex, and every new round of training is measured on the same exam.