Our model comparisons

Qwen and Foundation-Sec on the same 277 cases

These engine v2 runs used the same 26 attacks and 251 harmless alerts, with behaviour rules and guards enabled. Program evidence has a favourable history baseline that separates the classes by construction, so those results are upper bounds. Host evidence avoids that particular flaw but still uses narrow lab recordings.

All remaining cases went to a person. With program evidence, Qwen left 3 attacks and 47 harmless alerts to a person; Foundation-Sec left 3 attacks and 187 harmless alerts. With host evidence, Qwen left 5 attacks and 43 harmless alerts; Foundation-Sec left 3 attacks and 191 harmless alerts. No case had an execution error in these four runs.

Qwen cleared more harmless alerts in these runs. With host evidence, Foundation-Sec confirmed two more attacks. These are Hydra system outcomes after code checks, not standalone model accuracy. Granite was not measured on this 277-case set.

Download the matched model comparison and run-file hashes (JSON) →

Qwen on the same alerts, three evidence settings

Each setting used the same 277 recorded cases: 26 attacks and 251 harmless alerts. The behaviour rules, dismissal guard and evidence checks were enabled in all three runs. A case sent to a person is neither a confirmed attack nor a cleared alert.

Program evidence describes the history of the program involved. Host evidence describes activity on the machine. No Wazuh evidence omits those recorded tool results; Hydra still has the alert and its code-based checks. Clearing more harmless alerts is only one part of the result: without Wazuh evidence, fewer attacks were confirmed and more were left to a person.

Why the program-evidence result is an upper bound

The injected history was built from the recordings' harmless events. Attack programs therefore appeared new, while harmless programs with history appeared familiar. This separates the two classes by construction. Real environments also have new harmless programs and attacks using familiar programs. We have not yet measured that fairer mix.

The host-evidence run does not have that particular separation. It confirmed 21 of 26 attacks and cleared 208 of 251 harmless alerts, with none dismissed and no false alarms in the sample. It still uses activity from the period around an attack and repetitive lab background, so it is not a production estimate either.

What we ran

• Model configuration: Qwen3.8-27B, FP8 (8-bit), thinking switched off, served with vLLM on one NVIDIA A100 40 GB.
• Investigation: Hydra engine v2, with fixed questions, recorded tool evidence, code-based confidence and guards, and checks on evidence quotes.
• Cases: recorded Windows attack-log cases from seven lab campaigns, covering one attack family. The harmless alerts are background activity from the lab, not customer traffic.
• Runs: one run per evidence setting, on the same 277 cases. No model was rerun to prepare this publication.
• Verification: the published counts were recomputed from the individual saved outcomes. They are final Hydra decisions, not the model's unchecked answers.

The development and test split

The rules and checks were developed on the development half. For the program-evidence run, the benchmark split, recomputed on the full case set, gives the following results. Both halves share the favourable program-history setup, so the test half does not remove that limitation.

What these results do not prove

• Zero observed errors is not a guarantee. There are only 26 attack cases. No attacks dismissed in this sample does not establish a zero dismissal rate in use.
• This is a narrow lab test. It does not establish performance across other attack families, operating systems, cloud alerts or customer environments.
• These are system results. The model, behaviour rules and guards work together. The counts should not be presented as Qwen's standalone accuracy.
• It is not an independent evaluation. The team prepared the cases and labels. External relabelling and adversarial review are still needed.
• Each configuration was run once. We have not measured Qwen's variation across repeated runs here.
• It is not a bank burst test. These counts do not establish production capacity, response-time guarantees or cost per customer.
• It is not the production configuration. Engine v2 remains disabled in production, and the Qwen GPU stays off unless a separate run is requested.

What comes next

We plan a fairer history baseline, independent review of labels and rules, and a separate burst test. Customer work with real data will not start until the Cyber Security Authority answers our licence position in writing.

Download the aggregate results and run-file hashes (JSON) →

Talk to us about the test →

Earlier comparison: 42 alerts, October 2026

These previously published results are retained for comparison. They use 18 attacks and 24 harmless alerts, with a different setup from the newer test. Qwen was not measured on this set. Granite was run before the later fixes, so this is not a controlled ranking of all three models.

The test

• 42 alerts: 18 real attacks and 24 harmless look-alikes.
• 20 from recorded real-world data. These come from public attack recordings (OTRF Security-Datasets, MIT licence). The attack tool's own events are labelled as attacks. The other events from the same recordings are labelled harmless.
• 22 we wrote ourselves from well-known attack and normal patterns: a Word document starting PowerShell, Kerberoasting, password spraying, backup software, patching, VPN logons and so on.
• Titles and severity do not give away the answer. Alerts are titled and rated by the rule that fired, never by the label, and a test checks that severity does not separate attacks from harmless alerts within a rule.
• A new customer's situation. We removed anything a brand-new customer would not have yet: stored reputation lookups, related alerts and analyst history.

How we score it

The results

Every attack Hydra did not flag was passed to a person as "needs a person". None was called harmless.

What the AI alone does without the checks

In this earlier test, the small model on its own dismissed real attacks. The model-only table and guarded system table measure different things. No Gemini result is included in this comparison.

The five attacks it did not flag

We list them because you should know where the checks are weak.

• A memory-dumping tool running from a user's Desktop folder, seen on its own as a program starting. Hydra only noted that it ran from a folder anyone can write to.
• A reconnaissance command (queries about the user, the system or accounts). Hydra noted it as weak evidence.
• A program listening for connections on port 8000.
• Two reads of the Windows Credential Manager vault.

Each of these scored too low for Hydra to call an attack in that earlier run, so each went to a person. These failure cases belong to the 42-alert test, not the newer 277-case test.

Five harmless alerts were also left to a person rather than cleared: two ordinary logons, two process starts and one registry change.

What this test does not show

• We wrote 22 of the 42 cases, knowing which checks existed. That flatters the result.
• Nobody outside the company has relabelled it. The labels are ours.
• We tuned the checks while looking at half of the alerts. We scored the other half once, with the checks frozen: 3 of 6 attacks flagged and 5 of 9 harmless cleared. We then fixed two bugs, so the current figures on that half (4 of 6 and 6 of 9) are no longer a clean test.
• It covers Windows events only. It says nothing about Okta, AWS, Google Cloud or Linux alerts.
• The recordings are small and from 2020. Each has between 68 and 122 events.
• One computer, two small models. Results with other models or hardware will differ.
• It does not test the reputation lookups or analyst history, because a new customer has neither.
• It shows the checks work on these patterns. It does not show how Hydra will do on your alerts.
• It is small. Zero dismissed among 18 attacks does not prove a zero dismissal rate in use.