In September 2026 the automated triage pipeline in our detection engineering lab reported "malicious activity confirmed" on a Windows 11 workstation, paged the on-call engineer and added four external IP addresses to the perimeter firewall's block table. Nothing was compromised. The "trojan downloader" traffic was Windows Delivery Optimization fetching Microsoft Defender definition updates from Microsoft Connected Cache nodes.
The wrong verdict had a second, quieter cause: four weeks earlier the lab's SIEM indexer had stopped accepting new events, and nothing in the stack reported it. This write-up covers both findings, how to verify this class of alert in a few minutes, and the design rules we took for AI-assisted triage.
Key findings
- Microsoft Connected Cache traffic matches a common hunting signature. The Emerging Threats rule ET HUNTING SUSPICIOUS Dotted Quad Host MZ Response (SID 2021076) fires whenever Delivery Optimization pulls an executable from a cache node by IP address. Seen through an alert and a whois lookup alone, the cache node looks like anonymous infrastructure serving executables.
- A single-node SIEM indexer can stop indexing with every surface check green. Unassignable replica shards still count against the default limit of 1,000 shards per node. Once the limit was reached, every new daily index was refused, while agents stayed connected, alerts kept reaching case management and the shipper's own connectivity test still passed.
- Automated triage turned missing evidence into a confident verdict. With only the IDS alert text and whois data, and no host telemetry because of finding 2, the triage agent classified routine updates as a dropper and acted on it without a human checkpoint.
The lab
Our detection engineering lab is where we build and test detections, AI-assisted investigation and security automation before they reach a client environment. It runs Suricata on the perimeter firewall's internal and external interfaces, a Wazuh SIEM across dozens of endpoints with Sysmon on the Windows hosts, and an automated triage pipeline: every hour it groups open alerts by source address and hands each group to an LLM-based investigation agent. The agent could read the alert data and query whois and the IDS alert log, and at the time it was allowed to add addresses to the perimeter firewall's block table.
Timeline
| Date (2026) | Event |
|---|---|
| 8 Aug | A probe we run on Windows hosts starts writing PowerShell policy-test scripts as SYSTEM, which raises about 980 Sysmon-based alerts a day on the workstation. |
| 10 Aug | First SID 2021076 alert for the workstation. The first triage run is inconclusive. |
| 11 Aug | The SIEM indexer reaches 1,000 of 1,000 shards. From this point every new daily index is refused and no new events are indexed. |
| 11 Aug – 6 Sep | The workstation is re-investigated every day because of the probe noise. Verdicts alternate between "no issues", "unknown" and "confirmed". Each run notes that the SIEM search returns nothing. |
| 7 Sep | The agent classifies the 2021076 hits as "dropper / trojan beaconing", blocks four cache-node addresses and pages, on a verdict it rates LOW and closes in the same run. |
| 8 Sep | It repeats, re-adding two addresses whose blocks had aged out. Manual investigation finds both root causes and applies the fixes the same morning. |
Finding 1: Connected Cache traffic and SID 2021076
SID 2021076 is an informational hunting rule. It matches an HTTP response that carries a Windows executable (an MZ header) for a request whose Host is a bare IPv4 address. That pattern is common in malware staging, and it is also exactly how Delivery Optimization talks to a Microsoft Connected Cache node.
Connected Cache nodes are deployed inside internet service providers' networks, so they sit on the ISP's address space. A registry lookup returns the ISP or a generic customer allocation, and there is no Microsoft reverse DNS, by design. To a reviewer, or a model, holding only the alert and whois, the picture is "an anonymous server delivering executables over plain HTTP".
Three independent sources in our lab showed it was Windows Update:
- The IDS HTTP log. Every matching request carried the user agent
Microsoft-Delivery-Optimization/10.1and acacheHostOrigin=parameter naming a Microsoft origin, for example/d/msdownload/update/software/defu/…/am_delta_patch_<version>.exe?cacheHostOrigin=…download.windowsupdate.comand/filestreamingservice/files/<guid>?…&cacheHostOrigin=…tlu.dl.delivery.mp.microsoft.com. - Host telemetry. Sysmon process-creation events (event ID 1) on the workstation showed
wuaucltcore.exerunning the matchingAM_Delta_Patch_<version>.exefromC:\Windows\SoftwareDistribution\Download\Install\about two seconds after each alert, followed byMpSigStub.exe. - Windows' own record.
Get-DeliveryOptimizationLogon the workstation contained the response from Microsoft's Delivery Optimization service that assigned the cache host, with"CacheHost":"<the alerted address>".
Verifying this alert in a few minutes
- Search the IDS HTTP log for the destination address. Look for the
Microsoft-Delivery-Optimizationuser agent and acacheHostOrigin=parameter pointing at a Microsoft domain. - On the endpoint, look for
wuaucltcore.exeorMpSigStub.exeprocess creation within a few minutes of the alert. - Run
Get-DeliveryOptimizationLog | Where-Object Message -match '<address>'to confirm Microsoft assigned that cache host.
A user agent alone is easy to spoof, so step 1 is a filter, not a verdict; steps 2 and 3 are what make it conclusive. One trap: if your Sysmon configuration filters network connections, the absence of a network event (event ID 3) for the address proves nothing.
What we changed
The rule fires on behaviour that is routine on every Windows network that uses Delivery Optimization, and it is alert-only, so we suppressed it at the IDS and documented the Connected Cache context for the triage pipeline. If you prefer to keep it, enrich it instead: route a 2021076 hit to a human only when the HTTP record lacks the Delivery Optimization user agent or the Microsoft cacheHostOrigin, and have host telemetry decide the rest. Blocking the addresses does little damage, because Delivery Optimization falls back to Microsoft's CDN, but it is still a wrong block that recurs.
Finding 2: a SIEM indexer that stopped indexing without telling anyone
The lab's Wazuh indexer is a single node. Its default cluster.max_shards_per_node is 1,000, and two details made that limit a cliff:
- Unassignable replicas still count. One family of audit-log indices had been created with a replica. On a single node a replica can never be assigned, but it still occupies a slot: at the failure point the node held 797 started primaries and 203 unassigned replicas, exactly 1,000.
- Nothing aged out. There was no retention policy, so 453 daily indices had accumulated over eight months.
When the next day's index needed three more shards, the indexer refused to create it. From then on every bulk request from the shipper (Filebeat) failed with validation_exception: this action would add [3] total shards, but this cluster currently has [1000]/[1000] maximum shards open, more than 3,300 times before anyone looked.
Nothing looked broken. The agents were connected. The Wazuh manager kept raising alerts and forwarding them to case management through its own integration, which does not pass through the indexer. filebeat test output reported success, because it only tests the connection, not whether writes are accepted. Disk usage sat at 15%. The only symptom was that every SIEM search returned zero results, and the triage agent recorded that as a known gap and carried on without the data.
The fix
- Set replicas to 0 on the only indices that had them, and added an index template so new indices of that family are created with one shard and no replica.
- Attached retention policies: alerts 180 days, audit logs 30 days, monitoring and statistics 90 days. The policy engine's own system indices also needed replicas set to 0; on the Wazuh indexer that change is refused for the regular admin user and has to be made with the admin client certificate.
Indexing resumed within seconds. Once the policies ran, the node went from 458 indices and 806 shards to 183 indices and 467 shards, 47% of its budget, with a green cluster and no unassigned shards. The four weeks of events were not lost: the manager keeps compressed copies of every alert, so they could be re-ingested.
The monitoring we added
The lesson is to monitor the newest indexed event, not the pipeline's components. A small exporter now publishes indexer health as metrics, with these alerts:
| Condition | Alert after |
|---|---|
| Newest indexed alert older than 30 minutes (critical at 2 hours) while the indexer is reachable | 5 minutes |
| Shards above 85% of the per-node limit (critical at 97%) | 30 minutes (10 minutes) |
| Any unassigned shard | 6 hours |
| Cluster status red | 10 minutes |
| Indexer unreachable, or the health probe itself missing | 10 minutes (20 minutes) |
Finding 3: how automated triage reached "confirmed"
The investigation agent did what it was asked with the evidence it had, and that evidence was thin:
- No host telemetry. With the indexer down, every investigation of the workstation for four weeks ran without endpoint data.
- No protocol detail. The HTTP records that settled the question in minutes were not among the agent's tools; it saw the alert signature, the addresses and whois.
- A host it could not stop looking at. Our own policy-test probe and Windows' hourly compatibility-database merge generated enough Sysmon alerts to put the workstation back in the daily review queue every day, so the same thin evidence was re-judged daily until a run read it the wrong way.
- Actions not tied to the verdict. The run that blocked the addresses rated its own finding LOW and closed it, but the page and the block did not depend on that rating.
The impact was small. The pattern is the risk: a confident verdict, an automatic action and a page, all resting on two weak signals.
The rules we took for AI-assisted triage
- Evidence before verdict. An agent judging a host needs protocol-level logs and host telemetry for it. If a source is missing, the verdict is "insufficient evidence", never "confirmed".
- Signature class matters. Hunting and informational signatures can start an investigation; they never justify containment on their own.
- Notifications follow the verdict. Paging uses the same severity threshold as escalation.
- Containment needs a checkpoint. Blocking requires a high-severity verdict on a non-hunting signature and a human approval step.
- Fix your own noise first. Our probe's artifacts are now suppressed by narrow local rules, so the hosts we test stop cycling through review.
- Monitor the monitor. Finding 2 is what starved the agent.
Rules 3, 5 and 6 were in place within hours of the investigation. Rules 1, 2 and 4 change how the triage agent itself gathers evidence and acts, and they are the remaining work in our pipeline.
Why this matters beyond our lab
Each of these failure modes exists on every platform we work with, whether that is Palo Alto Networks Cortex XSIAM, CrowdStrike Falcon or an open-source stack. Hunting content produces alerts that read like conclusions. Ingestion can stall while every connector reports healthy, so the only honest health signal is the age of the newest event per source. And AI-assisted triage and SOAR automation (Tines or anything else) inherit the gaps in whatever evidence they are given. Our default for every engagement follows from that: detections are measured against real telemetry, ingestion freshness is monitored per source, and automation that can contain has a human checkpoint where judgment belongs.