When events vanish without an error

The hardest event problems rarely throw red banners. A filter discards a malformed payload. A retry budget expires. A consumer acknowledges a message it never fully applied. From the outside, volume looks almost healthy.

Count at both ends of the path

If you only measure publish rate, silent loss at the consumer stays invisible. Pair producer counts with consumer applied counts for the same event type and the same time window. Even a coarse daily reconciliation catches gaps that live charts miss.

Treat schema drift as an anomaly precursor

Quiet drops often follow a field rename or a newly required attribute. When reviewing streaming application event flow, we ask when schemas last changed and who was notified. Teams that ship schema changes without a paired observation check invite silent failure.

Document the intentional discards

Some filters are supposed to drop traffic — test traffic, duplicate IDs, expired sessions. Write those rules where operators can find them. Otherwise every intentional discard looks like an incident waiting to be misdiagnosed.

If your last three customer complaints involved missing updates despite “green” dashboards, a quiet-drop review is usually more valuable than another threshold tweak.