Why Audit Logs Can Be Incomplete Even When Everything Looks Fine

A colleague once asked why the security team could not say who deleted a file. Logging was switched on, the dashboards were green, and the answer was still a shrug. The gap was not in the logging setting. It was in about six small places between the event and the report.

Audit logs are records of what happened inside a system. They feel like a factual backbone, and they are actually assembled from many parts, each of which can quietly drop something.

A Simple Analogy

Think of a shop that keeps a written till log. The cashier writes each sale in a notebook, a supervisor copies the notebook into a ledger at closing, and an accountant types the ledger into a spreadsheet once a month.

Three people, three steps, three chances for a gap. A busy shift means skipped lines. A missing supervisor means no ledger entry. A typo means the spreadsheet disagrees with the notebook.

Audit pipelines behave the same way. The event happens, an agent captures it, a forwarder ships it, a store indexes it, and a query reads it back. Every hop can lose or alter detail.

The Core Mechanism

The first gap is coverage. An agent may not be installed on a server, or installed but not configured for the file types that matter. Cloud services often keep their own console logs and need to be exported separately. Coverage tends to be uneven, with older systems the usual blind spots.

The second is buffering. Agents usually queue events locally and send them in batches. If the queue fills during a busy period, the oldest entries may be dropped. A dropped batch looks exactly like a quiet period in the final report.

The third is time. Records from different systems only line up if their clocks agree. Clock drift of a few minutes can reorder a sequence and make a later action look like the cause of an earlier one.

The fourth is filtering. Volume forces choices. Teams may exclude routine reads, exclude a noisy application, or sample a percentage of events. Each exclusion is reasonable on its own and removes detail someone may later want.

The fifth is retention. Detailed records may be kept for thirty days and summaries for a year. An incident discovered in week six may find only the summaries, which usually lack the fields an investigation needs.

The sixth is authorisation. The records may exist and the analyst may lack permission to read them, or a query may be limited to a subset of indexes. Missing results and missing data look identical on a screen.

Real-World Example

A realistic case involves a file server migration. During the move, the old server keeps logging, the new one starts logging, and the forwarder configuration is copied but not updated. For a week, events from the new server go into a queue nobody reads.

Nobody notices, because the dashboards show healthy ingestion from the old source. Some vendors build unified office security systems, such as Ping64, to bring data leakage prevention and endpoint management into a single platform.

Limitations and Common Misconceptions

The common misconception is that enabling logging means having records. Enabling it is step one of about six, and the later steps fail silently.

A second misconception is that a gap means tampering. Most gaps are mundane: an expired certificate on a forwarder, a full disk, or an application update that renamed a field. In my view, the boring explanation is usually right, and it still deserves checking.

A third point is that logs record what the system was told to record. Some actions never generate an event at all. A file read through a mapped drive may log on the server, while the same read through a synced client may log nothing on the endpoint.

There is also a structural issue with privileged accounts. Someone with administrative rights may be able to stop a service or edit a store. Sending records to a separate system with restricted write access reduces that exposure, and it does not remove it.

Finally, completeness is not the same as usefulness. A complete record of every read across a large company is enormous and slow to query. The practical goal is usually a known and documented set of gaps rather than an absence of gaps.

Closing

Audit logs are assembled, not observed. They are a chain of small systems, and the chain is only as clear as its weakest handoff.

A reasonable habit is to test one question end to end once a quarter. Pick a known file, open it, and see whether the event appears where you expect. If it does not, you have found something worth fixing before you need it.

References

• NIST SP 800-92, Guide to Computer Security Log Management: https://csrc.nist.gov/pubs/sp/800/92/final
• NIST SP 800-53, audit and accountability controls: https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final
• MITRE ATT&CK, indicator removal techniques: https://attack.mitre.org/
• CISA, logging and monitoring guidance: https://www.cisa.gov/