Who Deleted the File? Practical auditd Rules That Actually Answer It
The config file is gone. You know roughly when. You run: ausearch -f /etc/nginx/nginx.conf -i <no matches> Nothing is broken. auditd is doing exactly what it was told β which was nothing. Audit rules are prospe
The config file is gone. You know roughly when. You run:
ausearch -f /etc/nginx/nginx.conf -i
<no matches>
Nothing is broken. auditd is doing exactly what it was told β which was nothing. Audit rules are prospective: they describe what to record before the thing you care about happens. There is no flag that makes the kernel remember last Tuesday.
That is the first hard truth about Linux auditing, and it is the reason most auditd deployments produce gigabytes of noise and zero answers. This article is about the small, boring rule set that answers questions, and the traps that make well-written rules silently miss.
Three layers, two queues
It helps to separate what people call "audit" into three pieces:
- The kernel audit subsystem collects events selected by rules and puts them in a ring buffer. Its limit is the backlog.
- auditd drains that buffer over netlink, writes records to disk, and hands them to plugins through a second, userspace queue.
- auditctl / ausearch / aureport configure, query, and summarise.
Both queues can overflow, and they report it differently. That distinction explains most "audit is unreliable" complaints.
# Debian/Ubuntu
sudo apt install auditd audispd-plugins
# RHEL/Fedora
# sudo dnf install audit
sudo systemctl enable --now auditd
auditctl -s
The status output is the only health check that matters:
enabled 1
failure 1
pid 812
rate_limit 0
backlog_limit 8192
lost 0
backlog 3
backlog_wait_time 60000
pid 0 means the daemon is not running (the kernel side can still be configured). lost is the count of events the kernel discarded because the backlog was full. Non-zero lost means your audit trail has holes, and no amount of grepping will fill them.
The kernel default backlog is only 64 buffers β -b in auditctl. Distributions usually ship something larger in their default rules; check rather than assume. If you are auditing anything busy, 8192 is a reasonable floor, and --backlog_wait_time (default 60 Γ HZ) controls how long the kernel waits before queueing more events once the limit is reached.
The second queue is auditd's dispatcher queue, q_depth in auditd.conf, default 2000. When it overflows, auditd logs a warning and drops the event according to overflow_action. This is separate from the kernel backlog, and increasing -b will not fix it.
Anatomy of a rule that works
Here is the rule that answers "who deleted this file", in its modern form:
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat,renameat2 -F auid>=1000 -F auid!=unset -k delete
Read it left to right:
-
-a always,exitappends to the exit filter with thealwaysaction.alwaysmeans "write a record";nevermeans "suppress". -
-F arch=b64pins the syscall table. On a bi-arch x86_64 machine, a rule withoutarchapplies to both ABIs, and syscall numbers do not line up between them. -
-Stakes a comma-separated list. One rule with several syscalls is cheaper than several rules, because the kernel compares a bitmask instead of walking a list. -
-F auid>=1000 -F auid!=unsetkeeps interactive users and drops the "no login uid" case. More on that below β it is a trap as much as a filter. -
-k deleteattaches a searchable key, up to 31 bytes. Keys are how you turn a log file into an answer.
The key is not decoration. ausearch -k delete selects every event tagged with that key, regardless of which syscall fired.
Stop writing -w watches
Most tutorials still show this:
-w /etc/passwd -p wa -k identity
It works, but the auditctl man page calls the -w and -p forms deprecated and recommends converting them to syscall rules, citing poor system performance. The equivalent modern rule is:
-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -k identity
path= targets a single inode. For a directory tree you want dir=, not path=:
-a always,exit -F arch=b64 -F dir=/etc/sudoers.d -F perm=wa -k identity
Two details from the documentation that bite people:
-
permneedsarchahead of it. Withoutarch, the kernel ends up evaluating every syscall, which is both slower and broader than you intended. - A directory watch is recursive but stops at mount points. If something is mounted under the watched tree and you need it covered,
-q mount-point,subtreetells the kernel to treat that subtree as part of the watch.
The permissions are access types, not file modes: r read, w write, x execute, a attribute change. The docs note that plain read/write syscalls are deliberately not included, because they would swamp the log; the kernel looks at open flags instead.
A small rule set that answers real questions
Put this in /etc/audit/rules.d/50-locally-useful.rules:
## Bigger kernel backlog. 64 is the kernel default and far too small.
-b 8192
## Identity and privilege files: content and attribute changes.
-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -k identity
-a always,exit -F arch=b64 -F path=/etc/shadow -F perm=wa -k identity
-a always,exit -F arch=b64 -F path=/etc/sudoers -F perm=wa -k identity
-a always,exit -F arch=b64 -F dir=/etc/sudoers.d -F perm=wa -k identity
## Deletions and renames by logged-in users.
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat,renameat2 -F auid>=1000 -F auid!=unset -k delete
## The same operations with no auid filter: catches services and containers.
-a always,exit -F arch=b64 -S unlink,unlinkat,rename,renameat,renameat2 -F dir=/etc/nginx -k nginx_delete
## Permission failures on config reads.
-a always,exit -F arch=b64 -S open,openat,openat2 -F exit=-EACCES -F auid>=1000 -F auid!=unset -k access_denied
-a always,exit -F arch=b64 -S open,openat,openat2 -F exit=-EPERM -F auid>=1000 -F auid!=unset -k access_denied
## Lock the configuration only after everything above is verified.
# -e 2
That is deliberately narrow. Broad rules like "audit every openat" produce a log nobody reads and a measurable performance cost.
Notice the duplicate pair for deletions: one filtered by auid, one scoped by path with no auid filter. That is not redundancy, it is coverage. A systemd unit that deletes a file has no login uid, so the first rule will never match it.
The auid trap
auid is the login uid, set by PAM (pam_loginuid) when a session starts and preserved across su and sudo. That is why a record can show auid=1000 uid=0: the human is ali, the process is root. It is the single most useful field in the log.
It is also the field that silently breaks rules. When there is no login session β a systemd service, a cron job, many containers β auid is unset and prints as 4294967295. The audit rules documentation notes that -1, 4294967295, and unset are all the same value, and recommends writing -F auid>=1000 -F auid!=unset for "real users" rules.
So the rule is correct, and still misses the deletion you are investigating. Scope by path=/dir= when you do not know who the actor will be.
Load rules without restarting the daemon
Rules in rules.d are component files. augenrules concatenates them in natural sort order, strips comments and blank lines, and writes /etc/audit/audit.rules. It also hoists control directives: the last -D becomes line one, the last -b line two, the last -f line three, and the last -e becomes the final line. That is why -e 2 belongs at the end of the file conceptually, even if you write it in the middle.
sudo augenrules --check # would the compiled file change?
sudo augenrules --load # compile and load into the kernel
auditctl -l | grep -E 'delete|identity'
At boot, the audit-rules.service unit runs exactly augenrules --load; auditd itself Wants= it.
Do not restart auditd to apply rules. Its upstream unit sets RefuseManualStop=yes, so systemctl restart auditd is refused by design, and a restart is the wrong tool anyway: auditd loads rules at boot, but augenrules --load updates the kernel directly.
For auditd.conf changes, reload instead:
sudo auditctl --signal reload
That sends SIGHUP; auditd re-reads its configuration and logs a DAEMON_CONFIG event if it succeeded.
A live test rule, added and removed without touching files:
sudo auditctl -a always,exit -F arch=b64 -F path=/etc/hosts -F perm=wa -k hosts_test
sudo touch /etc/hosts
sudo ausearch -k hosts_test -i --start recent
sudo auditctl -d always,exit -F arch=b64 -F path=/etc/hosts -F perm=wa -k hosts_test
Deletion requires an exact match of syscalls and every field β a typo leaves the rule loaded, so verify with auditctl -l.
Reading a record without guessing
A deletion looks like this (values vary; the structure does not):
type=SYSCALL msg=audit(1795151520.412:8821): arch=c000003e syscall=263 success=yes exit=0 a0=ffffff9c a1=55d3c0e2a1b0 a2=0 a3=0 items=1 ppid=1841 pid=1902 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4 comm="rm" exe="/usr/bin/rm" key="delete"
type=CWD msg=audit(1795151520.412:8821): cwd="/home/ali"
type=PATH msg=audit(1795151520.412:8821): item=0 name="/etc/nginx/nginx.conf" inode=131842 dev=08:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=DELETE
-
msg=audit(epoch.millis:serial)β the event id. Every record from one syscall shares it.ausearch -a 8821retrieves the whole event. -
arch=c000003eis x86_64;syscall=263isunlinkaton that table. -
auid=1000 uid=0β ali logged in, the process ran as root. -
commis the short task name,exethe binary path,keythe rule that fired. -
nametype=DELETEon the PATH record is the actual file.
Then query by meaning instead of by offset:
ausearch -k delete -i --start this-week
ausearch -k delete -i -f /etc/nginx/nginx.conf
ausearch -k identity -i --start today
aureport --key --summary --start this-week
aureport --failed --summary --start today
ausearch -k delete --raw | aureport --user --summary
-i interprets uids, syscall numbers, and addresses. One documented caveat: for unenriched logs it resolves names using the current machine's accounts, so a renamed or deleted user can make old events look like someone else. If logs travel between machines, log_format = ENRICHED in auditd.conf stores the resolved names at write time.
Four more traps
io_uring bypasses syscall rules. Operations submitted through io_uring do not take the normal per-operation syscall path, so a rule on openat may never see them. audit has a dedicated filter for this:
-a always,io_uring -S openat,openat2 -F key=access
The io_uring filter takes no arch field; it is implicit. If your workload uses io_uring, syscall-only auditing gives you a false sense of coverage.
auditd rotates its own logs. max_log_file is in MiB, num_logs is the count kept, and max_log_file_action decides what happens at the limit β rotate, keep_logs, syslog, exec, suspend, or ignore. SIGUSR1 forces an immediate rotation. Pointing logrotate at audit.log fights the daemon; use the daemon's own knobs. If logging suspends after a queue overflow or disk event, SIGUSR2 resumes it.
-e 2 is a one-way door until reboot. It locks the configuration; further changes are audited and denied, and only a reboot clears it. Test every rule first, confirm with auditctl -l, then enable it.
Audit is not tamper-proof by default. Root can turn the kernel side off with auditctl -e 0, kill the daemon, or edit the file. -e 2 plus shipping records off-box is the minimum for anything that has to stand up later.
The uncomfortable summary
If the rule was not loaded before the event, the event does not exist. Everything else β backlog sizes, keys, -i β is about making sure that when it is loaded, you can find it and trust it.
Start with three rules: identity file changes, deletions scoped to the directories that matter, and failed permission checks. Give each a key. Check auditctl -s for lost. Then leave it alone until you need it, which is the whole point.
Sources
- auditctl(8) β https://man7.org/linux/man-pages/man8/auditctl.8.html
- audit.rules(7) β https://man7.org/linux/man-pages/man7/audit.rules.7.html
- ausearch(8) β https://man7.org/linux/man-pages/man8/ausearch.8.html
- aureport(8) β https://man7.org/linux/man-pages/man8/aureport.8.html
- auditd.conf(5) β https://man7.org/linux/man-pages/man5/auditd.conf.5.html
- auditd(8) β https://man7.org/linux/man-pages/man8/auditd.8.html
- augenrules(8) β https://man7.org/linux/man-pages/man8/augenrules.8.html
- audit userspace source and units β https://github.com/linux-audit/audit-userspace
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.