Starting in Oracle Solaris 11.4.87 the auditd plugin audit_binfile, which is responsible for writing to the local binary format audit files, now marks the active audit file as append_only when it is first created, and will remove all write permissions when it is closed and renamed from <start_time>.not_terminated.<hostname> to <start_time>.<end_time>.<hostname>. This means we can now use ZFS File Retention to provide additional protection for the audit trail files.
The ability for files stored on ZFS to be protected by marking as “retained” was added in Oracle Solaris 11.4.45. A retained file is one that can not be edited, renamed or deleted; even by a user with all privileges such as root when using the mandatory retention policy. One of the main use cases for this is critical system backups and anything containing records that need to be retained for legal/compliance reasons. The File Retention functionality is also available on the Oracle ZFS Storage Appliance, and has been independent assessed for suitability in environments requiring the controls of SEC17a-4(f), and similar, but this article is about a local Oracle Solaris specific use case.
A file is marked as retained by removing all of its write permissions, or setting the readonly attribute, this works locally as well as over NFS or SMB. Once retained that file is then not able to be deleted, modified or renamed until the current time plus the value of retention.period.default. If the access time (atime) is set into the future before the write permissions are removed then that is used as the retention time instead (providing it is between retention.period.min and retention.period.max). Full documentation for the functionality is in the “Retaining Files” section of the ZFS documentation.
A change to auditd was made so that if ZFS File Retention is enabled the closed audit trial will automatically become retained. The rest of this article demonstrates an example audit trail retention scenario where we wish to keep at least the last 30 days of audit information locally, as well as ensuring its deletion.
ZFS file retention must be enabled at the time the filesystem is created. So to enable retention for the audit trail we need to (re)create the ZFS filesystem that will be mounted as /var/share/audit (or an alternate location if audit_binfile p_dir has been set elsewhere). Before creating temporarily disable auditing by running # audit -t. If necessary empty the content of the existing /var/share/audit.
Now lets create the new audit filesystem with retention enabled:
# zfs create retention.policy=mandatory rpool/VARSHARE/audit
# zfs set retention.period.default=30d rpool/VARSHARE/audit
With the above settings the audit files generated by audit_binfile will be retained for 30 days. Note that if ‘privileged‘ retention policy is used the root user can still delete the audit files. So in this case only ‘mandatory‘ will be effective in protecting the audit trail from the local root user, or any user with sufficient privilege via RBAC profiles. It recommended to also set a quota and enable compression for a ZFS filesystem used for audit data files.
We should also control when audit_binfile switches to new files. Since we have a retention policy expressed in days it makes sense to configure audit_binfile to perform audit file rotation daily:
# auditconfig -setplugin audit_binfile p_age=1d
With the above policies in place the system will rotate the active audit file on a daily basis and ZFS will ensure it is protected from modification, deletion or rename for the following 30 days.
However the settings so far assume that we will always have a clean shutdown, so that auditd/audit_binfile has a chance to close out the active audit trail and mark, thus making it retained. However sometimes systems don’t get shutdown cleanly, due to various unplanned events. To ensure that any files that were active during an unclean reset of the system are also retained we can instruct ZFS to retain files that may still be marked as writable but haven’t been written to for a period of time using the grace period:
# zfs set retention.period.grace=1445minutes rpool/VARSHARE/audit
In the above example I’ve used 1,445 minutes rather than 1 day because we need to give audit_binfile a little bit of time to switch over the active trail and perform the rename during normal operation. Five minutes is more than we should need, but one minute felt a little to tight if the system is very busy. Further details on configuring the audit subsystem are available from the “Managing Auditing” section in the Oracle Solaris documentation set.
Automatically deleting older audit files:
We can also use the ZFS File Retention to automate the removal of old audit trail files by instructing ZFS on what to do when a retained file reaches the expiry time.
# zfs set retention.policy.onexpiry=delete rpool/VARSHARE/audit
The above setting instructs ZFS to delete the retained files when they reach their expiry date.
We can also provide a little “grace” period post retention before ZFS deletes them by setting, for example if we wanted to be able to manually remove them when they expire but have ZFS automatically do that 3 days after expiry we can add a delete grace period:
# zfs set retention.period.deletegrace=3d rpool/VARSHARE/audit
ZFS will periodically check if any expired retained files are ready for deletion, and will actively start that deletion. ZFS also removes them from the directory listing on any attempted access of the directory they are contained in. So a automatically deleted retained file will not be seen, but could still be taking up the disk space for a short while after deletion. For this reason and for easy of audit trail analysis it is recommended to also set the p_size property for audit_binfile to keep the files a reasonable size, what that size is depends a lot on how often the system generates audit records which is heavily influenced by the rest of the configured audit policy.
If we only wanted the automated deletion of the old local audit trail files and don’t need the modification/deletion prevention (maybe because the system also has audit_syslog or audit_remote configured) then use of the ‘privileged’ retention policy would be suitable. That would allow a privileged user, eg root, to manually delete audit trail files before the retention expiry and ZFS would ensure they were auto deleted at 30 days (in this use case I wouldn’t set the deletegrace property).
While the above has focused on the Oracle Solaris Audit trail files a similar concept could be applied to locally stored log files managed by syslog and logadm.
Happy logging!