Rsyslog is an open-source utility, developed as a client/server architecture service and can achieve both roles independently. It can run as a server and gather all logs transmitted by other devices over the network or it can run as a client by sending all internal system events logged to a remote Syslog server. Rsyslog is the most popular logging mechanism in a huge number of Linux distributions and It’s also the default logging service in Oracle Linux.

Rsyslog daemon can be configured to run as a server in order collect log messages from multiple systems. These systems act as clients and are configured to transmit their logs to a rsyslog server.

An Rsyslog client always sends the log messages in plain text, if not specified otherwise; by default there’s no encryption or anything applied while the message is in transit, and that might not be acceptable due to security policies. If your organization needs a higher level of security, you need to set up secure logging to remote log server. Secured remote logging leverages TLS.

This document describes a secure way to set up secure rsyslog, with TLS certificates, to transfer logs to remote server. A secure logging environment requires more than just encrypting the transmission channel. Below are some of the security benefits with secure remote logging using TLS:

  • syslog messages are encrypted
  • the syslog sender authenticates to the syslog receiver
  • the syslog receiver authenticates to the syslog sender
  • the mutual authentication prevents man-in-the-middle attacks

Demo Setup

Two different Oracle Linux 7 Update 9 nodes (the same procedure can be applied to Oracle Linux 8) to demonstrate secure logging to remote log user using rsyslog with TLS certificates:

  • ol7client, the machine acting as log forwarder to the server
  • ol7server, the collector machine, the remote log server

Both the nodes are installed with Oracle Linux 7 Update 9. In this article ol7client will act as a client which will forward the rsyslog messages to ol7server (remote log server).

Software installation

  • Install required RPM packages on client machine (ol7client):
[root@ol7client ~]# yum -y install gnutls-utils rsyslog audit gnutls rsyslog-gnutls
  • Install required RPM packages on server machine (ol7server):
[root@ol7server ~]# yum -y install gnutls-utils rsyslog audit gnutls rsyslog-gnutls

Generate certificates, required for encrypted TCP/TLS communication

The first(s) operations need to be executed on client machine:

  • Create dedicated folder for certificates
[root@ol7client ~]# mkdir /root/certificates
  • Generate the private key
[root@ol7client certificate]# certtool --generate-privkey --outfile ca-key.pem

Generating a 2048 bit RSA private key...
  • Now create the (self-signed) CA certificate. This command does different queries and you should have to use appropriate responses. One of them is to specify that the certificates belongs to an authority. The certificate is also used to sign other certificates. In any case, please check the highlighted answers (bold).
[root@ol7client certificate]# certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem
Generating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Common name: ol7client
UID:
Organizational unit name:
Organization name:
Locality name:
State or province name:
Country name (2 chars):
Enter the subject's domain component (DC):
This field should not be used in new certificates.
E-mail:
Enter the certificate's serial number in decimal (default: 7016453781409621598):
Activation/Expiration time.
The certificate will expire in (days): 3650

Extensions.
Does the certificate belong to an authority? (y/N): y
Path length constraint (decimal, -1 for no constraint): -1
Is this a TLS web client certificate? (y/N): n
Will the certificate be used for IPsec IKE operations? (y/N): n
Is this a TLS web server certificate? (y/N): n
Enter a dnsName of the subject of the certificate: ol7client
Enter a dnsName of the subject of the certificate:
Enter a URI of the subject of the certificate:
Enter the IP address of the subject of the certificate:
Enter the e-mail of the subject of the certificate:
Will the certificate be used to sign OCSP requests? (y/N): n
Will the certificate be used to sign code? (y/N): n
Will the certificate be used for time stamping? (y/N): n
Will the certificate be used to sign other certificates? (y/N): y
Will the certificate be used to sign CRLs? (y/N): y
Enter the URI of the CRL distribution point:
X.509 Certificate Information:
         Version: 3
         Serial Number (hex): 615f738a17dde65e
         Validity:
                  Not Before: Thu Oct 07 22:24:11 UTC 2021
                  Not After: Sun Oct 05 22:24:15 UTC 2031
         Subject: CN=ol7client
         Subject Public Key Algorithm: RSA
         Algorithm Security Level: Medium (2048 bits)
                  Modulus (bits 2048):
                           00:c2:ba:e8:1c:5f:a6:00:c4:e1:20:15:30:24:d7:8b
                           85:2e:cf:e7:ce:c5:39:7e:61:72:9d:7e:1f:8a:c0:cc
                           59:e5:8a:e5:71:12:54:c8:a5:a6:78:30:db:64:c3:e6
                           33:a5:ef:f1:fb:b5:bb:e3:e5:3d:d2:79:06:73:52:13
                           30:47:8e:4a:b5:7a:b5:0d:fe:b3:96:17:a5:02:04:cd
                           d1:2e:ee:af:77:69:ac:f4:a8:96:fd:9b:d6:31:20:7d
                           1c:9a:14:fe:c4:13:dd:07:fe:88:93:b8:56:b5:1a:a0
                           2e:6d:00:4a:d9:ef:85:33:ea:08:b5:a9:30:34:d1:22
                           63:a8:a0:5c:03:20:6c:d2:54:be:e9:0d:d1:3d:f6:6c
                           27:43:1b:55:a8:68:08:b8:49:cf:6d:89:f5:56:f0:2b
                           99:bd:41:bd:15:5a:d1:a7:d0:0b:de:34:db:92:58:76
                           6b:6e:d5:93:19:f2:ac:ef:21:fd:61:ca:5e:73:75:4c
                           b1:cb:82:a7:9d:fb:40:3b:87:e7:80:04:57:68:88:9f
                           38:70:99:a2:34:d5:0d:59:92:17:2f:d4:91:db:26:c3
                           b7:ca:44:71:7d:b9:c0:e3:f8:08:16:79:16:b4:e4:26
                           91:31:43:4f:bc:c9:77:9e:8f:3e:38:98:1c:bb:f9:86
                           69
                  Exponent (bits 24):
                           01:00:01
         Extensions:
                  Basic Constraints (critical):
                           Certificate Authority (CA): TRUE
                  Subject Alternative Name (not critical):
                           DNSname: ol7client
                  Key Usage (critical):
                           Certificate signing.
                           CRL signing.
                  Subject Key Identifier (not critical):
                           fa449b4f6fb13892a9ac079d138d3ff43ca3cc30
Other Information:
         Public Key ID:
                  fa449b4f6fb13892a9ac079d138d3ff43ca3cc30
         Public key's random art:
                  +--[ RSA 2048]----+
                  |                 |
                  |                 |
                  |       o         |
                  |      o o        |
                  |     . =So       |
                  |    . Eooo= .    |
                  |     ..*+=.+ o   |
                  |     ..oBoo.o    |
                  |    .oo....o.    |
                  +-----------------+

Is the above information ok? (y/N): y

Signing certificate...
[root@ol7client certificate]#
NOTE: ca-key.pem is a private key of certificate authority and ca.pem is a public key that we are going to distribute to the other nodes.
  • In this step, we generate certificates for both the server and the client machines.
[root@ol7client certificate]# certtool --generate-privkey --outfile ol7server-key.pem --bits 2048
** Note: Please use the --sec-param instead of --bits
Generating a 2048 bit RSA private key...
[root@ol7client certificate]#

NOTE: The remote log server still is ol7server, and the signing requests is what it needs to get the certificate signed. So just the fact of having private key is not enough. It must be signed by a certificate authority. Here we are raising a request using certtool to load ol7server-key.pem private key and sign this private key into outfile i.e. ol7server-request.pem.

  • This step will prompt, again, with many different questions, answer them appropriately based on your environment while looking at highlighted answers below.
[root@ol7client certificate]# certtool --generate-request --load-privkey ol7server-key.pem --outfile ol7server-request.pem
Generating a PKCS #10 certificate request...
Common name: ol7server
Organizational unit name:
Organization name:
Locality name:
State or province name:
Country name (2 chars):
Enter the subject's domain component (DC):
UID:
Enter a dnsName of the subject of the certificate: ol7server
Enter a dnsName of the subject of the certificate:
Enter a URI of the subject of the certificate:
Enter the IP address of the subject of the certificate:
Enter the e-mail of the subject of the certificate:
Enter a challenge password:
Does the certificate belong to an authority? (y/N):
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/n): n
Will the certificate be used for encryption (RSA ciphersuites)? (Y/n): n
Will the certificate be used to sign code? (y/N): n
Will the certificate be used for time stamping? (y/N): n
Will the certificate be used for IPsec IKE operations? (y/N): n
Will the certificate be used to sign OCSP requests? (y/N): n
Is this a TLS web client certificate? (y/N): n
Is this a TLS web server certificate? (y/N): n
[root@ol7client certificate]#
  • Next step shows how to sign the certificates, that is going to be used by ol7server, by the private key of the certificate authority, and that is what is going to make sure that ol7server is going to be trusted by all the client machines. Take a look to the highlighted answers and act for your own environment.
[root@ol7client certificate]# certtool --generate-certificate --load-request ol7server-request.pem --outfile ol7server-cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem
Generating a signed certificate...
Enter the certificate's serial number in decimal (default: 7016457513872759965):
Activation/Expiration time.
The certificate will expire in (days): 3650
Extensions.
Do you want to honour the extensions from the request? (y/N):
Does the certificate belong to an authority? (y/N):
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N):
Is this a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: ol7server
Enter a dnsName of the subject of the certificate:
Enter a URI of the subject of the certificate:
Enter the IP address of the subject of the certificate:
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/n): n
Will the certificate be used for encryption (RSA ciphersuites)? (Y/n): n
Will the certificate be used to sign OCSP requests? (y/N): n
Will the certificate be used to sign code? (y/N): n
Will the certificate be used for time stamping? (y/N): n
X.509 Certificate Information:
         Version: 3
         Serial Number (hex): 615f76ef20019c9d
         Validity:
                  Not Before: Thu Oct 07 22:38:41 UTC 2021
                  Not After: Sun Oct 05 22:38:45 UTC 2031
         Subject: CN=ol7server
         Subject Public Key Algorithm: RSA
         Algorithm Security Level: Medium (2048 bits)
                  Modulus (bits 2048):
                           00:c4:d1:02:6a:57:84:23:62:dc:75:95:ba:13:ae:c5
                           e0:1f:0e:ec:49:6c:92:be:a7:82:05:1a:8d:f2:81:32
                           c3:8e:ee:57:c3:44:9c:43:95:89:51:aa:b1:62:39:bf
                           2a:99:8b:a3:66:96:61:86:1e:a9:0c:93:e6:7a:2c:70
                           1d:1e:bb:f2:81:42:b2:97:20:ca:1e:31:e5:91:64:42
                           47:f8:9e:5e:52:36:c5:2f:ab:8e:c6:33:17:53:a1:3a
                           28:85:c9:a7:ed:f8:5c:16:e8:ca:41:55:dd:e6:92:d5
                           f3:cd:05:35:04:9f:40:ad:3b:ae:53:ba:b3:bc:fb:60
                           b4:9c:8e:9d:59:74:48:8e:1d:2e:55:72:f4:86:b8:d0
                           4a:50:d6:21:d3:ae:d4:5e:c8:00:ab:72:6e:bc:56:a7
                           fb:57:ba:6b:0b:5b:a2:20:59:f4:2e:93:9e:77:72:dc
                           c0:41:4c:e9:69:d6:bd:5e:f5:b8:b8:68:03:16:ba:1e
                           f4:bf:7c:9c:e0:63:92:b0:07:5a:0f:e2:23:6d:2d:d6
                           83:4a:ef:53:d0:8e:96:62:17:1d:4e:29:82:0c:0b:75
                           e2:1b:08:99:62:e1:ac:6c:7f:db:f1:15:58:bc:97:e5
                           16:28:6e:62:69:7a:44:db:39:be:98:0e:bd:d7:10:77
                           7f
                  Exponent (bits 24):
                           01:00:01
         Extensions:
                  Basic Constraints (critical):
                           Certificate Authority (CA): FALSE
                  Key Purpose (not critical):
                           TLS WWW Client.
                           TLS WWW Server.
                  Subject Alternative Name (not critical):
                           DNSname: ol7server
                  Subject Key Identifier (not critical):
                           d7e66eb45b583d950af42a64a3ace7aba2e12d89
                  Authority Key Identifier (not critical):
                           fa449b4f6fb13892a9ac079d138d3ff43ca3cc30
Other Information:
         Public Key ID:
                  d7e66eb45b583d950af42a64a3ace7aba2e12d89
         Public key's random art:
                  +--[ RSA 2048]----+
                  |           .     |
                  |          . .   .|
                  |         + . . ..|
                  |      . + ..o ...|
                  |       oS...o....|
                  |      .  ..o.o  .|
                  |...  . .   .o..  |
                  |Eoo.  o    .o.   |
                  | oo....o.  .o.   |
                  +-----------------+
Is the above information ok? (y/N): y
Signing certificate...

[root@ol7client certificate]#
  • Check that all files have been created and none of them is empty:
[root@ol7client certificate]# ls -l
total 28
-rw-------. 1 root root 5816 Oct  8 00:21 ca-key.pem
-rw-r--r--. 1 root root 1107 Oct  8 00:25 ca.pem
-rw-r--r--. 1 root root 1168 Oct  8 00:39 ol7server-cert.pem
-rw-------. 1 root root 5823 Oct  8 00:31 ol7server-key.pem
-rw-------. 1 root root 2348 Oct  8 00:35 ol7server-request.pem
[root@ol7client certificate]#

Distribute TLS certificates to enable secure remote logging

  • We created the certificates on client machine and now we must copy these keys (certificates) to our server machine node; before we copying the keys we create a dedicated directory where to store those keys.
[root@ol7server ~]# mkdir /etc/rsyslog-keys
  • Copy the keys from ol7client to ol7server
[root@ol7client certificate]# scp ol7server-cert.pem ol7server-key.pem ca.pem ol7server:/etc/rsyslog-keys/
root@ol7server's password:
ol7server-cert.pem 100% 1168     1.0MB/s   00:00
ol7server-key.pem  100% 5823     6.5MB/s   00:00
ca.pem             100% 1107     1.1MB/s   00:00
[root@ol7client certificate]#

 

Open Firewall ports on server to grant client access to Rsyslog service.

  • With “firewalld” we need to open port “6514” the TCP/TLS port will be leveraged on this example.
[root@ol7server ~]# firewall-cmd --permanent --add-port=6514/tcp
success
[root@ol7server ~]# firewall-cmd --reload
success
[root@ol7server ~]#

 

Configure “Rsyslog” on server to accept remote logs.

Now we need to do some configuration changes on the log server (ol7server) to receive messages from the client (ol7client) over TCP using TLS certificates.

  • Create a new file /etc/rsyslog.d/ol7server.conf. Save following content in the file:
# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/rsyslog-keys/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog-keys/ol7server-cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog-keys/ol7server-key.pem

$ModLoad imtcp  # TCP listener
$InputTCPServerStreamDriverMode 1  # run driver in TLS-only mode
$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 6514  # start up listener at port 10514
$template RemoteLogs,"/var/log/rsyslog/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
  • Restart the rsyslog service and check its status
[root@ol7server ~]# systemctl restart rsyslog
[root@ol7server ~]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-08 00:50:03 CEST; 2s ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 9165 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─9165 /usr/sbin/rsyslogd -n

Oct 08 00:50:03 ol7server systemd[1]: Starting System Logging Service...
Oct 08 00:50:03 ol7server rsyslogd[9165]:  [origin software="rsyslogd" swVersion="8.24.0-57.0.1.el7_9.1" x-pid="9165" x-info="http://www.rsyslog.com"] start
Oct 08 00:50:03 ol7server systemd[1]: Started System Logging Service.
[root@ol7server ~]#

 

Configure SELinux to allow “Rsyslog” processes proper access to audit logs on Client machine

  • Create new “rsyslog” SELinux dedicated directory
[root@ol7client ~]# mkdir selinux
[root@ol7client ~]# cd selinux/
[root@ol7client selinux]#
  • Create a new SELINUX policy file /root/selinux/rsyslog.te. Save following content in the file:
module rsyslog 1.0;
require {
        type syslogd_t;
        type auditd_log_t;
        class file read;
        class file ioctl;
        class file open;
        class file getattr;
        class dir search;
        class dir read;
        class dir open;
        class dir getattr;
}
#============= syslogd_t ==============
allow syslogd_t auditd_log_t:file read;
allow syslogd_t auditd_log_t:file ioctl;
allow syslogd_t auditd_log_t:file open;
allow syslogd_t auditd_log_t:file getattr;
allow syslogd_t auditd_log_t:dir search;
allow syslogd_t auditd_log_t:dir read;
allow syslogd_t auditd_log_t:dir open;
allow syslogd_t auditd_log_t:dir getattr;
  • Compile the SELinux code with the following commands
[root@ol7client selinux]# checkmodule -M -m -o rsyslog.mod rsyslog.te
checkmodule:  loading policy configuration from rsyslog.te
checkmodule:  policy configuration loaded
checkmodule:  writing binary representation (version 19) to rsyslog.mod
[root@ol7client selinux]# semodule_package -o rsyslog.pp -m rsyslog.mod
  • Import the new policy to SELinux configuration
[root@ol7client selinux]# semodule -i rsyslog.pp

 

Client configuration to send log messages (audit included) securely.

Now let’s configure the client (ol7client) to transfer the logs securely to the remote log server (ol7server).

  • The first step would be to create a directory to store our key and copy the ca.pem to the same directory
[root@ol7client ~]# mkdir /etc/rsyslog-keys
[root@ol7client selinux]# cp /root/certificate/ca.pem /etc/rsyslog-keys/
  • Create a new rsyslog configuration file, /etc/rsyslog.d/ol7client.conf to add required configuration parameters:
# certificate files
$DefaultNetStreamDriverCAFile /etc/rsyslog-keys/ca.pem

# make gtls driver the default
$DefaultNetStreamDriver gtls
$ActionSendStreamDriverMode 1     # run driver in TLS-only mode
$ActionSendStreamDriverAuthMode anon

# add audit log
$ModLoad imfile
$InputFileName /var/log/audit/audit.log
$InputFileTag tag_audit_log:
$InputFileStateFile audit_log
$InputFileSeverity info
$InputFileFacility local6
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

*.*     @@(o)ol7server:6514                 # forward everything to remote server

NOTE: This will forward every syslog message, as well as audit log, to your remote log server ol7server.

  • Restart the rsyslog service and check the status
[root@ol7client selinux]# systemctl restart rsyslog
[root@ol7client selinux]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-08 01:02:56 CEST; 3s ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 9247 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─9247 /usr/sbin/rsyslogd -n

Oct 08 01:02:56 ol7client systemd[1]: Starting System Logging Service...
Oct 08 01:02:56 ol7client rsyslogd[9247]:  [origin software="rsyslogd" swVersion="8.24.0-57.0.1.el7_9.1" x-pid="9247" x-info="http://www.rsyslog.com"] start
Oct 08 01:02:56 ol7client systemd[1]: Started System Logging Service.
[root@ol7client selinux]#
 

Verify the remote logging works

  • The following example shows how to send a dummy message from our client to the server and verify our configuration correctly works.
[root@ol7client ~]# logger "MESSAGE FROM OL7CLIENT"
  • Check the syslog on the server
[root@ol7server ~]# cat /var/log/rsyslog/ol7client/root.log
Oct  8 01:04:38 ol7client root: MESSAGE FROM OL7CLIENT
[root@ol7server ~]#
  • To possibly check “audit” events just try to disconnect and reconnect to the ol7client machine and verify that the operation (login) is also logged on the server (ol7server):
[root@ol7server ol7client]# cat /var/log/rsyslog/ol7client/tag_audit_log.log |grep sshd
Oct  8 01:04:16 ol7client tag_audit_log: type=USER_LOGIN msg=audit(1633647849.515:79): pid=9260 uid=0 auid=0 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=login id=0 exe="/usr/sbin/sshd" hostname=dhcp-10-175-46-130.vpn.oracle.com addr=10.175.46.130 terminal=/dev/pts/0 res=success'
Oct  8 01:04:16 ol7client tag_audit_log: type=USER_START msg=audit(1633647849.539:80): pid=9260 uid=0 auid=0 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=login id=0 exe="/usr/sbin/sshd" hostname=dhcp-10-175-46-130.vpn.oracle.com addr=10.175.46.130 terminal=/dev/pts/0 res=success'
[root@ol7server ~]#

Feel free to leave your comments on my twitter account here.