Quick method to send syslog events over a ssh tunnel to a remote syslog server. This procedure used Oracle Linux machines using systemd to get ssh-tunnel restarted as a Linux service. The syslog daemon in use is rsyslog. This method will refer to the server running rsyslogd as the log server (ol7-server), and the client that is sending the logs remotely will be referred to as the syslog client (ol7-client).

Create syslog-remote users

Create a user on both the syslog client (ol7-client) and the syslog-server (ol7-server). This user will be used for the ssh tunnel between the two systems.

  • ol7-client (root access):
    • # groupadd -g 10514 rsyslog-remote
    • # useradd -u 10514 -g 10514 rsyslog-remote
  • ol7-server (root access):
    • # groupadd -g 10514 rsyslog-remote
    • # useradd -u 10514 -g 10514 rsyslog-remote

NOTE: do not add a password to “rsyslog-remote” user so ssh access will be only granted by ssh keys.

Configuring ssh public key authentication

This assumes that both systems sshd is configured to allow authentication with public keys.
On both the syslog client and the syslog server, create a public and private keypair for the rsyslog-remote user.

  • ol7-client (root access):
    • # su - rsyslog-remote
    • # ssh-keygen -t rsa
  • ol7-server (root access):
    • # su - rsyslog-remote
    • # ssh-keygen -t rsa

On the syslog client, copy the rsyslog-remote user’s public key to syslog server:

On the syslog server, copy the rsyslog-remote user’s public key to syslog client:

On the syslog client, create the “rsyslog-remote” authorized_keys file containing both the local (syslog client) and remote (syslog server) keys and then grant proper permissions to the file.

  • ol7-client (root access):
    • # cp /tmp/ol7-server.public /home/rsyslog-remote/.ssh/authorized_keys
    • # cat /home/rsyslog-remote/.ssh/id_rsa.pub >> /home/rsyslog-remote/.ssh/authorized_keys
    • # chown rsyslog-remote:rsyslog-remote /home/rsyslog-remote/.ssh/authorized_keys
    • # chmod 600 /home/rsyslog-remote/.ssh/authorized_keys

On the syslog server, create the “rsyslog-remote” authorized_keys file containing both the local (syslog server) and remote (syslog client) keys and then grant proper permissions to the file.

  • ol7-server (root access):
    • # cp /tmp/ol7-client.public /home/rsyslog-remote/.ssh/authorized_keys
    • # cat /home/rsyslog-remote/.ssh/id_rsa.pub >> /home/rsyslog-remote/.ssh/authorized_keys
    • # chown rsyslog-remote:rsyslog-remote /home/rsyslog-remote/.ssh/authorized_keys
    • # chmod 600 /home/rsyslog-remote/.ssh/authorized_keys

Once the setup is completed, verify that ssh connection can happen without any password (with user “rsyslog-remote“) and test access from:

  • syslog server to syslog server
  • syslog server to syslog client
  • syslog client to syslog client
  • syslog client to syslog server

Configuring ssh Reverse Tunnel

We are now going to create a reverse ssh tunnel on the syslog server that listens on port 10514 on the loopback interface (127.0.0.1 and ::1 ) on the client and empties out on our syslog server on port 6514; so, any packet that is sent to 127.0.0.1:10514 on the syslog client, will be encrypted by the reverse ssh session and be available to be read on port 6514 on the syslog server.
On the syslog server, the reverse tunnel can be created with the following command:

  • ol7-server (rsyslog-remote access):
    • # ssh -nN -R 10514:<syslog-server-host>:6514 <syslog-client-host>
    • Example:
    • # ssh -nN -R 10514:ol7-server:6514 ol7-client

Create Linux SystemD service for reverse tunnel

To avoid any stop or issue for systems rebooted, with those instructions we can create a Linux systemd service that will automatically start the reverse tunnel for rsyslog on syslog server.

Create a script file (start-rsyslog-<log client>.sh) with ssh string defined above and add proper permissions to the same:

  • ol7-server (root access):
    • # echo "ssh -nN -R 10514:ol7-server:6514 ol7-client" > /usr/local/bin/start-rsyslog-ol7-client.sh
    • # chown rsyslog-remote:rsyslog-remote /usr/local/bin/start-rsyslog-ol7-client.sh
    • # chomd 700 /usr/local/bin/start-rsyslog-ol7-client.sh

Create a systemd service file dedicated to rsyslog-remote reverse tunnel creation; use your preferred editor to create and edit the file

  • ol7-server (root access):
    • # vi /etc/systemd/system/rsyslog-remote@.service
    • # cat /etc/systemd/system/rsyslog-remote@.service

[Unit]
Description=rsyslog-remote by ssh %i
After=rsyslog.service

[Service]
Type=simple
User=rsyslog-remote
ExecStart=/bin/bash /usr/local/bin/start-rsyslog-%i.sh

[Install]
WantedBy=multi-user.target

Enable and start the “rsyslog-remote” service on your system.

  • ol7-server (root access):
    • # systemctl enable rsyslog-remote@<log client>
    • Example:
      • # systemctl enable rsyslog-remote@ol7-client
    • # systemctl start rsyslog-remote@<log client>
    • Example
      • # systemctl start rsyslog-remote@ol7-client

Configure SELinux to allow “Rsyslog” processes proper access to audit logs on Client machine

Create new “rsyslog” SELinux dedicated directory.

  • ol7-client (root access):
    • # mkdir selinux
    • # cd selinux

Create a new SELINUX policy file /root/selinux/rsyslog.te and save following content in the file:

module rsyslog 1.0;
require {
        type syslogd_t;
        type auditd_log_t;
        class file read;
        class file ioctl;
        class file open;
        class file getattr;
        class dir search;
        class dir read;
        class dir open;
        class dir getattr;
}
#============= syslogd_t ==============
allow syslogd_t auditd_log_t:file read;
allow syslogd_t auditd_log_t:file ioctl;
allow syslogd_t auditd_log_t:file open;
allow syslogd_t auditd_log_t:file getattr;
allow syslogd_t auditd_log_t:dir search;
allow syslogd_t auditd_log_t:dir read;
allow syslogd_t auditd_log_t:dir open;
allow syslogd_t auditd_log_t:dir getattr;

Compile the SELinux code with the following commands:

  • ol7-client (root access):
    • # checkmodule -M -m -o rsyslog.mod rsyslog.te
    • # semodule_package -o rsyslog.pp -m rsyslog.mod

Import the new policy to SELinux configuration

  • ol7-client (root access):
    • # semodule -i rsyslog.pp

Configuring rsyslog on syslog server

Apply following configuration changes on the log server (ol7-server) to receive messages from the log client (ol7-client) over TCP using SSH reverse tunnel.

  • Create a new file /etc/rsyslog.d/ol7server.conf and save following content in the file:
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 6514
$template RemoteLogs,"/var/log/rsyslog/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
  • Enable (if needed) and restart the rsyslog service:
    • # systemctl enable rsyslog
    • # systemctl restart rsyslog

Configuring rsyslog on syslog client

Apply following configuration changes on the log client (ol7-client) to transfer the logs securely to the remote log server (ol7-server).

  • Create a new rsyslog configuration file /etc/rsyslog.d/ol7client.conf to add required configuration parameters

# add audit log
$ModLoad imfile
$InputFileName /var/log/audit/audit.log
$InputFileTag tag_audit_log:
$InputFileStateFile audit_log
$InputFileSeverity info
$InputFileFacility local6
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

*.* @@127.0.0.1:10514

NOTE: This will forward every syslog message, as well as audit log, to your remote log server.

Restart the rsyslog service and check the status

  • ol7-client (root access):
    • # systemctl restart rsyslog
    • # systemctl status rsyslog

Verify the remote logging works

The following example shows how to send a dummy message from our client to the server and verify our configuration correctly works.

  • ol7-client (root access):
    • # logger "MESSAGE FROM OL7CLIENT"

Check the syslog reported on the server

  • ol7-server (root access):
    • # cat /var/log/rsyslog/ol7client/root.log

To possibly check “audit” events just try to disconnect and reconnect to the ol7-client machine and verify that the operation (login) is also logged on the server (ol7-server).

Appendix A: add further log clients

The configuration above just covers one syslog client and one syslog server; the idea on this kind of architecture/solutions is to have more clients with one single server.

The idea is to have all the clients sendings their logs to the same local port (10514) with TCP protocol while the log server will create one tunnel (with dedicated port) for each client; so, in case we’re adding one new client, the configuration will include a progressive port number for Rsyslog process listening on the server. The first port was 6514 and on this example we’re going to leverage +1 (6515).

The configuration on the client is the same as specified on section “Configuring rsyslog on syslog client” above.

Create the script file (start-rsyslog-<log client>.sh) to start “rsyslog-remote” service and add new line dedicated to the new log client

  • ol7-server (root access):
    • # cat /usr/local/bin/start-rsyslog-ol7-client2.sh
ssh -nN -R 10514:ol7-server:6515 ol7-client2

So, the new client (ol7-client-new) port (10514) will be redirected to log server (ol7-server) port 6515 (6514+1); this port number has to be progressive for each log client node added.

New port (for this client 6515) needs to be accepted by SELinux policies for Rsyslog listening process.

Execute following command to enable “progressive” port 6515, to SELinux “accepted ports” configuration, for “rsyslog“

  • ol7-server (root access):
    • # semanage port -a -t syslog_tls_port_t -p tcp 6515

The other required change is to get “rsyslog” on log server (ol7-server) aware of logs coming also on port 6515.

  • Create a new file /etc/rsyslog.d/ol7-client2.conf and save following content in the file:
# provides TCP syslog reception
$InputTCPServerRun 6515

Enable and start the “rsyslog-remote” service on your system.

  • ol7-server (root access):
    • # systemctl enable rsyslog-remote@<log client>
    • Example:
      • # systemctl enable rsyslog-remote@ol7-client2
    • # systemctl start rsyslog-remote@<log client>
    • Example
      • # systemctl start rsyslog-remote@ol7-client2