As you saw in the Turning SFTP cloud native blog post, with a bit of imagination, you can get your file-based integration rid of dust and design a fresh and polished, fully backward compatible, cloud native SFTP solution on Oracle Cloud Infrastructure (OCI).

Moving from a fancy architecture to a running solution isn’t just a theory. The process involves multiple components, several infrastructure provisioning, and configuration activities. But cloud technologies offer infrastructure-as-code, smoothing manual processes associated to infrastructure provisioning, configuration, and management. Using the OCI Terraform Provider and OCI Resource Manager to create a cloud native SFTP couldn’t be easier.

If you want to see your cloud native SFTP running, open the oci-cloud-native-sftp GitHub repository and click

A graphic depicting the Deploy to Oracle Cloud button..

If you want to know the technical details, keep reading.

Cloud native SFTP briefing

A cloud native SFTP solution has the following components:

Using Events, you can get updates when a file is uploaded, modified, or deleted. To get everything working, a lot of boring and sometimes not trivial tasks are required.

  1. Create the network where the network load balancer and the Compute instances are deployed. Alternatively, you can use an existing virtual cloud network (VCN) and subnets, but the network configuration must match some requirements. The network load balancer subnet must be public with a security list that enables TCP traffic on port 22 from any source.

    Make the SFTP servers subnet private with a security list that enables TCP traffic on port 22 from at least the network load balancer subnet. SFTP servers require a connection to Object Storage and internet access to install some software packages.

    If your Compute instances use an Oracle Linux image, having a service gateway attached to SFTP servers subnet is enough. Through a service gateway, you can access both Object Storage and regional yum mirrors hosted by OCI. Otherwise, attach a NAT gateway to the SFTP servers subnet, because the required software packages for others Linux distributions are available in repositories external to OCI.

  2. Create the Object Storage bucket where files are going to be stored. The objects within the bucket have to be configured for emitting events.

  3. Create a customer secret key for accessing Object Storage using the S3 Compatibility API.

  4. Create and configure the Compute instances used for running SFTP servers. Mount the Object Storage bucket using s3fs. Instances must share a HostKey to avoid issues with SFTP clients. Create and enable SFTP users for secure access to shared files.

  5. Create the network load balancer for SFTP user session distribution. Add the Compute instances as backends to the network load balancer.

  6. Subscribe to meaningful events published when something occurs in your shared bucket.

You can create the cloud native SFTP resources in a simple and straightforward way, thanks to the OCI Terraform Provider, which supports conditional logic based on the provided input variables. For example, you can create the required network resources from scratch or reuse existing ones.

The trickiest part is the SFTP server configuration. While the creation of the Compute instances is about selecting the right options, their initialization requires more tooling. The industry-standard multidistribution method for cross-platform cloud instance initialization, supported across all major public cloud providers and available out-of-the-box within OCI platform images, is cloud-init.

Bootstrapping with cloud-init

Cloud-init automates the initialization of cloud instances during system boot. You can configure it to perform the following tasks:

  • Create users and groups

  • Install packages

  • Write files

Cloud-init uses YAML-formatted file instructions to perform the following tasks:

  • When a Compute instance boots, the cloud-init service starts and searches for and runs the instructions.

  • Tasks complete during the first boot or on subsequent boots of your Compute instances.

For configuring the Compute instances to run SFTP servers with the required configuration, the following cloud-init modules are used:

  • Users and groups for creating the SFTP user and group and Compute’s instance default user

  • SSH for configuring the same SSH host keys on every Compute instance

  • Write files for copying the customized SSH server configuration and the installation or configuration script for mounting Object Storage bucket using s3fs

  • Runcmd for running the s3fs installation and configuration script

SSH server configuration

By default, SFTP is not enabled in SSH server configuration. So, update the sshd_config file to include SFTP configuration with the following command:

Subsystem sftp internal-sftp
Match Group sftp
ChrootDirectory /mnt/sftp/%u
ForceCommand internal-sftp

The Match Group keyword is a conditional block: The subsequent keywords are applied only if the user being authenticated belongs to sftp group.

A chroot is performed after the authentication to enforce a proper resources isolation. The /mnt/sftp/%u directory, where %u is the username, must be root-owned and not writable by any other user or group.

Because Terraform TLS provider support for Ed25519 certificates is missing, configure the SSH server to use only RSA and ECDSA keys with the following commands:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

The server’s private keys are generated by Terraform and then copied to the Compute instances by cloud-init using the SSH module.

SFTP directories

Because of the chroot directory permission requirements, to enable the SFTP user to create files and directories, we need to create a subdirectory and assign to it read/write permissions for the SFTP user. Our example uses the username ‘foo’ and the subdirectory ‘private.’

  • /mnt/sftp/foo is owned by root:root, so the foo user has no write permissions.

  • /mnt/sftp/foo/private is owned by foo:sftp, so the foo user has full write permissions.

The bootstrap script run by cloud-init creates the SFTP directories using the Runcmd module.

Mounting the bucket

The SFTP server instances enable access to the same shared Object Storage bucket, mounting it as filesystem using s3fs. After installing s3fs, the following configuration activities are required:

  • Creation of file /etc/s3fs/oci_passwd for storing S3 compatibility API credentials

  • Adding the required instruction to /etc/fstab to enable automatic mounting of the bucket

The instruction added to /etc/fstab looks like the following code block:

s3fs#<bucket-name> /mnt/sftp/<sftp-user> fuse _netdev,allow_other,use_path_request_style,passwd_file=/etc/s3fs/oci_passwd,url=https://<bucket-namespace>.compat.objectstorage.<oci-region>.oraclecloud.com/ 0 0

  • <bucket-name> is the name of the bucket.

  • <sftp-user> is the username of the SFTP user.

  • <sftp-user> is the username of the SFTP user.

  • <bucket-namespace> is the Object Storage namespace.

  • <oci-region> is the name of the OCI region, such as eu-frankfurt-1.

The s3fs installation and the configuration required for mounting the Object Storage bucket are run by the bootstrap script run by cloud-init using the Runcmd module.

Be aware of updates

To get notified when a file is upload to your Cloud Native SFTP, configure a Notifications service subscription to send an e-mail when actions occur on the files shared through SFTP. You can also implement updates from Object Storage events using OCI Streaming or OCI Functions services or use another Notifications subscription, such as Slack or a custom HTTPS endpoint.

Deployment

Instead of deploying your cloud native SFTP using the Terraform CLI, use Oracle Resource Manager with the following steps:

  1. Click A graphic depicting the Deploy to Oracle Cloud button.

  2. If you aren’t already signed in, enter the tenancy and user credentials.

  3. Review and accept the terms and conditions.

  4. Select the region where you want to deploy the stack.

  5. Follow the on-screen prompts and instructions to create the stack.

  6. After creating the stack, click Terraform Actions, and select Plan.

  7. Wait for the job to be completed, and review the plan. To make any changes, return to the Stack Details page, click Edit Stack, and make the required changes. Then, run the Plan action again.

  8. If no further changes are necessary, return to the Stack Details page, click Terraform Actions, and select Apply.

Want to know more?

As someone said, talk is cheap. If you want to see the code, you can find it within the oci-cloud-native-sftp GitHub repository. Want to talk it over? Leave us a comment.