Creating ULN Mirrors for Exadata Database Server Updates

January 28, 2024 | 13 minute read
Alex Blyth
Senior Principal Product Manager - Exadata
Text Size 100%:

Did you know that Exadata Patch Manager (patchmgr) - the tool for updating Exadata with our regular updates - can use a dnf (formerly yum) repository as a source when updating database servers?

Did you also know that Exadata monthly updates are available as channels on the Unbreakable Linux Network (ULN)?

Hopefully, you said yes to both of these questions. However, I received enough Slack and email messages that indicate this is not well-known and compelled me to write this blog.

So, let's start at the start. As I said above, when updating KVM Hosts, KVM Guests, DomU, or Dom0(X5-X8), you are able to give patchmgr a URL location of the update you wish to apply, and patchmgr will happily use this to update packages. Looking at the syntax for patchmgr, you can see the --yum_repo switch available for the precheck and upgrade commands. 

./patchmgr --dbnodes database_node_file
  { --backup [--rolling] [--unkey] |
  --precheck { --yum_repo base_URL | --iso_repo zipped_iso_file } [ --target_version version ] [--unkey] |
  --upgrade { --yum_repo base_URL | --iso_repo zipped_iso_file } [ --target_version version ] [--rolling] [--unkey] | 
  --complete [ --target_version version ] [--unkey] |
  --rollback [--rolling] [--unkey] |
  --cleanup  [--unkey] }
  [ --log_dir { log_directory | auto } ]

Normally, you might simply download the iso file from My Oracle Support to the server you will be executing patchmgr from, but what if you have a very large fleet of Exadata (including simply having many VM Clusters) or a wide range of target versions you need to apply, or if you have many administrators that are managing and apply updates to all these assets. Might be good to centralize the patch source management.

To make widespread distribution of packages easy and standardized for Exadata database server resources, we package the database server updates as a channel in the ULN.

 

Sidebar - Storage Servers are updated out of place into their inactive system partition, so a repository is not required. The RoCE and management switches updates are delivered in a switch-specific manner, so they also do not need a package repository. 

 

Let's take a quick look in ULN. Login into the ULN at https://linux.oracle.com, click on the Channels tab and you should see something like the following.

Oracle Linux Channel list

If you look at the bottom, once you browse past all the other Oracle Linux 8 repositories, you will see channels named "Exadata release 23.1.5.0.0 db server installation packages (x86_64)" and Exadata release 23.1.9.0.0 db server installation packages (x86_64)" and so forth. These correspond to actual repository names - "exadata_dbserver_23.1.5.0.0_x86_64_base" and "exadata_dbserver_23.1.9.0.0_x86_64_base" in the examples I just used, and each has about 1,060 or so packages. 

Exadata ULN Channels

You'll also note that we provide these updates as full installations rather than using a "latest" repo. We curate, test, and certify the packages in each release across the entire Exadata hardware and software stack.

So, we can see the channels in ULN, but how do you use them?

I'm glad you asked. By design, patchmgr does not talk directly to ULN. To allow patchmgr to use these channels, we need to mirror them to an internal server that the database servers and system executing patchmgr can access.

Enter the dnf mirror!!! 

A dnf (or yum) mirror is simply a Linux server that has been registered with ULN, been configured to download (mirror) specific channels, and make them available via HTTP. 

So let's get cracking. I am going to make a few assumptions here:

  1. You have a VM or server running Oracle Linux 8 you can use as a dnf mirror server (mine is called "yumrepo" in the example below).
  2. You have root access to dnf mirror server.
  3. The dnf mirror server is accessible to all the Exadata database servers (KVM Hosts, KVM Guests, Dom0's, and DomU's) you will be updating.
  4. You have a CSI active in ULN.

The core steps we're going to cover now are as follows:

  1. Register the dnf mirror server with ULN
  2. Subscribe the server to one or more Exadata channels
  3. Mirror said channels to the server
  4. Configure Apache HTTP to serve our mirrored channels

Register the dnf mirror server with ULN

This step is straightforward. Each OL8 installation includes a simple utility called uln-register to "connect" our dnf mirror server to the ULN and give the server knowledge of the channels we want to mirror.

As the root user on the dnf mirror server, execute the uln-register utility.

[root@yumrepo ~]# uln-register

The subsequent steps are very straightforward, and well documented here - https://docs.oracle.com/en/operating-systems/oracle-linux/software-management/sfw-mgmt-ConfigureaSystemtoUseULN.html#register-with-uln

You will also need to tell the ULN that this server is a "Yum Server". A good way of doing this is by using the uln-channel command.

[root@yumrepo ~]# uln-channel --enable-yum-server

If prompted, specify the appropriate ULN user name and password.

Add Exadata Channels

Now that the dnf mirror server is registered with the ULN, we can subscribe to the relevant Exadata channels. To do so, navigate to the System Detail page on linux.oracle.com for the dnf mirror server you just registered. It should look something like the following:

DNF Mirror System Details

Notice the "Manage Subscriptions" button? Click it - you'll be taken to a shuttle menu where you can add the appropriate Exadata release channels. You can see below, I've added the channels for Exadata System Software 23.1.9, 23.1.8, and 23.1.7. 

Exadata System Software Channel selection for mirroring

Click "Save Subscriptions" and the next time you do a dnf update on your server, these repos will be available for use - or more importantly, for mirroring. 

Before we move on, take the time to look through the available channels - you will notice there are a multitude of Exadata Releases available. So you can use a single dnf mirror server to host Exadata System Software 22.1 (based on Oracle Linux 7), 23.1 (based on Oracle Linux 8), and any future releases. All from a single mirror!

Another way of adding channels to your dnf mirror is using uln-channel. Lets look at the help.

[root@yumrepo ~]# uln-channel --help
Usage: uln-channel [options]

Options:
-c CHANNELS, --channel=CHANNELS
                        name of channel you want to (un)subscribe
-a, --add             subscribe to channel
-r, --remove          unsubscribe from channel
-l, --list            list channels
-b, --base            show base channel of a system
-L, --available-channels
                        list all available child channels
-v, --verbose         verbose output
-u USER, --user=USER  your user name
-p PASSWORD, --password=PASSWORD
                        your password
--enable-yum-server   set yum server
--disable-yum-server  set yum server
-h, --help            show this help message and exit

So to add one or more channels from the command line, we can so something like the following.

uln-channel -a -c channel [-c channel …]

To check the channels our dnf mirror server us subscribed to uln-channel --list is used. 

[root@yumrepo ~]# uln-channel --list
ol8_x86_64_UEKR7
ol8_x86_64_baseos_latest
ol8_x86_64_appstream
ol8_x86_64_ksplice
exadata_dbserver_23.1.9.0.0_x86_64_base
exadata_dbserver_23.1.8.0.0_x86_64_base
exadata_dbserver_23.1.7.0.0_x86_64_base

Before we mirror any of these, lets get a little information about them - specifically how much space each will use when mirrored. 

[root@yumrepo ~]# dnf repoinfo exadata*

This system is receiving updates from Unbreakable Linux Network or Spacewalk.
Exadata release 23.1.7.0.0 db server installation packages (x86_64)                               491 kB/s | 1.4 MB     00:02
Exadata release 23.1.8.0.0 db server installation packages (x86_64)                               3.3 kB/s | 3.0 kB     00:00
Exadata release 23.1.9.0.0 db server installation packages (x86_64)                               3.3 kB/s | 3.0 kB     00:00
Repo-id            : exadata_dbserver_23.1.7.0.0_x86_64_base
Repo-name          : Exadata release 23.1.7.0.0 db server installation packages (x86_64)
Repo-status        : enabled
Repo-revision      : 1698083972
Repo-updated       : Mon 23 Oct 2023 05:59:32 PM GMT
Repo-pkgs          : 1,069
Repo-available-pkgs: 1,029
Repo-size          : 1.3 G
Repo-baseurl       : https://linux-update.oracle.com/XMLRPC/GET-REQ/exadata_dbserver_23.1.7.0.0_x86_64_base
Repo-expire        : 1 second(s) (last: Mon 22 Jan 2024 03:50:32 AM GMT)

Repo-id            : exadata_dbserver_23.1.8.0.0_x86_64_base
Repo-name          : Exadata release 23.1.8.0.0 db server installation packages (x86_64)
Repo-status        : enabled
Repo-revision      : 1700242893
Repo-updated       : Fri 17 Nov 2023 05:41:33 PM GMT
Repo-pkgs          : 1,070
Repo-available-pkgs: 1,030
Repo-size          : 1.4 G
Repo-baseurl       : https://linux-update.oracle.com/XMLRPC/GET-REQ/exadata_dbserver_23.1.8.0.0_x86_64_base
Repo-expire        : 1 second(s) (last: Mon 22 Jan 2024 12:45:36 AM GMT)

Repo-id            : exadata_dbserver_23.1.9.0.0_x86_64_base
Repo-name          : Exadata release 23.1.9.0.0 db server installation packages (x86_64)
Repo-status        : enabled
Repo-revision      : 1703126080
Repo-updated       : Thu 21 Dec 2023 02:34:40 AM GMT
Repo-pkgs          : 1,070
Repo-available-pkgs: 1,030
Repo-size          : 1.5 G
Repo-baseurl       : https://linux-update.oracle.com/XMLRPC/GET-REQ/exadata_dbserver_23.1.9.0.0_x86_64_base
Repo-expire        : 1 second(s) (last: Mon 22 Jan 2024 12:45:37 AM GMT)
Total packages: 3,209

You can see each channel is about 1.5 GB in size, and each contains roughly 1,050 packages. Remember that the Exadata releases include only packages needed to run Oracle Database and Exadata System Software, so the package list is much smaller when compared to the base Linux 8 repo which clocks in at over 1,700 packages!

Prepare mirror directories

Next, we need to prepare a location for the channels to be mirrored into. 

Let's use /u01/yum as our base.

[root@yumrepo ~]# mkdir -p /u01/yum

Install Apache HTTP Server and dnf-utils

To mirror the channels locally, we need to make sure yum-utils is installed on our server. 

[root@yumrepo ~]# rpm -qa | grep -i yum-utils
yum-utils-4.0.21-23.0.1.el8.noarch

If the above does not return an entry, you'll need to install the package.

[root@yumrepo ~]# dnf install -y yum-utils

As we need an HTTP server to serve our mirrored channels, we also need to install Apache HTTP Server.

[root@yumrepo ~]# dnf install -y httpd

Apache HTTP Server Configuration

Now that Apache is installed and we have a location for our channels to be mirrored into, lets do a few configuration steps.

Link the base location from earlier to /var/www/html/yum.

[root@yumrepo ~]# ln -s /u01/yum /var/www/html/yum

If your Oracle Linux server has SELinux enabled in enforcing mode, you need to set the correct context for the mirror directory.

[root@yumrepo ~]# semanage fcontext -a -t httpd_sys_content_t "/u01/yum(/.*)?"
[root@yumrepo ~]# restorecon -RFv /u01/yum

We can now edit the Apache httpd.conf file.

[root@yumrepo ~]# sed -i "/^#ServerName www.example.com:80/a ServerName $(hostname -i):80" /etc/httpd/conf/httpd.conf

And verify that the FollowSymLinks option is active. Under the <Directory "/var/www/html"> heading, verify that FollowSymLinks is included in the Options line.

[root@yumrepo ~]# cat /etc/httpd/conf/httpd.conf | grep "/var/www/html" -A20
DocumentRoot "/var/www/html"
    
#
# Relax access to content within /var/www.
#

    AllowOverride None
    # Allow open access:
    Require all granted

    
# Further relax access to the default document root:

    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.4/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride None

Next, let's start the web server and check its status.

[root@yumrepo ~]# systemctl enable --now httpd
[root@yumrepo ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2024-01-19 03:17:36 GMT; 3 days ago
    Docs: man:httpd.service(8)
Process: 230188 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 155561 (httpd)
Status: "Running, listening on: port 80"
    Tasks: 213 (limit: 5408)
Memory: 11.2M
CGroup: /system.slice/httpd.service
        ├─155561 /usr/sbin/httpd -DFOREGROUND
        ├─230194 /usr/sbin/httpd -DFOREGROUND
        ├─230195 /usr/sbin/httpd -DFOREGROUND
        ├─230196 /usr/sbin/httpd -DFOREGROUND
        └─230197 /usr/sbin/httpd -DFOREGROUND

Jan 21 03:09:02 yumrepo systemd[1]: Reloading The Apache HTTP Server.
Jan 21 03:09:02 yumrepo systemd[1]: Reloaded The Apache HTTP Server.
Jan 21 03:09:02 yumrepo httpd[155561]: Server configured, listening on: port 80
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Finally, let's make sure inbound traffic to port 80 can get to our server.

[root@yumrepo ~]# firewall-cmd --permanent --add-service=http
[root@yumrepo ~]# firewall-cmd --reload

Sync Repositories to the Local Yum Mirror

We can now mirror our Exadata channels! This is done using the dnf reposync command as follows. Note that I am mirroring three channels in one command here by using multiple --repoid options. Depending on your network connection and the number of channels you are mirroring, this may take some time. 

[root@yumrepo ~]# dnf reposync --delete --download-metadata -p /u01/yum --repoid exadata_dbserver_23.1.7.0.0_x86_64_base --repoid exadata_dbserver_23.1.8.0.0_x86_64_base --repoid exadata_dbserver_23.1.9.0.0_x86_64_base

This will now download all the packages in our three subscribed channels into /u01/yum, and Apache will immediately serve these in the form of http://yumrepo/yum/exadata_dbserver_xxx_x86_64_base where xxx is the release number, e.g. release 23.1.7.0.0 would be http://yumrepo/yum/exadata_dbserver_23.1.7.0.0_x86_64_base.

Connecting patchmgr

To use the yum mirror with Exadata's patchmgr utility is as simple as it gets. Both the precheck and upgrade steps have the option to provide a --yum-repo as input. 

patchmgr --dbnodes database_node_file --precheck --yum_repo base_URL --target_version version
patchmgr --dbnodes database_node_file --upgrade --yum_repo base_URL --target_version version --rolling

Putting that into a more concrete example, lets assume you are updating a set of Exadata database servers to the 23.1.9.0.0 release.

patchmgr --dbnodes database_node_file --precheck --yum_repo http://yumrepo/yum/exadata_dbserver_23.1.9.0.0_x86_64_base --target_version 23.1.9.0.0
patchmgr --dbnodes database_node_file --upgrade --yum_repo http://yumrepo/yum/exadata_dbserver_23.1.9.0.0_x86_64_base --target_version 23.1.9.0.0 --rolling

Note that I am not covering how to use patchmgr fully in the above, merely how to have it use the mirror you created. You should always take a backup of the server before you update it. The Exadata documentation is a great place to start on the use of patchmgr and its correct order of execution. 

Final thoughts

There you have it, creation of and subsequent use by patchmgr of a yum mirror of Exadata software channels. Remember that these mirrors are only for database server updates, 

As you can see, setting up the mirror server is not particularly difficult and in the long run, makes life easier when you are maintaining your Exadata database servers. As I mentioned earlier, the Exadata channels are full releases, so you will need to subscribe each month to the latest channels and use dnf reposync to mirror it. Take 

Since you now have an HTTP server running with the purpose of centralizing your Exadata software, you can also use this as a location for the storage servers to automatically update from. Read more about this here.

Resources

Oracle Linux Documentation - Managing Software in Oracle Linux

Setting up a local ULN Mirror

Updating Oracle Exadata Database Servers

Alex Blyth

Senior Principal Product Manager - Exadata

Alex Blyth is a Product Manager for Oracle Exadata with over 25 years of IT experience mainly focused on Oracle Database, Engineered Systems, manageability tools such as Enterprise Manager and most recently Cloud. Prior to joining the product management team, Alex was a member of the Australia/New Zealand Oracle Presales community and before that a customer of Oracle's at a Financial Services organisation.


Previous Post

Exadata System Software Updates - January 2024

Alex Blyth | 2 min read

Next Post


Exadata System Software Updates - February 2024

Alex Blyth | 2 min read