At Oracle Solaris, while we help customers to move from old Oracle Solaris 10 software to its modern versions running on Oracle Solaris 11.4, we understand such a route may not be viable for all Oracle Solaris users. The reasons might be various – for example, the SW is in legacy mode and there is no newer version supporting Oracle Solaris 11.4, maybe the SW was built in house and its source code got lost in time and it cannot be rebuilt under Oracle Solaris 11.4, or the SW has been working fine as such and our customer does not want to invest time and effort to manually move the SW from Oracle Solaris 10 to 11.4, or possibly the admins that originally installed the SW no longer work for the company. However, if such customers still need or want to get away from Oracle Solaris 10 or solaris10 branded zones, we are now offering the sysdiff helper tool in our Oracle Solaris GitHub contrib section.
sysdiff is a Python script that analyzes an existing solaris10 branded zone, finds binaries, libraries, data and configuration files that are not part of Oracle Solaris 10 itself, and through a series of well defined steps, an IPS package is created. Such package may later be directly installed on a Oracle Solaris 11.4 system, ideally into a native or Kernel zone. After every sysdiff step, a manual intervention is possible, to adjust a list of files to include into a 3rd party app package.
While this is not always guaranteed to work, our tests show that there is good chance that it will work fine, thanks to Oracle Solaris binary guarantee. For more information on stability of Oracle Solaris interfaces, see stability(7) manual page.
Note that sysdiff was only uploaded to the Github Oracle contrib area and is not part of Oracle Solaris 11.4 yet. Whether it will be integrated into the Oracle Solaris system or not, largely depends on feedback we get. If we get no or limited feedback, the tool will probably stay in the contrib area.
First step – migrate your Oracle Solaris 10 installation to solaris10 zone
As sysdiff uses Oracle Solaris 11.4 Image Packing System pkg retrieval client, sysdiff itself must be run from within a Oracle Solaris 11.4 host. For that reason, if the 3rd party SW which you want to migrate from Oracle Solaris 10 to Oracle Solaris 11.4 runs on bare metal, you first need to migrate the system installation to a solaris10 branded zone.
To migrate a Oracle Solaris 10 system into a solaris10 branded zone, use the flar command.
With the flash archive file, you can easily install a solaris10 branded zone on Oracle Solaris 11.4. Make sure you have the system/zones/brand/brand-solaris10 package installed. For example:
solaris-114:~# pkg install system/zones/brand/brand-solaris10 # You will probably want to add more to the configuration as needed, e.g. # network settings, as the following only creates the default, almost empty, # configuration. solaris-114:~# zonecfg -z s10zone create -t SYSsolaris10 solaris-114:~# zoneadm -z s10zone install -u -a archive.flar solaris-114:~# zoneadm -z s10zone boot # Now go through sysconfig UI on the console to finish the system configuration. solaris-114:~# zlogin -C s10zone
You do not really need to boot and finish the configuration as sysdiff needs the zone installed, not running, since it only works with the zone’s ZFS datasets. However, you may want to boot the system and verify the 3rd part SW works as expected. If it does not, you should solve that problem first as the migrated SW would probably not work on Oracle Solaris 11.4 either.
Getting some 3rd party software to verify the app migration process
We do not really have any legacy SW running on Oracle Solaris 10 around so I will manually build and install OpenSSH 9.0 on Oracle Solaris 10 to emulate working software running on my solaris10 branded zone. I configured it with --without-openssl. The reason is that sysdiff will not package dependencies that are part of Oracle Solaris itself due to security concerns. For that reason, if your 3rd software is linked against OpenSSL 0.9.7d shipped with Oracle Solaris 10 (in Oracle Solaris 11.4, we presently support 1.0.2, 1.1, and 3.0 but not 0.9.*), you would have to manually add those library files into the manifest after the diff phase. However, if you do not do that, the depresolve phase will fail and identify files that are missing from the manifest. You will only need to update the manifest and start again from the depresolve phase.
Now, I verify that the compiled OpenSSH daemon works fine from within the solaris10 branded zone. I manually start it in the debug mode:
root@/root/openssh-9.0p1# /usr/local/sbin/sshd -p 2222 -ddd debug2: load_server_config: filename /usr/local/etc/sshd_config debug2: load_server_config: done config len = 3181 debug2: parse_server_config_depth: config /usr/local/etc/sshd_config len 3181 debug3: /usr/local/etc/sshd_config:41 setting AuthorizedKeysFile .ssh/authorized_keys ...
And log in from another machine using the command mode:
another:~$ ssh -p 2222 test@s10zone uname -a test@s10zone's password: Environment: <redacted> SunOS s10zone 5.10 Generic_Virtual i86pc i386 i86pc
With this working, we are now ready for the app migration from Oracle Solaris 10 to Oracle Solaris 11.4.
Get sysdiff
You need the whole directory with sysdiff as it also contains data files necessary for it to work, and cloning the whole Oracle Solaris contribution area is the easiest way:
git clone https://github.com/oracle/oraclesolaris-contrib.git Cloning into 'oraclesolaris-contrib'... ...
Run the diff phase
Halt the s10zone zone if it is running. sysdiff would fail on running zones anyway. You need to specify the new package name, the zone name, and the results directory parent where sysdiff stores data.
root@myhost:~# zoneadm -z s10zone halt root@myhost:~# root@myhost:~# /data/oraclesolaris-contrib/OracleSolaris_sysdiff/sysdiff.py \ diff --pkg-name s10-openssh-9.0 --results-dir-parent /data/sysdiff-outputdir \ s10zone Loading excluded dirs list from file: /data/oraclesolaris-contrib/OracleSolaris_sysdiff/excludeDirs passwd entries: 19 group entries: 22 known new files loaded: 712 known mod files loaded: 63 pkgs loaded: 1889 contents entries: 191343 no non-Solaris pkgs found Skipping dir |/system| Skipping dir |/tmp| Skipping dir |/mnt| Skipping dir |/opt/ocm| Skipping dir |/var/adm| Skipping dir |/var/tmp| Skipping dir |/var/crash| Skipping dir |/var/log| Skipping dir |/var/svc| Skipping dir |/var/cache| Skipping dir |/var/spool| Skipping dir |/var/fm| Skipping dir |/var/sadm| Skipping dir |/var/ocm| Skipping dir |/.SUNWnative| Skipping dir |/dev| Skipping dir |/rpool| Skipping dir |/boot| Skipping dir |/etc/svc| Skipping dir |/net| Skipping dir |/proc| Skipping dir |/root| Skipping dir |/devices| new_files: 649 matched_files: 182638 mod_files: 461 mod_vol_files: 74 Writing /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m Writing /data/sysdiff-outputdir/s10zone-20220425171714/mod_files.p5m Writing /data/sysdiff-outputdir/s10zone-20220425171714/mod_vol_files.p5m Writing /data/sysdiff-outputdir/s10zone-20220425171714/matched_files.p5m IPS package manifest generated: /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m Please review and adjust it as needed before running the next step. Please provide the following directory as results-dir for the next steps: /data/sysdiff-outputdir/s10zone-20220425171714 The next step should be 'sysdiff depresolve'
Note the output dir /data/sysdiff-outputdir/s10zone-20220425171714. It contains the state of the whole SW migration and you need to pass that directory name to all subsequent phases.
Now it is the right time to go through the package manifest /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m which was part of the above output and manually remove files and directories that obviously have nothing to do with your 3rd party SW. In my case it was quite easy, I removed anything not having the prefix /usr/local as I installed nothing into that directory other than OpenSSH. Save the modified manifest file and continue with the next phase as suggested above by the diff subcommand output. You can also add whatever you want to the manifest, of course.
Run the depresolve phase
This phase will resolve dependencies. If there is anything missing in the manifest that the diff phase did not find or you manually removed but is a required dependency for anything else in the manifest, you will be notified during this phase.
root@myhost:~# /data/oraclesolaris-contrib/OracleSolaris_sysdiff/sysdiff.py \ depresolve /data/sysdiff-outputdir/s10zone-20220425171714 running command: pkgdepend generate -md /system/zones/s10zone/root /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m running command: pkgdepend resolve -m /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m.dep 'sysdiff depresolve' successful, next step is 'sysdiff lint'
Run the lint phase
Note that you have to provide access to the Oracle Solaris pkg publisher to run this phase. It just runs pkglint to perform series of checks on the manifest.
Note that you need to provide an existing 11.4 repository via the --s11.4-repo repo-link option.
root@myhost:~# /data/oraclesolaris-contrib/OracleSolaris_sysdiff/sysdiff.py \ lint --s11.4-repo https://pkg.oracle.com/solaris/support/ \ /data/sysdiff-outputdir/s10zone-20220425171714 running command: pkglint -c /data/sysdiff-outputdir/s10zone-20220425171714.cache -r https://pkg.oracle.com/solaris/support/ /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m.dep.res 'sysdiff lint' successful, next step is 'sysdiff publish'
Run the publish phase
Now we will create and publish the package into our target repository (see --target-repo option). If you do not have the repository created yet, use the -c --publisher publisher-name options. With the option to create the repository, give it an empty directory to start with.
root@myhost:~# zfs create rpool/data/sysdiff-repo root@myhost:~# root@myhost:~# /data/oraclesolaris-contrib/OracleSolaris_sysdiff/sysdiff.py \ publish -c --publisher sysdiff-pub --target-repo /data/sysdiff-repo \ /data/sysdiff-outputdir/s10zone-20220425171714 running command: pkgrepo create /data/sysdiff-repo running command: pkgrepo -s /data/sysdiff-repo set publisher/prefix=sysdiff-pub running command: pkgsend -s /data/sysdiff-repo publish -d /system/zones/s10zone/root /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5m.dep.res 'sysdiff publish' successful, the package can be converted to an archive using 'sysdiff archive'
Optionally, create a p5p archive
I will work directly with the repository created above but you can create a p5p archive if you want to:
root@myhost:~# /data/oraclesolaris-contrib/OracleSolaris_sysdiff/sysdiff.py \ archive --source-repo /data/sysdiff-repo \ /data/sysdiff-outputdir/s10zone-20220425171714 running command: pkgrecv -s /data/sysdiff-repo -a -d /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5p s10-openssh-9.0@1.0 /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5p created successfully root@myhost:~# root@myhost:~# pkg list -g /data/sysdiff-outputdir/s10zone-20220425171714/s10-openssh-9.0.p5p NAME (PUBLISHER) VERSION IFO s10-openssh-9.0 (sysdiff-pub) 1.0 ---
Verify the repository contains the new package
root@myhost:~# pkg list -ng /data/sysdiff-repo NAME (PUBLISHER) VERSION IFO s10-openssh-9.0 (sysdiff-pub) 1.0 ---
Install the package onto 11.4 and test it
On modern Oracle Solaris, any application software should run in a zone. For this test, we will install the created package into a native zone, see solaris(7) for more information. I will use tzone-114 I already have around on the same host where we have the s10zone zone.
As we have both zones on the same host, we can set the newly created publisher on the host itself, and tzone-114 can use it for free via the always running package zone proxy. Note that when using the proxy, the zone itself does not need to be networked at all as it accesses the repository through the host.
Set the publisher on the 11.4 host:
root@myhost:~# pkg set-publisher -p file:///data/sysdiff-repo pkg set-publisher: Added publisher(s): sysdiff-pub
Verify the publisher is available from within the native zone (called the syspub type of publisher):
root@tzone-114:~# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris (non-sticky, syspub) origin online T sysdiff-pub (syspub) origin online F
Check the created package is present from within the 11.4 zone, and install it there:
root@tzone-114:~# pkg list -af s10-openssh-*
NAME (PUBLISHER) VERSION IFO
s10-openssh-9.0 (sysdiff-pub) 1.0 ---
root@tzone-114:~#
root@tzone-114:~# pkg install s10-openssh-9.0
Packages to install: 1
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 1/1 32/32 1.9/1.9 32.6M/s
PHASE ITEMS
Installing new actions 48/48
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done
Updating package cache 2/2
Now I started the daemon from the just installed package on the 11.4 native zone in debug mode:
root@tzone-114:~# /usr/local/sbin/sshd -d -p 2222 debug1: sshd version OpenSSH_9.0, without OpenSSL debug1: Unable to load host key: /usr/local/etc/ssh_host_rsa_key debug1: private host key #1: ssh-ed25519 SHA256:bEZNXX5AmEF1aXxI3eLclMzlAKA60+yH7/p6UbU6+uM debug1: rexec_argv[0]='/usr/local/sbin/sshd' ...
And logged in from any other machine again, but now I’m logging into a 11.4 zone with the installed SW built and migrated from the solaris10 branded zone:
another:~$ ssh -p 2222 test@tzone-114.oracle.com uname -a test@tzone-114.oracle.com's password: Environment: <redacted> SunOS tzone-114.oracle.com 5.11 11.4.45.119.0 i86pc i386 i86pc non-global-zone
That concluded that the app migration worked (I only tested the daemon though).
What if migrated SW does not work on Oracle Solaris 11.4?
Based on what “it does not work” really means, it is up to the admin to debug the problem. However, if there are obvious missing files not moved over from the solaris10 branded zone to the package, go back to update the manifest, then continue again from the depresolve phase. You can repeat that as many times you want to.
Remove the -c option for the publish phase if you start iterating. Also note that running this phase will always create the package with a new timestamp. sysdiff never deletes older package versions. For example, after I run publish two more times, I get the following, with the already installed package in the 11.4 zone being the oldest one:
root@myhost:~# pkg list -g /data/sysdiff-repo -v -af FMRI IFO pkg://sysdiff-pub/s10-openssh-9.0@1.0:20220426T110838Z --- pkg://sysdiff-pub/s10-openssh-9.0@1.0:20220426T110556Z --- pkg://sysdiff-pub/s10-openssh-9.0@1.0:20220426T084032Z --- root@tzone-114:~# pkg list -v s10-openssh-9.0 FMRI IFO pkg://sysdiff-pub/s10-openssh-9.0@1.0:20220426T084032Z i--
As expected, you can update the package after every iteration:
root@tzone-114:~# pkg update pkg://sysdiff-pub/s10-openssh-9.0
Packages to update: 1
Create boot environment: No
Create backup boot environment: Yes
PHASE ITEMS
Updating modified actions 1/1
Updating package state database Done
Updating package cache 1/1
Updating image state Done
Creating fast lookup database Done
Updating package cache 2/2
root@tzone-114:~#
root@tzone-114:~# pkg list -v pkg://sysdiff-pub/s10-openssh-9.0
FMRI IFO
pkg://sysdiff-pub/s10-openssh-9.0@1.0:20220426T110838Z i--
Conclusion
The sysdiff tool is not part of Oracle Solaris 11.4 yet. We made it available through the Oracle Solaris GitHub contrib area to see if our customers find it useful. If you do, please let us know through the GitHub Issues option, as if there are enough customers happy with sysdiff, we can integrate it into a future Oracle Solaris 11.4 SRU. Even if you are not a customer with existing Oracle Solaris 11.4 support, you can test your Oracle Solaris 10 app migration first using the non-commercial Oracle Solaris CBE release to see if it worked for you.
It is also absolutely essential that both the source solaris10 branded zone and the final version of the results directory are preserved for the lifetime of the transitioned application. The reason is that the generated package may have dependencies on packages which might get obsoleted and removed from Oracle Solaris 11.4, and the package would have to be regenerated based on these resources. File executionExample contains sample of such an issue and steps needed to resolve it.