If you are an administrator of Oracle Solaris 11 systems you will very likely be aware that several of the critical installation and management tools are implemented in Python rather than the more traditional C and/or shell script. Python is often a good choice as a systems development language and Oracle Solaris is not alone in choosing to use it more extensively.
If you have done any programming with Python or have installed other Python tools you are probably aware that there are compatibility issues between the Python 2.x and 3.x language runtimes. In Oracle Solaris 11.4.7 we deliver multiple releases of Python: 2.7, 3.4, 3.5. Future updates to the Oracle Solaris support repository will change the exact version mix. The Python community will no longer be supporting and providing security or other fixes for the 2.7 environment from the year 2020. That is currently just about 9 months away.
A default installation of Oracle Solaris 11.4 will currently have at least Python 2.7 and a version of Python 3, /usr/bin/python by default will run Python 2.7. There are also paths that allow explicitly picking a version, this is very useful for the #! line of a script: eg /usr/bin/python2, /usr/bin/python3 or even more specific /usr/bin/python3.5. The packaging system allows you to choose now which version of Python /usr/bin/python represents; there are two package mediators used to control this. The 'python' mediator allows selecting which /usr/bin/python represents and the 'python3' mediator allows selecting which minor version of the 3.x release train the /usr/bin/python3 link points to. In a future support repository update we will change the default to be a version of Python 3.
The Oracle Solaris 11 package tools (IPS: Image Packaging System) is one of the critical OS tools implemented in Python, in current releases of 11.4 this is still using Python 2.7, but we have also delivered a parallel version for testing that uses Python 3.4. We don't expect to have a single support repository update where we switch over all of the OS Python consumers to Python 3 in one release. When we implement operating system tools using Python the #! line should always be precise as to which major.minor version it is the script is expected to run with. Our package dependencies then ensure the correct versions of pkg:/runtime/python are installed.
As a result of using Python to implement parts of the OS we deliver a fairly large number of Python modules from open source communities. When we deliver a Python module via IPS it is always installed below /usr/lib/pythonX.Y/vendor-packages/. The intent of the vendor-packages area is to be like the core Python site-packages area but to make it clear these modules came as part of the OS and should be updated with the OS tools. This provides some level of isolation between OS installed modules and administrator/developer use of tools such as 'pip' package installer for Python.
We have seen some recent cases where use of 'pip' to add additional modules, or different versions of modules delivered to vendor-packages has broken critical system administration tools such as /usr/bin/pkg and /usr/bin/beadm. The reason for this is because vendor-packages is really just an extension of the core Python site-packages concept. While it is possible to instruct the Python interpreter to ignore the user and site packages area for isolation reasons that doesn't help when the OS core tools are implemented in Python and need access to content installed in the vendor-packages area. Oracle Solaris is not the only operating system impacted by this problem, other OS vendors are developing their own solutions to this, some of the concepts and recommendations you will find are similar.
Firstly, like others do we strongly recommend the use of Python Virtual Environments when using pip or other Python install tools to add modules not delivered by Oracle Solaris packages. If you wish to leverage the fact that Oracle Solaris has already included some of the modules you need, you can use '--system-site-packages' when you create the virtual environment. Using that allows pip/python to reference the system site-packages and thus find the vendor-package are but it won't install content into /usr/lib/pythonX.y/site-packages. For example:
$ python3 -m venv --system-site-packages /usr/local/mytool $ source /usr/local/mytool/bin/activate (mytool) $ python Python 3.5.6 (default, Mar 7 2019, 08:31:32) [C] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-sunos5', '/usr/lib/python3.5/lib-dynload', '/usr/local/mytool/lib/python3.5/site-packages', '/usr/lib/python3.5/site-packages', '/usr/lib/python3.5/vendor-packages'] >>>
We can see from the above that the Python module search path includes the system and the 'mytool' application area as well as the Solaris IPS delivered content in vendor-packages. If we now run 'pip install' it will add any modules it needs to the site-packages directory below 'mytool' rather than the one under /usr/lib/python3.5.
So that we can keep providing open source Python modules for administrator/developer use and continue to use Python for the OS itself we need a solution that protects the Python modules in vendor-packages from being overridden by administrator installed ones in site-packages. We don't wish to duplicate whole Python environments, since as I mentioned above we currently use multiple Python versions for the OS itself. The solution we have come up was based on the following requirements:
We will be delivering parts of the solution to this over time when they are ready and tested to our satisfaction, there should be no impact to any of your tooling that uses Python we deliver.
The first change we will deliver is to further harden the core Python tools that we author against content in the 'site-packages' area that is potentially toxic to our tools. Ideally we should have been able to just pass -S as part of the #! line of the script, our tools already use -s to ignore the per user 'site-packages' but that would mean the vendor-packages area is never setup since it depends on /usr/lib/pythonX.Y/site-packages/vendor-packages.pth to get added. So instead we deliver a new module 'solaris.no_site_packages' that removes the system site-packages leaving vendor-packages. This module only makes sense for Oracle Solaris core components, in particular any 3rd party FOSS tools that are in Python do not include this.
We are still investigating the best the solution to ensure that tools such as pip can not uninstall IPS delivered content in vendor-packages. Patching the version of pip we deliver would only be a partial solution, since it would only work until an admin runs 'pip install --upgrade pip', since that would effectively remove any patch we made to ignore the vendor-packages area. Even with that restriction we may still apply a patch to pip.
We could potentially deliver an /etc/pip.conf file that forces the use of virtual environments, as described in this Python document virtual-env. However doing that could potentially have unexpected side effects if you are managing to safely use 'pip install' to place content into the system 'site-packages' area.
Instead we intend to use a feature of Oracle Solaris that allows us to tag files as not being able to be removed even by the root user (even when running with all privileges), a feature I described in this blog post 11 years ago! IPS can deliver system attributes (sysattr) for files already. Currently we only use that in a single package (pkg://solaris/release/name) to ensure that the version files that contain the value used for 'uname -v' are not easy to "accidentally" change, for that we add sysattr=readonly to the package manifest for those files. We are exploring the possibility of using the 'nounlink' or maybe even the 'immutable' attribute to protect all of the Python modules delivered by IPS to vendor-packages directory. In theory, we could do this for all content delivered via IPS that is not explicitly marked with the preserve attribute but we probably won't go that far in the initial use and instead focus just on hardening the Python environment.
In summary Python on Oracle Solaris is very much here to stay as /usr/bin/python and we are working to ensure that you can continue to use the Python modules we deliver while providing an environment where our core tools implemented in Python are safe from any additional Python modules added to the system.