X

News, tips, partners, and perspectives for the Oracle Solaris operating system

Building Opensource on Solaris 11.4

Chris Beal
Senior Principal Software Engineer

I love opensource software, I use it every day, but sometimes a component I need is either not present in or at the wrong version in Oracle Solaris. We often hear the same requests from our customers too. Please can I have version x.y of opensource software FOO on Oracle Solaris 11.4. Oracle Solaris depends heavily on opensource software and we try to keep them reasonably up to date, while being careful not to break anything. However we don't have every peice of opensource available, so I thought I'd address this by writing a blog to show how simple it was to use our Userland build infrastructure to build a piece of opensource I actually wanted to use this week.

So what's the software? Well I use cscope a fair bit. Internally we have two versions cscope and cscope-fast. The former came to us when we licensed AT&T System V Unix, and we have evolved it ever since. The latter is a fork to optimise search speed (at the expense of it being slower to generate the index file). cscope-fast isn't that good at doing python either, which I am doing a lot more of these days. I noticed that in 2001 SCO, who bought the rights to AT&T System V unix, had opensourced cscope under the BSD license, so thought it was worth seeing how it compared, both in terms of speed and how it handles python.

It might now be worth a brief discussion about what "userland" actually is and how it works.

Userland is our way of importing and building opensource software in a simple repeatable way. It does so by downloading the source of a project (either as a tar ball, or by using the source code management system to take a copy), it then applies any patches we need to get it to work on Oracle Solaris (thankfully fewer and fewer these days) and then builds it. It can even generate IPS packages if you create a package manifest.

I've done everything in 11.4 GA (General Availability) using the Oracle Technical Network (OTN) versions of Oracle Developer Studio and Oracle Solaris, but it you have a support contract obviously you should use the latest versions you can.

I went to OTN and registered for the developer studio package repository (I won't cover that here, it's pretty straight forward and documented on the OTN site). You need to register to get certificates and keys to access it.

So I booted up my 11.4 GA vm and installed the following packages to get me started

$ sudo pkg install flex git gcc-7

I know I'm going to need gcc and git, I only found out later I needed flex to build cscope.

I decided to install Developer Studio 12.6 (The OTN version of Studio 12.4 doesn't install on 11.4 as 11.4 removed the Python2.6 interpreter, you can get a patched version of Studio 12.4 if you really need to)

$ sudo pkg install --accept developerstudio-126

We need studio to do the setup for the userland build infrastructure, we won't be using it to build cscope.

OK, so now we're ready to start. First off create a directory where you want to do the work

$ git clone https://github.com/oracle/solaris-userland.git Cloning into 'solaris-userland'... remote: Enumerating objects: 1251, done. remote: Counting objects: 100% (1251/1251), done. remote: Compressing objects: 100% (683/683), done. remote: Total 91074 (delta 677), reused 935 (delta 530), pack-reused 89823 Receiving objects: 100% (91074/91074), 137.57 MiB | 8.05 MiB/s, done. Resolving deltas: 100% (34629/34629), done.

Now we have a local copy to work with

Next we need to setup the build environment to work. Because we're not in Oracle and using Studio 12.6 we need to change a couple of things

in make-rules/shared-macros.mk, find the SPRO_VROOT definition and change it to

SPRO_VROOT ?= $(SPRO_ROOT)/developerstudio12.6

and remove the line starting INTERNAL_ARCHIVE_MIRROR=

and in make-rules/ips-buildinfo.mk find CANONICAL_REPO line (it's currently the last one) and change it to point to the Oracle Solaris release repo

CANONICAL_REPO ?= http://pkg.oracle.com/solaris/release/

Now we can setup the build environment

$ gmake setup /bin/mkdir -p /export/home/chris/userland-src/solaris-userland/i386 Generating component list... Generating component dependencies... Generating component list... Generating component dependencies... /bin/mkdir -p /export/home/chris/userland-src/solaris-userland/i386/logs /bin/mkdir -p /export/home/chris/userland-src/solaris-userland/i386/home /usr/bin/pkgrepo create file:/export/home/chris/userland-src/solaris-userland/i386/repo /usr/bin/pkgrepo add-publisher -s file:/export/home/chris/userland-src/solaris-userland/i386/repo nightly /usr/bin/pkgrepo add-publisher -s file:/export/home/chris/userland-src/solaris-userland/i386/repo userland-localizable /usr/bin/pkgrepo create file:/export/home/chris/userland-src/solaris-userland/i386/repo.experimental /usr/bin/pkgrepo add-publisher -s file:/export/home/chris/userland-src/solaris-userland/i386/repo.experimental nightly /usr/bin/pkgrepo add-publisher -s file:/export/home/chris/userland-src/solaris-userland/i386/repo.experimental userland-localizable building tools... /usr/gnu/bin/make -C ../tools clean make[1]: Entering directory '/export/home/chris/userland-src/solaris-userland/tools' make[1]: Leaving directory '/export/home/chris/userland-src/solaris-userland/tools' /usr/gnu/bin/make -C ../tools setup make[1]: Entering directory '/export/home/chris/userland-src/solaris-userland/tools' make[1]: Leaving directory '/export/home/chris/userland-src/solaris-userland/tools' Generating pkglint(1) cache from CANONICAL_REPO http://pkg.oracle.com/solaris/release/...

The last bit takes a while.

Now create a directory for cscope

$ mkdir components/cscope $ cd components/cscope

We need to create a Makefile. This is what mine looks like

$ cat Makefile # # CDDL HEADER START # # The contents of this file are subject to the terms of the # Common Development and Distribution License (the "License"). # You may not use this file except in compliance with the License. # # You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE # or http://www.opensolaris.org/os/licensing. # See the License for the specific language governing permissions # and limitations under the License. # # When distributing Covered Code, include this CDDL HEADER in each # file and include the License file at usr/src/OPENSOLARIS.LICENSE. # If applicable, add the following below this CDDL HEADER, with the # fields enclosed by brackets "[]" replaced with your own identifying # information: Portions Copyright [yyyy] [name of copyright owner] # # CDDL HEADER END # # # Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. # BUILD_BITS= 64 COMPILER=gcc include ../../make-rules/shared-macros.mk COMPONENT_NAME= cscope COMPONENT_VERSION= 15.9 COMPONENT_SRC= $(COMPONENT_NAME)-$(COMPONENT_VERSION) IPS_COMPONENT_VERSION= $(COMPONENT_VERSION) BUILD_VERSION= 1 COMPONENT_PROJECT_URL= https://cscope.sourceforge.net #COMPONENT_BUGDB= utility/cscope #COMPONENT_ANITYA_ID= 1865 COMPONENT_ARCHIVE= cscope-15.9.tar.gz COMPONENT_ARCHIVE_URL= https://sourceforge.net/projects/cscope/files/latest/download COMPONENT_ARCHIVE_HASH= COMPONENT_MAKE_JOBS= 1 BUILD_STYLE= configure TEST_TARGET= $(NO_TESTS) include $(WS_MAKE_RULES)/common.mk REQUIRED_PACKAGES += library/ncurses

This is about as small as I could make it, You need the location where to download the tarball, package and component names, and I only found later I needed to put a dependency on library/ncurses to get packaging to resolve the dependency correctly. Later on you will want to populate the COMPONENT_ARCHIVE_HASH to be sure you're getting the file you intended.

BUILD_STYLE tells the makefiles to run configure within the build directory

We will need to create a package manifest, but if we wait till we're building cleanly we can use the gmake sample-manifest to get something close to usable

So assuming I've got the Makefile right should be able to build it

$ gmake install

<stuff deleted>

This is gmake getting the source for us

/export/home/chris/userland-src/solaris-userland/tools/userland-fetch --file cscope-15.9.tar.gz --url 'https://sourceforge.net/projects/cscope/files/latest/download' Source cscope-15.9.tar.gz... not found, skipping file copy Source https://sourceforge.net/projects/cscope/files/latest/download... downloading... validating signature... skipped (no signature URL) validating hash... skipping (no hash) hash is: sha256:c5505ae075a871a9cd8d9801859b0ff1c09782075df281c72c23e72115d9f159 /usr/bin/touch cscope-15.9.tar.gz

<More stuff deleted>

This is it running configure (and where I found I needed flex installed)

(cd /export/home/chris/userland-src/solaris-userland/components/cscope/build/amd64 ; /usr/bin/env CONFIG_SHELL="/bin/bash" PKG_CONFIG_PATH="/usr/lib/amd64/pkgconfig" CC="/usr/gcc/7/bin/gcc" CXX="/usr/gcc/7/bin/g++" PATH="/usr/bin/amd64:/usr/bin:/usr/gnu/bin" CC_FOR_BUILD="/usr/gcc/7/bin/gcc -m64" CXX_FOR_BUILD="/usr/gcc/7/bin/g++ -m64" CPPFLAGS="-m64" "ac_cv_func_realloc_0_nonnull=yes" "NM=/usr/gnu/bin/nm" INTLTOOL_PERL="/usr/perl5/5.22/bin/perl" CFLAGS="-m64 -O3" CXXFLAGS="-m64 -O3" LDFLAGS="" /bin/bash \ /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/configure --prefix=/usr --mandir=/usr/share/man --bindir=/usr/bin --sbindir=/usr/sbin --libdir=/usr/lib/amd64 ) checking for a BSD-compatible install... /usr/bin/ginstall -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/bin/gmkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking build system type... x86_64-pc-solaris2.11 checking host system type... x86_64-pc-solaris2.11 checking for style of include used by make... GNU checking for gcc... /usr/gcc/7/bin/gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables...

<More stuff deleted>


Here it's starting gmake install in the build area

make[4]: Entering directory '/export/home/chris/userland-src/solaris-userland/components/cscope/build/amd64/src' /usr/gcc/7/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src -I.. -I/usr/include/ncurses -m64 -m64 -O3 -MT fscanner.o -MD -MP -MF .deps/fscanner.Tpo -c -o fscanner.o /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src/fscanner.c mv -f .deps/fscanner.Tpo .deps/fscanner.Po /usr/gcc/7/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src -I.. -I/usr/include/ncurses -m64 -m64 -O3 -MT egrep.o -MD -MP -MF .deps/egrep.Tpo -c -o egrep.o /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src/egrep.c mv -f .deps/egrep.Tpo .deps/egrep.Po /usr/gcc/7/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src -I.. -I/usr/include/ncurses -m64 -m64 -O3 -MT alloc.o -MD -MP -MF .deps/alloc.Tpo -c -o alloc.o /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src/alloc.c mv -f .deps/alloc.Tpo .deps/alloc.Po /usr/gcc/7/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src -I.. -I/usr/include/ncurses -m64 -m64 -O3 -MT basename.o -MD -MP -MF .deps/basename.Tpo -c -o basename.o /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src/basename.c mv -f .deps/basename.Tpo .deps/basename.Po /usr/gcc/7/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src -I.. -I/usr/include/ncurses -m64 -m64 -O3 -MT build.o -MD -MP -MF .deps/build.Tpo -c -o build.o /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src/build.c mv -f .deps/build.Tpo .deps/build.Po /usr/gcc/7/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src -I.. -I/usr/include/ncurses -m64 -m64 -O3 -MT command.o -MD -MP -MF .deps/command.Tpo -c -o command.o /export/home/chris/userland-src/solaris-userland/components/cscope/cscope-15.9/src/command.c mv -f .deps/command.Tpo .deps/command.Po

Which ends up creating a prototype directory under $(BUILD) to mirror the directory structure of the installed package. This is very useful as pkg(7) can use this to generate a manifest for us.

So at this point we can run cscope from the prototype area, but that's not really very interesting, we what IPS packages to install. Well fortunately now we have the proto area, we can run the gmake sample-manifest

$ gmake sample-manifest /export/home/chris/userland-src/solaris-userland/tools/manifest-generate \ /export/home/chris/userland-src/solaris-userland/components/cscope/build/prototype/i386 | \ /usr/bin/pkgmogrify -D PERL_ARCH=i86pc-solaris-thread-multi-64 -D PERL_VERSION=5.22 -D IPS_COMPONENT_RE_VERSION=15\\.9 -D COMPONENT_RE_VERSION=15\\.9 -D PYTHON_2.7_ONLY=# -D PYTHON_3.4_ONLY=# -D PYTHON_3.5_ONLY=# -D SQ=\' -D DQ=\" -D Q=\" -I/export/home/chris/userland-src/solaris-userland/components/cscope -D SOLARIS_11_3_ONLY="#" -D SOLARIS_11_5_ONLY="#" -D SOLARIS_11_3_4_ONLY="" -D SOLARIS_11_4_5_ONLY="" -D SOLARIS_11_4_ONLY="" -D PY3_ABI3_NAMING="#" -D PY3_CYTHON_NAMING="" -D ARC_CASE="" -D TPNO="" -D BUILD_VERSION="1" -D OS_RELEASE="5.11" -D SOLARIS_VERSION="2.11" -D PKG_SOLARIS_VERSION="11.5" -D CONSOLIDATION="userland" -D CONSOLIDATION_CHANGESET="" -D CONSOLIDATION_REPOSITORY_URL="https://github.com/oracle/solaris-userland.git" -D COMPONENT_VERSION="15.9" -D IPS_COMPONENT_VERSION="15.9" -D HUMAN_VERSION="" -D COMPONENT_ARCHIVE_URL="https://sourceforge.net/projects/cscope/files/latest/download" -D COMPONENT_PROJECT_URL="https://cscope.sourceforge.net" -D COMPONENT_NAME="cscope" -D HG_REPO="" -D HG_REV="" -D HG_URL="" -D GIT_COMMIT_ID="" -D GIT_REPO="" -D GIT_TAG="" -D MACH="i386" -D MACH32="i86" -D MACH64="amd64" -D PUBLISHER="nightly" -D PUBLISHER_LOCALIZABLE="userland-localizable" /dev/fd/0 /export/home/chris/userland-src/solaris-userland/transforms/generate-cleanup | \ sed -e '/^$/d' -e '/^#.*$/d' | /usr/bin/pkgfmt | \ cat /export/home/chris/userland-src/solaris-userland/transforms/manifest-metadata-template - >/export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-generated.p5m

So we have a manifest in /home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-generated.p5m. It's not useable but is close to it. For me it was a bit hit and miss adding and removing things till it worked. The diffs I ended up with are

 

$ diff -u /export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-generated.p5m cscope.p5m --- /export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-generated.p5m 2019-03-28 16:51:42.034736910 +0000 +++ cscope.p5m 2019-03-28 15:58:57.122806976 +0000 @@ -23,26 +23,17 @@ # Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. # + default mangler.man.stability volatile> set name=pkg.fmri \ - value=pkg:/$(IPS_PKG_NAME)@$(IPS_COMPONENT_VERSION),$(BUILD_VERSION) -set name=pkg.summary value="XXX Summary XXX" -set name=com.oracle.info.description value="XXX Description XXX" -set name=com.oracle.info.tpno value=$(TPNO) + value=pkg:/site/developer/cscope@$(IPS_COMPONENT_VERSION),$(BUILD_VERSION) +set name=pkg.summary value=cscope +set name=com.oracle.info.description value="Source Browser" set name=info.classification \ - value="org.opensolaris.category.2008:XXX Classification XXX" + value=org.opensolaris.category.2008:Development/System set name=info.source-url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream-url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.arc-caseid value=PSARC/YYYY/XXX -set name=org.opensolaris.consolidation value=$(CONSOLIDATION) - -license $(COPYRIGHT_FILE) license='$(COPYRIGHTS)' - file path=usr/bin/cscope file path=usr/bin/ocs file path=usr/share/man/man1/cscope.1 +license BSD license=BSD

So I set the category, package name, summary, description and license manually. I also add a transform for the man page. This is required.

A quick point about package names, I've put it in a hierachy that Oracle will never publish packages in (site).

We need to also consider the license. You can't delete it, so I copied the license from the cscope home page in to a file called BSD in the cscope component directory.

Remove generated package manifest (else it'll try to use that instead) and run pkgfmt on your manifest to make it acceptable to pkg and simply

$ gmake publish /usr/bin/env; /usr/bin/pkgdepend generate \ -m -d /export/home/chris/userland-src/solaris-userland/components/cscope/build/prototype/i386/mangled -d /export/home/chris/userland-src/solaris-userland/components/cscope/build/prototype/i386 -d /export/home/chris/userland-src/solaris-userland/components/cscope/build -d /export/home/chris/userland-src/solaris-userland/components/cscope -d cscope-15.9 -d /export/home/chris/userland-src/solaris-userland/licenses /export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-cscope.mangled >/export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-cscope.depend /usr/bin/pkgdepend resolve -e /export/home/chris/userland-src/solaris-userland/components/cscope/build/resolve.deps -m /export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-cscope.depend /usr/bin/touch /export/home/chris/userland-src/solaris-userland/components/cscope/build/.resolved-i386

<More stuff deleted>

/usr/bin/pkgsend -s file:/export/home/chris/userland-src/solaris-userland/i386/repo publish --fmri-in-manifest --no-catalog -d /export/home/chris/userland-src/solaris-userland/components/cscope/build/prototype/i386/mangled -d /export/home/chris/userland-src/solaris-userland/components/cscope/build/prototype/i386 -d /export/home/chris/userland-src/solaris-userland/components/cscope/build -d /export/home/chris/userland-src/solaris-userland/components/cscope -d cscope-15.9 -d /export/home/chris/userland-src/solaris-userland/licenses -T \*.py /export/home/chris/userland-src/solaris-userland/components/cscope/build/manifest-i386-cscope.depend.res pkg://nightly/developer/cscope@15.9,1:20190328T160907Z PUBLISHED

<More stuff deleted>

Right we have packages in a repository, we can install it

$ sudo pkg set-publisher -p file:/export/home/chris/userland-src/solaris-userland/i386/repo pkg set-publisher: Added publisher(s): nightly Updated publisher(s): userland-localizable $ sudo pkg install cscope Packages to install: Create boot environment: No Create backup boot environment: No DOWNLOAD PKGS FILES XFER (MB) SPEED Completed 1/1 4/4 0.2/0.2 -- PHASE ITEMS Installing new actions 18/18 Updating package state database Done Updating package cache 0/0 Updating image state Done Creating fast lookup database Done Updating package cache 4/4

So lets run it on cscope and see what it looks like

$ cd cscope-15.9/src $ cscope -R

 
So there you have it, cscope, cscoping cscope source
 

Join the discussion

Comments ( 1 )
  • David Stes Tuesday, June 18, 2019
    It's an interesting article, although I'm not convinced that using "userland" is much of an improvement over manually creating a .p5m manifest file and creating the IPS package manually. However if you create many packages, "userland" is probably a big win. Also I wonder what the "consolidation" parameters are, from the user point of view there is also the consolidation/userland/userland-incorporation package , a "consolidation" of the packages built with "userland", I'm not sure I fully understand what the role of the consolidation is for example for "cscope", I assume that this consolidates "cscope" with the other libraries and tools that are 'required' for this package ... Anyway thanks for this interesting article.
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.