Local Repository Mirror

In my previous entry I wrote about creating a portable version of the repository using a USB stick. That option is great for taking the repository on the road and even sharing it with folks at conferences where the Internet connection usually sucks.

However, a more practical use of this repository image to many of you may be setting up a local mirror. In this situation, you get the best of both words - fast access to the repository with continued access to updates as they become available.

Setting Up the Local Repository

We're going to do this in a generic fashion so that upgrading to a newer version of the repository is simply a matter of downloading a new ISO.

If you haven't already, download the ISO image of the repository (about 7 GB). Once the download completes, rename the ISO to something more generic:

bleonard@opensolaris:~$ mv Download/osol-repo-0906-full.iso Download/osol-repo-full.iso 

Create a directory at which to mount the ISO:

 bleonard@opensolaris:~$ sudo mkdir /repo

Mount the ISO to the /repo directory:

sudo mount -F hsfs `sudo lofiadm -a ~/Download/osol-repo-full.iso` /repo
Configure the package server to use the local repository:
svccfg -s application/pkg/server setprop pkg/inst_root=/repo/repo
svccfg -s application/pkg/server setprop pkg/readonly=true

You can also choose a port number different from the default of 80, which I did because I'm running Apache on port 80:

svccfg -s application/pkg/server setprop pkg/port=81

Copy the repo/cfg_cache file from the ISO mount to another location on the local file system so it can be edited:

sudo cp /repo/repo/cfg_cache /etc/repo_cfg_cache

Edit the etc/repo_cfg_cache file, changing the origins property from http://pkg.opensolaris.org/release  to http://<domainname>:<port>, where <domainname> is network accessible and <port> is needed if you used something other than the default of 80. For example:

origins = http://opensolaris:81

Configure the package server service to use the local configuration file:

svccfg -s application/pkg/server setprop pkg/cfg_file=/etc/repo_cfg_cache 

Then refresh the package server service to pick up the configuration changes and start it:

svcadm refresh application/pkg/server
svcadm enable application/pkg/server

Making the Mount Permanent

The Quick and Dirty Way

Add the following file to /etc/rc3.d:

# cat /etc/rc3.d/S99mountiso
#!/bin/sh
#
mount -F hsfs `lofiadm -a /export/home/bleonard/Download/osol-repo-full.iso` /repo

The SMF Way

Save the following repo.xml SMF manifest to /var/svc/manifest/system/filesystem., which defines a new service, repo.

Install the repo service:

bleonard@opensolaris:~$ svccfg import /var/svc/manifest/system/filesystem/repo.xml

Start the repo service:

bleonard@opensolaris:~$ svcadm enable repo 
bleonard@opensolaris:~$ svcs -l repo
fmri         svc:/system/filesystem/repo:default
name         Solaris ISO repository mounter
enabled      true
state        online
next_state   none
state_time   Mon Oct 11 09:40:07 2010
logfile      /var/svc/log/system-filesystem-repo:default.log
restarter    svc:/system/svc/restarter:default
dependency   require_all/none svc:/system/filesystem/local (online)

Accessing the Local Repository

Now all those wishing to use the local package server simply need to make the following change:

sudo pkg set-publisher -m http://opensolaris:81 opensolaris.org

Which will result in the following:

bleonard@opensolaris:~$ pkg publisher
PUBLISHER                             TYPE     STATUS   URI
opensolaris.org          (preferred)  origin   online   http://pkg.opensolaris.org/release/
opensolaris.org          (preferred)  mirror   online   http://opensolaris:81/

If you only want to use the local repository, first remove the mirror:

sudo pkg set-publisher -M http://opensolaris:81 opensolaris.org 

Then:

sudo pkg set-publisher -O http://opensolaris:81 opensolaris.org
For more information see the README.
Comments:

Hello Brian,

thank you for this very useful post, it helped me set up my repository. However I think there is a mistake when configuring the location of the repository:

svccfg -s application/pkg/server pkg/inst_rool=/rpool/repo/repo

Should read

svccfg -s application/pkg/server setprop pkg/inst_root=/rpool/repo/repo

It should use "setprop" and the property is pkg/inst_root not inst_rool !

Regards

Steve

Posted by Stephen Myles on August 13, 2009 at 05:28 AM GMT #

Hi Stephen,

Thanks for the correction. I have a general rule to copy and paste directly from the terminal - which I obviously broke this time. Should be all set now.

/Brian

Posted by Brian Leonard on August 13, 2009 at 08:57 AM GMT #

Hi Brian,

Looks much better.

Regards

Steve

Posted by Stephen Myles on August 17, 2009 at 03:50 AM GMT #

Hi,
It took me only 39 minutes to copy the entire FS on a MacBook Pro.

Cheers

Posted by KBerrah on September 08, 2009 at 08:47 AM GMT #

Hi,

Good job, but how do you update your local repository from version 111 to latest (currently 122) ?

Thanks

Posted by Christophe Pauliat on September 10, 2009 at 05:49 AM GMT #

Hi Christophe,

I believe only the 2009.06 release repository has been made available in this way. Regardless, I don't think it would be possible to update the local repository. I think would you have to download and copy it again - which would be a time consuming endeavor every 2 weeks.

I would try the pkg-discuss e-mail alias for a more thorough answer.

Regards,
Brian

Posted by Brian Leonard on September 10, 2009 at 06:00 AM GMT #

Hi Brian,

It worked fine ! excellent to speed up download when installing big packages (amp-dev, sunstudio, openoffice).

Two things:
- the default port is 10000, not 80
- the service refused to start until I added the hostname in /etc/hosts

With this post, the Switzerland OpenSolaris User group could setup his own IPS mirror. So, thanks again for you blog !

Karim

Posted by KBerrah on September 11, 2009 at 06:29 AM GMT #

Hi Karim,

Great news. With regards to your notes, if I run the following on a fresh installation:

bleonard@opensolaris:~$ svccfg -s application/pkg/server listprop pkg/port
pkg/port count 80

The default port is 80, not 1000. Also, my host name (opensolaris) was already in my /etc/hosts file:

bleonard@opensolaris:~$ cat /etc/hosts
...
::1 localhost
127.0.0.1 localhost loghost opensolaris

However, I am curious why it was different in your case.

Posted by Brian Leonard on September 11, 2009 at 08:13 AM GMT #

hello,
a question about mirrors, it's not clear for me.
Is it possible to keep all the release's repositories (local and remote) in case of local's one failure ?
But it seems that pkg always use the official remote depot instead of the local one?
# pkg publisher
PUBLISHER TYPE STATUS URI
opensolaris.org (preferred) origin online http://ultra20:81/
opensolaris.org (preferred) mirror online http://pkg.opensolaris.org/
With this config, "pkg install openffice-fr" is slow as if there was no local mirror.

Posted by gerard on November 12, 2009 at 04:42 AM GMT #

Hi Gerard,

IPS always attempts to use the configured mirror first. You're mirror is configured to use the remote depot. Try switching them and you should notice the performance improvement.

Regards,
Brian

Posted by Brian Leonard on November 12, 2009 at 12:06 PM GMT #

Hi Brian,

Thanks for this information. It has been very helpful.

I am currently trying to set up an environment which has local repositories for both release and dev. Can this be done on the same system? I have copies of both repositories but not sure if/how to configure pkg/server to see them both.

Thanks,
Tyson

Posted by Tyson on November 13, 2009 at 09:07 AM GMT #

Hi Tyson,

it's not a clear answer, but an idea that we tested here at CHOSUG and is working fine, with Brian's procedure
- installing an IPS server as a mirror for /release in a zone, here called IPS
- installing an IPS server from the latest build full ISO image in a separate zone, here called IPS-DEV

We rsynced the mounted ISO directly from the global zone the zone IPS-DEV directory

Now, for each new full ISO repository for a new build, we redo the RSYNC, then restart the pkg server in the IPS-DEV

We then have two independant network services running at low workload on a single system :)

You effectively have now one IP address per IPS server but, our next step is to add a reverse proxy zone to map /dev to zone IPS-DEV, and /release to zone IPS.

Each zone is very light, and I think that it takes less time to do it this way than trying the find how to do it in another way ...

HTH
Karim

Posted by Karim on November 13, 2009 at 09:34 AM GMT #

Hi Karim,

You propose an interesting solution. I was going to suggest running another instance of the pkg depot server. I haven't tried that myself yet though.

/Brian

Posted by Brian Leonard on November 13, 2009 at 09:51 AM GMT #

Thanks Karim for your response. I was wondering if I would have to use zones or some other virtual system to add the second repository. After seeing Brian's post I decided to look more into running another instance of the pkg/server.

This is how I got it to work:
(http://www.sun.com/bigadmin/features/articles/smf_example.jsp help me with this.)

# cp /var/svc/manifest/application/pkg-server.xml pkg-server-dev.xml

Made the following changes to /var/svc/manifest/application/pkg-server-dev.xml (not sure if all these changes are necessary):
Changed
<service_bundle type='manifest' name=':pkg-server'>
To
<service_bundle type='manifest' name=':pkg-server-dev'>

Changed
<service
name='application/pkg/server'
type='service'
version='1'>
To
<service
name='application/pkg/server-dev'
type='service'
version='1'>

Changed
<propval name='port' type='count' value='80' />
To
<propval name='port' type='count' value='81' />

Changed
image packaging repository
To
image packaging development repository

# svccfg
svc:> validate /var/svc/manifest/application/pkg/server-dev.xml
svc:> import /var/svc/manifest/application/pkg/server-dev.xml
svc:> quit

# svcadm enable -t svc:/application/pkg/server-dev

# svcs -a | grep pkg/server
online 13:02:48 svc:/application/pkg/server:default
online 13:02:48 svc:/application/pkg/server-dev:default

Now it is just a matter of following Brian's procedure but using the pkg/server-dev for the development repostiory instead of pkg/server.

Posted by Tyson on November 17, 2009 at 01:25 PM GMT #

Having set this up I noticed that my installations weren't any faster. After a bit of head scratching and because I was playing around with Custom AI manifests I found that I also had to change the following (for default)

/export/aiserver/osol-0906-ai-sparc/auto_install/default.xml

or if you're using a customer manifest

/export/aiserver/osol-0906-ai-sparc/auto_install/ai-newmanifest.xml

and modify

<main url="http://pkg.opensolaris.org/release" authname="opensolaris.org"/>

to

<main url="http://local-aihostname" authname="opensolaris.org"/>

Has anyone else experienced this?

Posted by Sean on June 14, 2010 at 07:41 AM GMT #

Hi Brian,
I am setting up local repository mirror according to your sample, with minor modification

These are the steps I create :
0. Open a New Terminal and run as root
1. I create another RAIDZ pool named dpool, and mount it to /opt
2. I followed your first step :
% mount -F hsfs `lofiadm -a ~/Download/osol-repo-0906-full.iso` /mnt

// This step run successfully

3. Next
% zfs create -o compression=on dpool/repo

// This step run successfully

4. Then copy the repository to the file system (this will take a little over an hour to complete):

% rsync -aP /mnt/repo /dpool/repo

// This command fails, with error messages saying that mount point /dpool/repo is not available
// So, what I did is I changed to directory /opt, which was the mount point for dpool, and from there,
// I run

% rsync -aP /mnt/repo repo

// This command runs successfully

5. Either this one fails,

% svccfg -s application/pkg/server setprop pkg/inst_root=/dpool/repo/repo

or this one fails

% pkg set-publisher -m http://opensolaris:81 opensolaris.org

6. Then the problem comes. I shutdown the computer, and when I turned on, the system fails at boot while mounting ZFS :
Mounting ZFS filesystems: (1/9) cannot mount '/opt/repo' : directory is not empty (9/9)
svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: exit status 1
Aug 8 00:20:04 svc.startd[7]: svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95.

Aug 8 00:20:04 svc.startd[7]: svc:/system/filesystem/local:default: failed fatally: transitioned to maintenance (see `svcs -xv` for details)

opensolaris console login:

7. Even login as root is unable.

8. What should I do ? I desperately need help. Can't figure out what to do this week, as this will be my working computer

Posted by Handojo on August 07, 2010 at 10:44 AM GMT #

Hi Handojo,

Try the following:

1) Boot your computer with the Live CD.
2) $ pfexec zpool import dpool
3) $ pfexec zfs set mountpoint=/dpool/repo dpool/repo

Good luck,
Brian

Posted by Brian Leonard on August 09, 2010 at 05:15 AM GMT #

I got the rep built just fine however I'm having trouble using it.

I added 'opensolaris' to my /etc/hosts. Started and stopped pkg service.
I'm not seeing anything listening on port 81.

$ sudo pkg set-publisher -m http://opensolaris:81 opensolaris.org
Certificate '/var/pkg/ssl/OpenSolaris_extras.certificate.pem' for publisher 'extra', needed to access 'https://pkg.sun.com/opensolaris/extra/', will expire in '5' days.
pkg set-publisher: Could not refresh the catalog for opensolaris.org
$ sudo pkg publisher
PUBLISHER TYPE STATUS URI
opensolaris.org (preferred) origin online http://pkg.opensolaris.org/release/
extra origin online https://pkg.sun.com/opensolaris/extra/

Posted by guest on September 30, 2011 at 02:23 PM GMT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

The Observatory is a blog for users of Oracle Solaris. Tune in here for tips, tricks and more as we explore the Solaris operating system from Oracle.

Connect with Oracle Solaris:


Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
20
21
22
23
24
25
26
27
28
29
30
   
       
Today