Thursday May 08, 2014

No time like the future

Zones have forever allowed different time zones.  Kernel zones kicks that up to 11 (or is that 11.2?) with the ability to have an entirely different time in the zone.  To be clear, this only works with kernel zones.  You can see from the output below that the zone in use has brand solaris-kz.

root@kzx-05:~# zoneadm -z junk list -v
  ID NAME             STATUS      PATH                         BRAND      IP    
  12 junk             running     -                            solaris-kz excl  

By default, the clocks between the global zone and a kernel zone are in sync.  We'll use console logging to show that...

root@kzx-05:~# zlogin -C junk
[Connected to zone 'junk' console]

vzl-178 console login: root
Last login: Fri Apr 18 13:58:01 on console
Oracle Corporation      SunOS 5.11      11.2    April 2014
root@vzl-178:~# date "+%Y-%m-%d %T"
2014-04-18 13:58:57
root@vzl-178:~# exit

vzl-178 console login: ~.
[Connection to zone 'junk' console closed]
root@kzx-05:~# tail /var/log/zones/junk.console 
2014-04-18 13:58:45 vzl-178 console login: root
2014-04-18 13:58:46 Password: 
2014-04-18 13:58:48 Last login: Fri Apr 18 13:58:01 on console
2014-04-18 13:58:48 Oracle Corporation      SunOS 5.11      11.2    April 2014
2014-04-18 13:58:48 root@vzl-178:~# date "+%Y-%m-%d %T"
2014-04-18 13:58:57 2014-04-18 13:58:57
2014-04-18 13:58:57 root@vzl-178:~# exit
2014-04-18 13:59:04 logout
2014-04-18 13:59:04 
2014-04-18 13:59:04 vzl-178 console login: root@kzx-05:~# 

Notice that the time stamp on the log matches what we see in the output of date.  Now let's pretend that it is next year.

root@kzx-05:~# zlogin junk
root@vzl-178:~# date  0101002015
Thursday, January  1, 2015 12:20:00 AM PST

And let's be sure that it still thinks it's 2015 in the zone:

root@kzx-05:~# date; zlogin junk date; date
Friday, April 18, 2014 02:05:53 PM PDT
Thursday, January  1, 2015 12:20:18 AM PST
Friday, April 18, 2014 02:05:54 PM PDT

And the time offset survives a reboot.

root@kzx-05:~# zoneadm -z junk reboot
root@kzx-05:~# date; zlogin junk date; date
Friday, April 18, 2014 02:09:18 PM PDT
Thursday, January  1, 2015 12:23:43 AM PST
Friday, April 18, 2014 02:09:18 PM PDT

So, what's happening under the covers?  When the date is set in the kernel zone, the offset between the kernel zone's clock and the global zone's clock is stored in the kernel zone's host data.  See solaris-kz(5) for a description of host data.  Whenever a kernel zone boots, the kernel zone's clock is initialized based on this offset.

Wednesday May 07, 2014

Zones Console Logs

You know how there's that thing that you've been meaning to do for a long time but never quite get to it?  And then one day something just pushes you over the edge?  This is a short story of being pushed over the edge.

As I was working on kernel zones, I rarely used zlogin -C to get to the console.  And then I'd get a panic.  If the panic happened early enough in boot that the dump device wasn't enabled, I'd lose all traces of what went wrong.  That pushed me over the edge - time to implement console logging!

With Solaris 11.2, there's now a zone console log for all zone brands: you can find it at /var/log/zones/zonename.console.

root@kzx-05:~# zoneadm -z junk boot
root@kzx-05:~# tail /var/log/zones/junk.console 
2014-04-18 12:59:08 syncing file systems... done
2014-04-18 12:59:11 
2014-04-18 12:59:11 [NOTICE: Zone halted]
2014-04-18 13:00:56 
2014-04-18 13:00:56 [NOTICE: Zone booting up]
2014-04-18 13:00:59 Boot device: disk0  File and args: 
2014-04-18 13:00:59 reading module /platform/i86pc/amd64/boot_archive...done.
2014-04-18 13:00:59 reading kernel file /platform/i86pc/kernel/amd64/unix...done.
2014-04-18 13:01:00 SunOS Release 5.11 Version 11.2 64-bit
2014-04-18 13:01:00 Copyright (c) 1983, 2014, Oracle and/or its affiliates. All rights reserved.

The output above will likely generate a few questions.  Let me try to answer those.

Will this log passwords typed on the console?

Generally, passwords are entered in a way that they aren't echoed to the terminal.  The console log only contains the characters written from the zone to the terminal - that is it logs the echoes.  Characters you type that are never printed are never logged.

Can just anyone read the console log file?

No.  You need to be root in the global zone to read the console log file.

How is log rotation handled?

Rules have been added to logadm.conf(4) to handle weekly log rotation.

I see a time stamp, but what time zone is that?

All time stamps are in the same time zone as svc:/system/zones:default.  That should be the same as is reported by:

root@kzx-05:~# svcprop -p timezone/localtime timezone

Really, what does the time stamp mean?

It was the time that the first character of the line was written to the terminal.  This means that if a line contains a shell prompt, the time is the time that the prompt was printed, not the time that the person finished entering a command.

I see other files in /var/log/zones.  What are they?

zonename.messages contains various diagnostic information from zoneadmd.  If all goes well, you will never need to look at that.

zoneadm.* contains log files from attach, install, clone, and uninstall operations.  These files have existed since Solaris 11 first launched.

Tuesday May 06, 2014

A tour of a kernel zone

In my earlier post, I showed how to configure and install a kernel zone.  In this post, we'll take a look at this kernel zone.

The kernel zone was installed within an LDom on a T5-4.

root@vzl-212:~# prtdiag -v | head -2
System Configuration:  Oracle Corporation  sun4v SPARC T5-4
Memory size: 65536 Megabytes
root@vzl-212:~# psrinfo | wc -l

The kernel zone was configured with:

 root@vzl-212:~# zonecfg -z myfirstkz create -t SYSsolaris-kz

Let's take a look at the resulting configuration.

root@vzl-212:~# zonecfg -z myfirstkz info | cat -n
     1    zonename: myfirstkz
     2    brand: solaris-kz
     3    autoboot: false
     4    autoshutdown: shutdown
     5    bootargs: 
     6    pool: 
     7    scheduling-class: 
     8    hostid: 0x2b2044c5
     9    tenant: 
    10    anet:
    11        lower-link: auto
    12        allowed-address not specified
    13        configure-allowed-address: true
    14        defrouter not specified
    15        allowed-dhcp-cids not specified
    16        link-protection: mac-nospoof
    17        mac-address: auto
    18        mac-prefix not specified
    19        mac-slot not specified
    20        vlan-id not specified
    21        priority not specified
    22        rxrings not specified
    23        txrings not specified
    24        mtu not specified
    25        maxbw not specified
    26        rxfanout not specified
    27        vsi-typeid not specified
    28        vsi-vers not specified
    29        vsi-mgrid not specified
    30        etsbw-lcl not specified
    31        cos not specified
    32        evs not specified
    33        vport not specified
    34        id: 0
    35    device:
    36        match not specified
    37        storage: dev:/dev/zvol/dsk/rpool/VARSHARE/zones/myfirstkz/disk0
    38        id: 0
    39        bootpri: 0
    40    capped-memory:
    41        physical: 2G
    42    suspend:
    43        path: /system/zones/myfirstkz/suspend
    44        storage not specified
    45    keysource:
    46        raw redacted

There are a number of things to notice in this configuration.

  • No zonepath.  Kernel zones install into a real or virtual disks - quite like the way that logical domains install into real or virtual disks.  The virtual disk(s) that contain the root zfs pool are specified by one or more device resources that contain a bootpri property (line 39).  By default, a kernel zone's root disk is a 16 GB zfs volume in the global zone's root zfs pool.  There's more about this in the solaris-kz(5) man page.  It's never been a good idea to directly copy things into a zone's zonepath.  With kernel zones that just doesn't work.
  • The device resource accepts storage URI's (line 37).  See suri(5).  Storage URI's were introduced in Solaris 11.1 in support of Zones on Shared Storage (rootzpool and zpool resources).  This comes in really handy when a kernel zone is installed on external storage and may be migrated between hosts from time to time.
  • The device resource has an id property (line 38).  This means that this disk will be instance 0 of zvblk - which will translate into it being c1d0.  We'll see more of that in a bit.
  • The anet resource has an id property (line 34).  This means that this anet will be instance 0 of zvnet - which will normally be seen as net0.  Again, more of that in a bit.
  • A memory resource control, capped-memory, is set by default (lines 40 - 41).  In the solaris or solaris10 brand, this would mean that rcapd is used to soft limit the amount of physical memory a zone can use.  Kernel zones are different.  Not only is this a hard limit on the amount of physical memory that the kernel zone can use - the memory is immediately allocated and reserved as the zone boots.
  • A suspend resource is present, which defines a location for to write out a suspend file when zoneadm -z zonename suspend is invoked.
  • The keysource resource is used for an encryption key that is used to encrypt suspend images and host data.  solaris-kz(5) has more info on this.

There are several things not shown here that may also be of interest:

  • Previously, autoshutdown (line 4) allowed halt and shutdown as values.  It now also supports suspend for kernel zones only.  As you may recall, autoshutdown is used by svc:/system/zones:default when it is transitioning from online to offline.  If set to halt, the zone (kernel or otherwise) is brought down abruptly.  If set to shutdown, a graceful shutdown is performed.  Now, if a kernel zone has it set to suspend, the kernel zone will be suspended as svc:/system/zones:default goes offline.  When zoneadm boot is issued for a suspended zone, the zone is resumed.
  • If there are multiple device resources that have bootpri set (i.e. bootable devices), zoneadm install will add all of the boot devices to a mirrored root zpool.

From the earlier blog entry, this kernel zone was booted and sysconfig was performed.  Let's look inside.

To get into the zone, you can use zlogin just like you do with any other zone.

root@vzl-212:~# zlogin myfirstkz
[Connected to zone 'myfirstkz' pts/3]
Oracle Corporation      SunOS 5.11      11.2    April 2014

As I alluded to above, a kernel zone gets a fixed amount of memory.  The value shown above matches the value shown in the capped-memory resource in the zone configuration.

root@myfirstkz:~# prtconf | grep ^Memory
Memory size: 2048 Megabytes

By default, a kernel zone gets one virtual cpu.  You can adjust this with the virtual-cpu or dedicated-cpu zonecfg resources.  See solaris-kz(5).

root@myfirstkz:~# psrinfo
0       on-line   since 04/18/2014 22:39:22

Because a kernel zone runs its own kernel, it does not require that packages are in sync between the global zone and the kernel zone.  Notice that the pkg publisher output does not say (syspub) - the kernel zone and the global zone can even use different publishers for the solaris repository.  As SRU's and updates start to roll out you will see that you can independently update the global zone and the kernel zones on it.

root@myfirstkz:~# pkg publisher
solaris                     origin   online F

Because a kernel zone runs its own kernel, it considers itself to be a global zone.

root@myfirstkz:~# zonename

The root disk that I mentioned above shows up at c1d0.

root@myfirstkz:~# format
Searching for disks...done

       0. c1d0 <kz-vDisk-ZVOL-16.00GB>
Specify disk (enter its number): ^D

And the anet shows up as net0 using physical device zvnet0.

root@myfirstkz:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      zvnet0

Let's jump on the console and see what happens when bad things happen...

root@myfirstkz:~# logout

[Connection to zone 'myfirstkz' pts/3 closed]

root@vzl-212:~# zlogin -C myfirstkz
[Connected to zone 'myfirstkz' console]

myfirstkz console login: root
Apr 18 23:47:06 myfirstkz login: ROOT LOGIN /dev/console
Last login: Fri Apr 18 23:32:28 on kz/term
Oracle Corporation      SunOS 5.11      11.2    April 2014
root@myfirstkz:~# dtrace -wn 'BEGIN { panic() }'
dtrace: description 'BEGIN ' matched 1 probe

panic[cpu0]/thread=c4001afbd720: dtrace: panic action at probe dtrace:::BEGIN (ecb c400123381e0)

000002a10282acd0 dtrace:dtrace_probe+c54 (252acb8f029b3, 0, 0, 33fe, c4001b75e000, 103215b2)
  %l0-3: 0000c400123381e0 0000c40019b82340 00000000000013fc 0000c40016889740
  %l4-7: 0000c4001bc00000 0000c40019b82370 0000000000000003 000000000000ff00
000002a10282af10 dtrace:dtrace_state_go+4ac (c40019b82340, 100, 0, c40019b82370, 16, 702a7040)
  %l0-3: 0000000000030000 0000000010351580 0000c4001b75e000 00000000702a7000
  %l4-7: 0000000000000000 0000000df8475800 0000000000030d40 00000000702a6c00
000002a10282aff0 dtrace:dtrace_ioctl+ad8 (2c, 612164be40, 2a10282bacc, 202003, c400162fcdc0, 64747201)
  %l0-3: 000000006474720c 0000c40019b82340 000002a10282b1a4 00000000ffffffff
  %l4-7: 00000000702a6ee8 00000000702a7100 0000000000000b18 0000000000000180
000002a10282b8a0 genunix:fop_ioctl+d0 (c40019647a40, 0, 612164be40, 202003, c400162fcdc0, 2a10282bacc)
  %l0-3: 000000006474720c 0000000000202003 0000000001374f2c 0000c40010d84180
  %l4-7: 0000000000000000 0000000000000000 00000000000000c0 0000000000000000
000002a10282b970 genunix:ioctl+16c (3, 6474720c, 612164be40, 3, 1fa5ac, 0)
  %l0-3: 0000c4001a5ea958 0000000010010000 0000000000002003 0000000000000000
  %l4-7: 0000000000000003 0000000000000004 0000000000000000 0000000000000000

syncing file systems... done
dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel sections: zfs
 0:04  90% done (kernel)
 0:05 100% done (zfs)
100% done: 127783 (kernel) + 12950 (zfs) pages dumped, dump succeeded

[NOTICE: Zone rebooting]
NOTICE: Entering OpenBoot.
NOTICE: Fetching Guest MD from HV.
NOTICE: Starting additional cpus.
NOTICE: Initializing LDC services.
NOTICE: Probing PCI devices.
NOTICE: Finished PCI probing.

SPARC T5-4, No Keyboard
Copyright (c) 1998, 2014, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.36.0, 2.0000 GB memory available, Serial #723535045.
Ethernet address 0:0:0:0:0:0, Host ID: 2b2044c5.

Boot device: disk0  File and args: 
SunOS Release 5.11 Version 11.2 64-bit
Copyright (c) 1983, 2014, Oracle and/or its affiliates. All rights reserved.
Hostname: myfirstkz
Apr 18 23:48:44 myfirstkz savecore: System dump time: Fri Apr 18 23:47:42 2014
Apr 18 23:48:44 myfirstkz savecore: Saving compressed system crash dump files in directory /var/crash

myfirstkz console login: Apr 18 23:49:02 myfirstkz savecore: Decompress all crash dump files with '(cd /var/crash && savecore -v 0)' or individual files with 'savecore -vf /var/crash/vmdump{,-<secname>}.0'

EVENT-TIME: Fri Apr 18 23:49:07 CDT 2014
PLATFORM: SPARC-T5-4, CSN: unknown, HOSTNAME: myfirstkz
SOURCE: software-diagnosis, REV: 0.1
EVENT-ID: f4c0d684-da80-425f-e45c-97bd0239b154
DESC: The system has rebooted after a kernel panic.

After disconnecting from the console (~.) I was back at the global zone root prompt.  The global zone didn't panic - the kernel zone did.

root@vzl-212:~# uptime; zlogin myfirstkz uptime
  9:53pm  up  8:03,  2 users,  load average: 0.03, 0.12, 0.08
 11:52pm  up 5 min(s),  0 users,  load average: 0.04, 0.26, 0.15

That's the end of this tour.  Thanks for coming, and please come again!

Tuesday Apr 29, 2014

Solaris 11.2 Beta zones man pages

For those of you trying out the latest zones features, beware that many of the man page updates did not make it into the beta build, so man(1) will not tell you the full story.  The man page updates are in the beta documentation, linked below for your convenience.

The zones book has been split apart and a new book has been introduced for kernel zones:

Install a kernel zone in 3 steps

One of the shiniest new features in Oracle Solaris 11.2 is Kernel Zones.  Kernel Zones provide the familiarity of zones while providing independent kernels.  This means that it's now possible to have zones that run different patch levels, act as CIFS servers, load kernel modules, etc.  So, let's get to installing a kernel zone.

If you have installed any other zone on Solaris before, this will look quite familiar.  After all, it is  just another zone, right?

For this procedure to work, there are some prerequisites that shouldn't be much of a problem in a production environment, but are a bit of a problem if your normal playground is VirtualBox or the 6 year old server you found on the loading dock.

Step 1: Configure

root@vzl-212:~# zonecfg -z myfirstkz create -t SYSsolaris-kz

Step 2: Install

root@vzl-212:~# zoneadm -z myfirstkz install
Progress being logged to /var/log/zones/zoneadm.20140419T032707Z.myfirstkz.install
pkg cache: Using /var/pkg/publisher.
 Install Log: /system/volatile/install.5368/install_log
 AI Manifest: /tmp/zoneadm4798.dAaO7j/devel-ai-manifest.xml
  SC Profile: /usr/share/auto_install/sc_profiles/enable_sci.xml
Installation: Starting ...

        Creating IPS image
        Installing packages from:
                origin:  http://ipkg/solaris11/dev/
        The following licenses have been accepted and not displayed.
        Please review the licenses for the following packages post-install:
        Package licenses may be viewed using the command:
          pkg info --license <pkg_fmri>

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            549/549   76929/76929  680.9/680.9  8.4M/s

PHASE                                          ITEMS
Installing new actions                   104278/104278
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 
Installation: Succeeded
        Done: Installation completed in 438.132 seconds.

Step 3: Celebrate!

At this point the kernel zone is installed and ready for boot. 

root@vzl-212:~# zoneadm -z myfirstkz boot
root@vzl-212:~# zlogin -C myfirstkz
[Connected to zone 'myfirstkz' console]
Loading smf(5) service descriptions: 220/220

Because a sysconfig profile was not provided during installation, sysconfig(1M) will ask a few things on first boot.

                           System Configuration Tool
     System Configuration Tool enables you to specify the following            
     configuration parameters for your newly-installed Oracle Solaris 11       
     - system hostname, network, time zone and locale, date and time, user     
       and root accounts, name services, keyboard layout, support
     System Configuration Tool produces an SMF profile file in
     How to navigate through this tool:
     - Use the function keys listed at the bottom of each screen to move       
       from screen to screen and to perform other operations.
     - Use the up/down arrow keys to change the selection or to move           
       between input fields.
     - If your keyboard does not have function keys, or they do not            
       respond, press ESC; the legend at the bottom of the screen will         
       change to show the ESC keys for navigation and other functions.         
  F2_Continue  F6_Help  F9_Quit

If you've read this far into this entry, you know how to take it from here.


I'm a Principal Software Engineer in the Solaris Zones team. In this blog, I'll talk about zones, how they interact with other parts of Solaris, and related topics.


« March 2015