Thursday Apr 16, 2009

Sun Shared Shell - A Cool Diagnostic Tool

[Updated 2010.10.12 with new URL]

As part of helping a customer out recently on an escalation, the SSE on the case suggested using Sun Shared Shell, a tool that allows you to see and optionally control a remote system. It supports SSH and Telnet.

This tool was instrumental in increasing my understanding of what was going on with the customer's system, and removed the need to wait for output via emails or just trying to understand things over the phone. The owner of the session, usually the customer, has the option of allowing you to enter commands (without hitting 'Return'), or even allowing the 'Return' as well. It also has logging and chatting capabilities.

When first logging in, it allows you to be the owner of the shell and share that with other participants, or to view someone else's shell session.

Once logged in, you have a terminal window, the people present on the connection, and a chat window. The icon before the name/email address shows whether you have view, type, or full control (the keyboard will also have a down-arrow with it).

Oh, and I forgot about the feature to scribble on the screen. I used that to diagram out an idea I had to solve a zone networking issue, and it helped the others understand what I was proposing a lot quicker!

In the spirit of 'asking for what you want instead of complaining about what you don't have', I submitted a few suggestions, and the owner(s) quickly responded with clarifications.

I see this as a great tool to help future cases where a shared view of operations will improve understanding or service delivery! Thanks to those who created and maintain it! Steffen

Wednesday Mar 26, 2008

How to BFU a System

Sometimes you want to try out a new feature not yet delivered into Solaris Nevada, and you have apply binaries using BFU. I imagine if you do this all the time, you know all the tricks and gotchas. I don't do it often enough and sometimes get caught up in some details. So here are the steps I tend to use.

First, get the latest BFU package from the ON (OS/Net) Consolidation. I typically only use the SUNWonbld tar file for my hardware.

Download the bits you want to install, such as those for Crossbow Beta or Clearview's snoop on loopback

To make life a little simpler, I add the following to root's .profile file.

if [ -d /opt/onbld ]
   FASTFS=/opt/onbld/bin/`uname -p`/fastfs ; export FASTFS
   BFULD=/opt/onbld/bin/`uname -p`/bfuld ; export BFULD
   GZIPBIN=/usr/bin/gzip ; export GZIPBIN

Now to apply the bits. After unpacking the bits into a temporary location, lets say /tmp/bfu, install the onbld package.

# pkgadd -d onbld all

Processing package instance  from 

OS-Net Build Tools(sparc) 11.11,REV=2008.
Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.


Installation of  was successful.
I re-read my .profile, and verify that the necessary BFU variables are set
# . /.profile
# echo $FASTFS
Now apply the BFU (this one is for Crossbow beta). You must use the full pathname!

Note: you may want to do this from the console, in case you loose your network connection.

# bfu `pwd`/nightly-nd
Copying /opt/onbld/bin/bfu to /tmp/bfu.1000
Executing /tmp/bfu.1000 /tmp/bfu/nightly-nd


Entering post-bfu protected environment (shell: ksh).
Edit configuration files as necessary, then reboot.

Note that you end up in the BFU shell. Now issue an automatic conflict resolution check.
bfu# /opt/onbld/bin/acr
Getting ACR information from /tmp/bfu/nightly-nd... ok

updating //platform/sun4v/boot_archive
Finished.  See /tmp/acr.nhaqVi/allresults for complete log.

bfu# exit
Exiting post-bfu protected environment.  To reenter, type:
LD_NOAUXFLTR=1 LD_LIBRARY_PATH=/tmp/bfulib LD_LIBRARY_PATH_64=/tmp/bfulib/64 
PATH=/tmp/bfubin /tmp/bfubin/ksh
Its time to reboot and run with the new bits!

Thursday Feb 07, 2008

Solaris 10 Update 5 Beta Program!

The Solaris 10 Update 5 Beta Program solicitation for external beta candidates has gone out. To apply, go here.

The new or enhanced features in S10 Update 5 Beta Release to be tested include but are not limited to:

  • Infiniband flash update tool
  • Sockets Direct Protocol (SDP)
  • Persistent Group Reservation for iSCSI target
  • iSNS Client for iSCSI target
  • IP addressing ability for IBTF interfaces
  • Graphical User Interface for PostgreSQL (pgAdmin 3)
  • Support for download of new firmware into SATA drives
  • Support for Enhanced Intel SpeedStep power management technology
  • SunVTS 7.0
  • SAS multipathing support
  • Flash 9
  • New Instant Messaging Client (pidgin 2.0)
  • Virtual Network Computing (VNC)
  • Capping CPU resource usage

Monday Jan 28, 2008

Breaking the stdio(3C) 256 file descriptor barrier in Solaris' stdio

In the mid 1990s I was trying to show a customer that Solaris can host 10,000 web sites on a single system. This was in the days of 400MHz processors and 1GB of memory cost more than a luxury automobile. One thing I tried to was to use the Apache web server and virtual hosting. However, since the customer required separate logs for each web site, each virtual host needed at least one log file and thus a file descriptor. Apache uses fopen(3C) for these files. So I had to use on hundred web servers each hosting 100 web sites.

Solaris has historically allowed only 256 stdio streams to be open, where the file descriptors are below 256. So applications can quickly run out of file descriptors when doing lots of fopen() calls. For 32-bit applications, it has not been possible to increase this limit, as it could cause binary compatibility issues for older applications (compatibility going back as far back as those compiled on SunOS 4.x). The dup(2) system call has been used to move other file descriptors above 256 to free up slots for fopen. But the application is still limited to a maximum of 256 stdio streams!

With the release of Solaris 10 8/07 (often referred to as update 4), there is a new interface to extend the FILE facility. Programming details are in the man page enable_extended_FILE_stdio(3C). And if you don't want to make any code changes, extendedFILE(5) describes how to do this for existing applications and binaries.

I am working with a customer who needs to host over 1,400 web sites. We are using portions of the coolstack, as well as customized versions of Apache and PHP. With virtual hosting, the setup quickly hit the 256 stdio file limit!

With a small change to apachectl, it is now possible to host all 1,400+ web sites within a single instance of Apache. I added the following to the configuration section of apachectl:

ulimit -n 3000
LD_PRELOAD_32=/usr/lib/ ; export LD_PRELOAD_32 

The ulimit -n 3000 increases the number of file descriptors a process can have open to 3000, up from the default of 256. Since apachectl is run as root, or with sufficient privileges using Role Bases Access Control, this is permitted.

The LD_PRELOAD_32 setting allows me to have the library provide special versions of library functions or system calls. In this case, it does special things when fopen is called, and automatically uses dup(2) to free up the lower 256 file descriptors.

The enable_extended_FILE_stdio(3C) man pages lists some of the requirements for an application to work well with this interposition library, such as not doing direct access into the fields of the FILE structure. Since Apache is using stdio for log files, it is unlikely that Apache is accessing the structures directly.

Testing with the customer's configuration has Apache serving up all 1,400 web sites using a single instance of the httpd server! Cool, success at last!




« July 2016