About a year and half ago I posted instructions about using Xvnc with Trusted Extensions. Those instructions apply to systems using dtlogin, the CDE Display Manager, such as Nevada releases and Solaris 10 update 7. However, OpenSolaris uses gdm, the GNOME Display Manager and requires a different set of configuration procedures.
There is an excellent blog by Abhimanyu on this topic that describes the configuration steps for OpenSolaris 2009.06. In general, it also applies to Trusted Extensions, so you should begin by following those instructions. However, there are a few more issues and procedures required to get this to work properly in a labeled environment.
The first problem is that the Xvnc server that is started by the xvnc-inetd SMF service is assigned to the user and group noaccess. While this is generally a good idea, it prevents Xvnc from binding to one of the multilevel ports (6000-6003 by default). You may notice that the DISPLAY variable starts with hostname:4 because that is the first unprivileged TCP port available. There are two workarounds for this problem:
One way to tell the X clients to use UNIX domain sockets is to set the hostname component of the DISPLAY variable to unix, e.g. unix:1. However, the next question is where to specify this setting. I couldn't find a supported way to do this, so I modified the script /etc/X11/gdm/Xsession. The first non-comment line sets the DISPLAY. I changed it as follows:
export DISPLAY=`echo $DISPLAY | sed -e "s/127.0.0.1/unix/"`
This assumes that the normal DISPLAY is already set to the IP address of localhost, which is the default for OpenSolaris TX. If yours is different, make the appropriate change. A major advantage of using UNIX domain sockets it the the labeled zones don't require a route to the global zone's X server.
The other approach is to add the privilege, net_bindmlp, which is required to bind to a multilevel port. This can be done by editing the xvnc-inetd service. Start by running these commands:
# svccfg -s xvnc-inetd
A gedit window will pop up. Look for the following line specifying the inetd_start/privileges property, remove the comment character and add the net_bindmlp privilege:
setprop inetd _start/privileges = astring: basic,net_bindmlp
Save the file, quit gedit, and exit svccfg. Then refresh the service, as follows:
# svcadm refresh xvnc-inetd
Once you've got this working, you'll probably want to replace the default GNOME login window with the OpenSolaris dialog. To do this, edit the file /etc/X11/gdm/custom.conf, as follows:
You may need to restart the gdm service for this to take effect:
# svcadm restart gdm