Friday May 11, 2012

Installing MacPorts Behind a Proxy

I needed to build a project on my MacBook that required GNU msgfmt. I could have built it from source, but instead I decided to install MacPorts in case I needed other open source type packages in the future.

Installing MacPorts is fairly straightforward, but the first thing you must do after installing the base image is synchronize with the MacPorts repository by running:

$ sudo port -d sync

That failed for me because MacPorts uses rsync to access its repositories, and I was sitting behind a firewall that requires the use of an HTTP proxy to get to the internet. Setting RSYNC_PROXY didn't help since (apparently) our proxy does not support rsync.

Fortunately there is an alternative. You can use Subversion to sync up with the repository content, and Subversion works over http. After making sure my HTTP proxy for svn was set (by checking my ~/.subversion/servers file). I followed the directions described in How to sync your ports tree using Subversion (over HTTP) with the following modification:

  1. Before checking out the initial copy of the ports tree I changed permission on /opt/local/var/macports/sources/trunk/dports to be owned by me (jdipol) and not root.
  2. I did the initial "svn co" as me and not root. This left the subversion workspace owned by me and not root.

Why did I do this? Because  "port sync" runs svn to make sure the ports tree is up to date. It runs svn as the user that owns the subversion workspace. I wanted that to be jdipol to always pick up my subversion settings from ~jdipol/.subversion -- not from root's environment. So now even though I run "port sync" as root it still runs "svn update" as me.

And don't forget you must install Apple's Xcode even if you don't plan on doing any development. Why? Because MacPorts requires "make" to  process some of the ports. So make sure you install Xcode including CLI commands:

  1. Install Xcode from the AppStore -- it's free
  2. Start Xcode
  3. Select Preferences->Downloads and install Command Line Tools

Once you do that, then you are ready to run:

$ sudo port -d sync

To get msgfmt all I needed to do was:

$ sudo port install gettext

If you want to see all the packages available to you, run

$ port list


Wednesday Jan 06, 2010

Windows TCP Receive Window Auto-Tuning

During the final testing of GlassFish Enterprise Server v3 the GlassFish QA team reported this Update Center bug:

11185 Installer hangs at 41% intermittently

We knew that at this phase of the install the update center was bootstrapping (downloading and installing) itself from the network repository into the installation image.  After some initial analysis we determined a few more things: this was only on Windows Vista and Windows 7, it has a hard hang, it was intermittent, we could only reproduce it inside SWAN (Sun's internal network). Further analysis showed that the client was blocked on a read from an HTTP connection to the repository, the repository thought everything was fine (and in fact showed no open connection from the client at all !?). Tom, through the use of WireShark, observed the TCP window size fluctuating significantly before the hang.

We were stumped -- so we Googled it. And the culprit was: TCP Receive Window Auto-Tuning

This feature is enabled by default on Vista, and attempts to automatically tune some TCP parameters including the Receive Window size to improve network performance. Unfortunately this can interact poorly with some pieces of network gear or firewalls leading to slowness or hangs. In our case disabling Auto-Tuning cleared up the problem.

For more details including how to disable auto-tuning check out the Microsoft blog posting on the subject or this article from www.vistax64.com.

Friday Jan 30, 2009

Screencast Tips

I recently produced a screencast for Update Center 2, and I've had some folks ask me how I did it. There are a zillion tools and techniques for producing screencasts. This blog describes one approach. I in no way claim it's the best approach -- and in fact I'm certain it is not. But it worked OK for me.

I did this on a Mac laptop running Mac OS 10.5. Windows users will likely need to look elsewhere for guidance.

I've broken down the job of creating a screencast into four steps:

  1. Recording raw content
  2. Editing content into a video
  3. Converting video into final screencast format
  4. Publishing screencast

Recording Raw Content

To record the raw content off of the screen I used  iShowU HD  from shinywhitebox. Benefits to this software:

  • It's inexpensive: $30
  • It's easy to use
  • It seems to do enough for basic screencasts

The main drawback is it's pretty basic. I imagine there are more sophisticated solutions out there. For example I needed to use some additional software to convert my Quicktime video into Flash Video for publishing to the web.

I really did not evaluate any other solution. You may want to before blindly following me.

Here are the iShowU HD settings I used for screen recording:

Output Video Size 720x540, 4:3
Output Video Codec Apple Animation, High Quality
Output Video Framerate 4 fps
Output Audio AAC, 44100 khz, 2 channel
Capture Area Same as video size (720x540)
Mouse Mode Fixed / Stretch


Notes:

  1. To set these I first selected "iMovie '08 Standard (4:3) Large", then changed the framerate
  2. 720x540 seemed to be a reasonable compromise. Large enough to be legible but not giant
  3. Throughout this process I made sure to always keep my video size the same so I never scaled the original content
  4. Note the low framerate. If your software uses snazzy animations you may want to bump this up a bit.
  5.  Audio quality is better than needed, but I down-converted it later
  6. I set my capture area to be the upper left corner of the screen. So the area of the screen being recorded is a 720x540 box in the upper left of the screen -- except when the mouse moves out of that box. See next item.
  7. The mouse mode means the capture area is fixed, but if the mouse leaves the capture area the video zooms out to keep the mouse in frame. More on this later.

General Recording Tips:

  1. Practice a few times before recording.
  2. I found the built-in mic on the Mac to be of pretty good quality, so I just used that. If you have a high quality external mic you may want to give it a try.
  3. Make sure you are in a quiet area where you won't be interrupted.
  4. Be aware of the location of your mouse. When it's "in frame" you will have a tight shot of the upper left corner of the desktop. If you move out of frame the video will zoom out to keep the mouse in frame. You don't want to oscillate back and forth. Don't wiggle the mouse nervously (talking with your hands!)
  5. Keep the mouse pointer near the upper left when you want to show detail. To show your entire app move the mouse to the lower right corner of your app. To show the entire desktop move your mouse to the lower right corner of the desktop.
  6. If you are using a terminal increase the font size until it just fits in 720x540. This will be legible.
  7. iShowU HD will preview what it will record when it is up. Use that to get a feel for the framing and zooming.
  8. I put post-its on the edge of my screen to mark the 720x540 boundary and remind me what was in frame.

Once you are ready to record, click Rec and go! Do your best to get it done in a single take. You want to minimize the amount of editing required. Consider breaking down your screencast to 5 minute clips (I violated this with my 15 minute take).

When you're done recording you will have a mov file in Movies/iShowU HD/.

Editing Content

I used iMovie '08 to assemble the content into a video. Import your video using File->Import Movies. I performed very minor edits -- basically splicing in an intro and conclusion that I had to "re-shoot". iMovie '08 is pretty weak -- rumor has it '09 is a significant improvement.

I then exported the video using "Share->Export Quicktime" with the following settings (click the "Options..." button in the export dialog)

Export Movie to QuickTime Movie
Video Settings
Compression Type H.264
Frame Rate 4
Key Frames Automatic
Compressor Quality High
Encoding Best quality
Data Rate Automatic
Audio Settings
Format Linear PCM
Channels Mono
Rate 22.050 kHz
Quality Normal
Sample Size 1


After this you will have a new .mov file with your screencast

Convert Video to Final Format

The problem with video formats is that it is still, after all these years, difficult to find one format that plays universally on all platforms. To get around this I converted to flash video format. To do this conversion I used  ffmpegx  to convert my Quicktime movie to flash video (.flv).

Here are the ffmpegX settings I used:

Video
Codec Flass Video (.FLV)
Size 720x540
Auto Size 4:3
Frame Rate 4 fps
Audio
Codec MP3
Bitrate 64 kbit/s
Sampling 22050
Channels Mono
Mode
CBR
Options
High Quality Two pass encoding|

Publishing Screencast

So, now you have a .flv file -- what the heck do you do with it?  Well, first you want to preview it, then you want to publish it to the world.

To preview it, place the file someplace accessible via an http URL, then put the following code in an HTML page:

<object type="application/x-shockwave-flash" width="720" height="556" wmode="transparent" 
    data="http://mediacast.sun.com/flash/jw-flv-player/mediaplayer.swf?file=
          http://yourserver.sun.com/path/to/video/screencast.flv&autoStart=false">
    <param name="movie"
        value="http://mediacast.sun.com/flash/jw-flv-player/mediaplayer.swf?file=
        http://yourserver.sun.com/path/to/video/screencast.flv&autoStart=false" />
    <param name="wmode" value="transparent" />
</object>

This uses the Flash NLV player from mediacast.sun.com to play the video in your web page.

For final publication Sun employees can upload the flv file to http://mediacast.sun.com. Once you do that you can click the "Share" area on the video to see various options for sharing.

Friday Aug 15, 2008

Dynamic Libraries, RPATH, and Mac OS

One of my responsibilities on the Update Center 2.0 project is to perform builds of Python and wxPython for all of our supported platforms. One unique aspect of our environment is that we need these builds to be relocatable as far as the filesystem is concerned in order to support our multi-install requirement. That is: plop the build anywhere on your system and it must work.

A key to making relocatable software work is relative paths. All references to files within the relocatable install image must be relative (or at least start out relative and only made absolute dynamically at runtime). This includes any dependencies that binaries in the install image might have on dynamic libraries in the install image.

You can always use LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH on the Mac) to force the runtime linker to locate the right dynamic libraries -- but that's cheating, and should only be used as a hack of last resort. Better to construct the binaries correctly so that they can locate their dependencies without the need to alter the user's environment.

I'm pretty familiar with dynamic libraries on Solaris and Linux. On those platforms you can embed an RPATH into a binary that is searched by the runtime linker to locate libraries. Plus both Solaris and Linux support the $ORIGIN token so you can make these paths relative to the install location of the binary.

On Solaris you can see these settings by using dump -Lv:

$ dump -Lv wx/_core_.so

wx/_core_.so:

  \*\*\*\* DYNAMIC SECTION INFORMATION \*\*\*\*
.dynamic:
[INDEX] Tag         Value
[1]     NEEDED          libCrun.so.1
[2]     NEEDED          libwx_GTK2u_richtext-2.8.so.0.4.0
[3]     NEEDED          libwx_GTK2u_aui-2.8.so.0.4.0
[4]     NEEDED          libwx_GTK2u_xrc-2.8.so.0.4.0
[5]     NEEDED          libwx_GTK2u_qa-2.8.so.0.4.0
[6]     NEEDED          libwx_GTK2u_html-2.8.so.0.4.0
[7]     NEEDED          libwx_GTK2u_adv-2.8.so.0.4.0
[8]     NEEDED          libwx_GTK2u_core-2.8.so.0.4.0
[9]     NEEDED          libwx_baseu_xml-2.8.so.0.4.0
[10]    NEEDED          libwx_baseu_net-2.8.so.0.4.0
[11]    NEEDED          libwx_baseu-2.8.so.0.4.0
[12]    INIT            0xc8ef4
[13]    FINI            0xc9000
[14]    RUNPATH         $ORIGIN/../wxWidgets/lib
[15]    RPATH           $ORIGIN/../wxWidgets/lib
[16]    HASH            0x94

Here the _core_.so binary has dependencies on a number of wx libraries. RPATH is set to look for libraries relative to the install location of _core_.so, and therefore the linker can find these libraries without needing to set LD_LIBRARY_PATH. You specify the value for RPATH at link time using the -R option.

Linux is similar. In this case you use objdump -p to inspect the binaries:

$ objdump -p wx/_core_.so

_core_.so:     file format elf32-i386
. . .

Dynamic Section:
  NEEDED      libwx_gtk2u_richtext-2.8.so.0
  NEEDED      libwx_gtk2u_aui-2.8.so.0
  NEEDED      libwx_gtk2u_xrc-2.8.so.0
  NEEDED      libwx_gtk2u_qa-2.8.so.0
  NEEDED      libwx_gtk2u_html-2.8.so.0
  NEEDED      libwx_gtk2u_adv-2.8.so.0
  NEEDED      libwx_gtk2u_core-2.8.so.0
  NEEDED      libwx_baseu_xml-2.8.so.0
  NEEDED      libwx_baseu_net-2.8.so.0
  NEEDED      libwx_baseu-2.8.so.0
  NEEDED      libstdc++.so.6
  NEEDED      libm.so.6
  NEEDED      libgcc_s.so.1
  NEEDED      libpthread.so.0
  NEEDED      libc.so.6
  RPATH       $ORIGIN/../wxWidgets/lib
  INIT        0x1d384

. . .

But what about the Mac? That's new territory for me. Here is what I learned -- I welcome comments since I likely know just enough to be dangerous.

On the Mac a dynamic library (dylib) has an "install name". The install name is a path baked into the dynamic library that says where to find the library at runtime. When you link against the dylib this path is saved in your binary so that your binary can find the dylib at runtime. Seems a bit backwards to me -- but that's how it works.

You can see the install name of a dylib by using otool. For example:

$ otool -D libwx_macu-2.8.0.4.0.dylib
libwx_macu-2.8.0.4.0.dylib:
/Users/dipol/wxPython/dist-darwin/wx-2.8/wxWidgets/lib/libwx_macu-2.8.0.dylib

You can also use otool to inspect binaries and list their dependencies:

$ otool -L wx/_core_.so
wx/_core_.so:
        /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
        /Users/dipol/wxPython/dist-darwin/wx-2.8/wxWidgets/lib/libwx_macu-2.8.0.dylib 
                                  (compatibility version 5.0.0, current version 5.0.0)
        . . .

So wx/_core_.so depends on libwx_macu-2.8.0.dylib, which is fine. But notice that evil absolute path. That won't work in our "install anywhere" world. So how do we fix this? What's the Mac OS equivalent of $ORIGIN and RPATH?

Well, there isn't any. But there is something we can do instead. After the build is complete I use the install_name_tool utility to fix up the dylib install names and dependencies in our binaries. Fortunately the Mac also supports a magic token: @loader_path that can be used in a fashion similar to $ORIGIN on Solaris/Linux.

So when building wxPython on the Mac I first do a complete build. This results in the absolute paths being used as mentioned above. Then as part of my "make install" step I fix up these paths to be relative using the install_name_tool utility.

For example, this command changes the install name of libwx_macu-2.8.0.4.0.dylib to be relative to the location of the binary using it:

$ install_name_tool -id "@loader_path/../wxWidgets/lib/libwx_macu-2.8.0.4.0.dylib"
                        libwx_macu-2.8.0.4.0.dylib

And this changes the dependency in a binary to use a relative path to locate the library (relative to the install location of the binary).

$ install_name_tool -change "/Users/dipol/wxPython/dist-darwin/wx-2.8/wxWidgets/lib/libwx_macu-2.8.0.dylib"
            "@loader_path/../wxWidgets/lib/libwx_macu-2.8.0.dylib" wx/_core_.so

This mechanism is still not as flexible as the Solaris/Linux approach, since binaries can't specify a search path. It's also a bit more difficult to automate since you must determine the current install name in order to replace it with the new install name. But it does work.

Note that for your project you may be able to simplify this. For example, if you build your dynamic libraries with the correct install names first, then your binaries will pick up the correct install names at link time and you shouldn't need to change the dependencies post-build.

About

jdipol

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today