Tuesday Apr 05, 2011
Saturday Apr 25, 2009
By tpm on Apr 25, 2009
Inspired by Tom Igoe's excellent articles in volume 14 of make magazine, called "Wireless Motion Sensing Made Easy", I've been playing with XBee 802.15.4 modules. These are Zigbee wireless endpoints in the form of little $20 circuit board modules that can can encode both digital and analog inputs to allow you to easily build various kinds of wireless sensor endpoints into wireless sensor networks. Here's one of mine next to a dime. You can see the fractal patterns on the 2.4GHz chip antenna at the top.
I used an XBee Explorer board from sparkfun.com that connects (another) XBee radio to any USB port using the FTDI FT232R serial chip to create a self-contained base station. You can see the mini USB socket at the front - and the other end of the cable I use plugs into a regular USB port on most computers. Here's my base station fully assembled!
My initial experiments were done on a Mac because I don't have a (physical) Windows machine, and so that was the path of least resistance to follow the make article and to be able to use the associated resources. Though I was pained that I had to use Windows to run the firmware updater for my XBee radios, I was thoroughly cheered up by the fact that I was able to use VirtualBox to host XP to do the update.
After I got things to work, I poked around the sparkfun site to look at some of the other things those guys have, as well as other sites, and I began to realize that the FT232R chip was pretty widely used as an interface device - though of course there are many similar devices out there in the same vein. So, I wondered just how hard it would be to put together a USB driver for opensolaris to support the chip. Particularly after I'd switched my everyday laptop back to opensolaris -- a few weeks before 2008.11 was released -- and I really wanted to see if I could get my XBee hobby going on my opensolaris laptop instead.
After a bit of noodling on the opensolaris Writing Device Drivers book , and applying the usual technique of creating a driver by cloning another driver in the same category from the opensolaris code base, it turned out to be a lot easier than I'd expected. I got a prototype working after a couple of Sunday afternoons. And after a few minor skirmishes with Java, I got the Processing development and runtime environment to work on opensolaris, and I was able to get the 3-axis wireless accelerometer I'd built to work over XBee via my usbftdi(7D) driver. Tom's XBee code now worked just as well as it worked on MacOS.
Hopefully I'll get a chance to write more about motion sensors, the other software components I had to tweak, and my experiences with getting Processing working on opensolaris in future posts.
Originally, I thought this was all just for me to experiment with, but after I happened to mention that I had this working to a couple of people at Sun, it turned out that a few of them had been experimenting with this device in various contexts, or they knew someone who had, and they all asked me to put it in the opensolaris distribution. In fact, as I poked around, I found that a number of other interesting embedded items were using the FT232R as an interface e.g.:
- The host interface for the open source Arduino electronics prototyping platform e.g. the Duemilanove.
- A breakout board and other items related to sensor technologies including miniature GPS receiver modules.
- A USB device in a "stick" form-factor containing a high quality random number generator from a different supplier.
Though it's a slight diversion from the story here, I can't resist posting a picture of a miniature GPS receiver attached to an FT232R breakout board. The green board on the left holds the complete GPS receiver; the small oblong on the extreme left is another chip antenna. The red board is the sparkfun FT232R breakout board.
Yes, that's a transparent Lego element taped to the breakout board -- a bit of double-sided tape holds it nicely -- though of course it's upside down in this picture, so you can't actually see the chip, or get a clear look at the mini USB connector that is slightly protruding on the right hand side. Obviously this is kludged together from prototyping boards, so it's a lot larger than it needs to be. I've removed the other Lego elements I use to protect the circuitry from harm so that you can see the receiver more clearly. I'll try and write some more about my experiences with the GPS software that talks to this hardware in yet another post.
Meanwhile, back from that diversion ... so as I began to realize just how handy this little interface chip was, what started as a "holiday project" and a Sunday afternoon hobby suddenly seemed a bit too useful for the device driver it to simply stay at home. Not because it directly addresses Sun's core mission around computing infrastructure, or has anything particularly to do with virtualization (my day job), but because it's fun, and fun encourages adoption, and adoption further expands the use of opensolaris into new domains. But in the end, the thing that pushed me into making the driver be part of the product was a real business opportunity involving support for the random number generator "stick" above.
And so my hobby intruded on my day job more than I originally planned, because on Easter Sunday afternoon, I integrated support for that device in the form of the usbftdi device driver into build 113. You can find the driver source code by looking for uftdi_dsd.c and usbser_uftdi.c using opengrok on opensolaris.org. Please send me code review comments -- especially if you find bugs.
In the past few years, I've spent more and more of my time on planning, architecture and oversight of other projects, and what code I have written has been integrated as part of the work of a project team i.e. someone else dealt with the responsibility of getting the work signed off and integrated. So from a code review, legal review, ARC review, documentation, the RTI process, the joy of testing, and of course the vagaries of the builds and my unfamiliarity with mercurial, it was good to be reminded of what an engineer has to go through to get working code delivered. And fixing, in build 114, a bug in the package I'd integrated in build 113 also reminded me of how difficult it is to get everything right first time. Since this wasn't my day job, i.e. it was a continually interrupted background task, it took a lot longer to get this done than I expected, which contributes, I think, to the feeling that I climbed a small mountain!
It was a very useful experience to me both in terms of finding a couple of issues that I'm going to try and improve in the process, as well as simply recalibrating my sense of what's involved in the details of getting OS technology delivered. It's all too easy when managing software projects to dismiss every problem as a SMOP (a simple matter of programming). And of course the simple sanity-inducing joy of working on bugs, finding solutions to them, and making the code base a better place was welcome too. For example, I found and fixed a bug with the generic USB serial support in opensolaris that caused any lwp issuing some of the slightly more obscure ioctls to simply hang, I added more termio ioctl decoding to truss, and upped the highest speed serial port a serial device can support to 921600 baud.
Of course I also had help: thanks to the USB team in Beijing -- Strony Zhang and Lin Guo -- Lin was particularly helpful for both code review and by patiently helping me do the right thing with warlock; to Jim Carlson for code review and advice on the opensolaris serial port architecture; to John Downing, and Mary Ding for help with testing; to John Weeks for trying the driver out with Wiring on his Arduino board; to Jerry Gilliam and Mike Sullivan for RTI and packaging review. And thanks to Patricia Levinson for fixing and integrating my somewhat approximate manpage.
Summing all this up, as of build 113 opensolaris has support for devices that use the FT232R interface and its clones. Plug one of these devices into a USB port and, like any other USB serial device, it will show up as /dev/cua/number and behave like a terminal device. Have fun with it!Technorati Tags: virtualbox, xbee, arduino, opensolaris.
Sunday Jun 08, 2008
By tpm on Jun 08, 2008
VirtualBox 1.6.2 now available for download. There are fixes to some critical bugs, and performance work in a few areas; as usual the Changelog has the details. And among those changes, the new improved icon!
Controlling VirtualBox via JAX-WS
Here's something that really caught our eye. Kohsuke Kawaguchi wrote a very interesting blog entry about how to control VirtualBox via its new web services management interface using JAX-WS. Since all the VirtualBox interfaces are specified in an XML format IDL, many different interface styles are possible, so he wrote a Groovy script to create the Java classes from that IDL. The resulting interface has better type safety and is more natural for Java programmers. You can find the various references to all the code he created - including the Groovy script, and the Javadoc for the API - by following the links in his blog.Technorati Tag: VirtualBox
Thursday May 22, 2008
By tpm on May 22, 2008
Interesting review of VirtualBox 1.6 here.
Another way to see more of what VirtualBox can do is to go to youtube and search for virtual box. People around the world have been sharing their experiences and have posted hundreds of videos of what they've been doing with it. And don't forget to check out the VirtualBox community page, and the particularly the forums .. lots of information there too. And of course you can visit other sites like technorati, digg, stumbleupon and wikipedia to see what people are saying about it.Technorati Tag: VirtualBox
Saturday May 17, 2008
By tpm on May 17, 2008
At the risk of sounding like a late-night TV commercial for toothpaste ... I'm occasionally asked the question "which virtualization technology should I use?" As if there's a one-size-fits-all answer. Well, admittedly it's more often a desire for a simplicity in the face of a complex world, and probably mostly because I've been confusing them with too many possibilities! Often, one technology does makes more sense than another, depending on what the customer is trying to do. However, there are many virtualization opportunities up and down the SW stack, and sometimes combining technologies can lead to more effective solutions.
Since Solaris Containers (Zones) and hardware virtualization components like hypervisors operate using different mechanisms at different levels in the SW stack, they can be used simultaneously to provide enhanced capabilities and efficiencies. One combined usage pattern we've seen treats the hardware virtualization facility almost as a static partitioning technology, doing coarse-grain resource partitioning, while exploiting the unique capabilities that hardware virtualization brings e.g. being able to run completely different operating systems, or operating system versions side by side. Obvious examples are x86/x64 machines using VMware products, or the xVM hypervisor built into OpenSolaris, or other x86 hypervisor solutions. Or customers with our CMT SPARC systems using the Logical Domains hypervisor built into the firmware.
Now, inside the Solaris or OpenSolaris domains running in these environments, customers use Solaris containers to encapsulate a set of applications as their unit of deployment, and manage fine-grain resource allocation for those containers from inside the global zone using e.g. processor pools, the fair share scheduler, and all the other resource management and resource accounting capabilities Solaris can bring to bear. Making the hypervisor configuration export a relatively static set of virtual hardware resources is a workaround for some of the more problematic aspects of two (or more) resource schedulers fighting over the same resource e.g. a multiprocessor CPU scheduler in the hypervisor at odds with an multiprocessor CPU scheduler in the guest, with the latter unaware of the presence of the former.
Of course in this picture, all the operating systems run as hypervisor guests, and thus all of them lose a bit of performance to the overheads associated with hardware virtualization technologies. And these overheads can be non-trivial for I/O intensive workloads - at least on todays x86 systems. And yet I think it's abundantly clear that the market is prepared to accept this overhead in exchange for the new management capabilities and business agility that a hypervisor-based virtualization appliance coupled with a sophisticated systems management solution brings; this observation is also at the core of our xVM Server and OpsCenter projects, as well as related efforts around both open source and proprietary hypervisors by several other companies. And as you'd expect, we're all working to reduce the overhead with hardware and software solutions, though some of the problems are thorny.
But is that the only useful virtualization technology combination?
Well, there's an interesting "flipped-over" version of this stack that the combination of Containers and a type 2 hypervisor like VirtualBox provides. Today you can run Solaris 10 or OpenSolaris as a host operating system directly "on the metal", running several Containers at the level efficiency and throughput that the container-style solutions bring, i.e. with no significant overhead involved. But what's new in the 1.6 release of VirtualBox is that you can now run other operating systems inside VirtualBox, with the VirtualBox hypervisor also running under the resource and namespace constraints of a Solaris Container i.e. hosted in a Solaris container.
As I noted in my "live" blog from Community One, VirtualBox 1.6 enables a new kind of Solaris container that can run any x86 OS that VirtualBox can support as a guest - and as people who've used VirtualBox know, that's a whole lot of different guest OSes. I was talking to a large customer two weeks ago about a workload dominated by a set of application servers. The application servers were already configured in containers; they'd done that because containers provided them with a convenient way to manage the app servers while consolidating the workload onto fewer machines. But they still have to have a number of Windows boxes to host some proprietary Windows applications that were a part of their stack that they hadn't yet found open source alternatives for. They'd tried running their entire workload on a commercial hypervisor platform, but the performance impact on their app server workload was just too high.
So what got both of us excited was the idea of putting those legacy Windows applications inside a VirtualBox container running Windows, because it meant the best of both worlds - their high performance applications will run as fast as native Solaris lets them - "on the metal" - while their low utilization legacy apps will run side-by-side in VirtualBox containers. The key point being that they would be paying for the overhead of hardware virtualization only where they needed to pay for it, not penalizing all the workloads running on the hardware. Which is also a reminder of an important consideration for assessing virtualization solutions - efficiency.
However, before anyone gets too excited, I have to point out that we're still working on integrating VirtualBox with the networking capabilities that arrive with the Crossbow project, so some of the server applications of this idea don't quite work yet. But if the purpose of the zone-hosted VirtualBox guest OS is to run as a client-side desktop, then the idea already works today. And that has an interesting impact on the Trusted Desktop.
So, let's combine VirtualBox, Containers and Trusted Extensions.
Recall that the Solaris Trusted Extensions technology that Glenn Faden has been blogging about here is built on zones. And what that means is that you can associate a label with a container, and because of VirtualBox hosted in a container, a label around an entire OS and the system resources it is consuming. All of which leads to the following screen shot that Christoph Schuba sent around internally 10 days after we announced the acquisition, and it was what Christoph was showing in one of the demos at the VirtualBox talk at Community One.
That's Vista, running in VirtualBox, hosted inside an OpenSolaris container with Trusted Extensions.
Spelling it out -- it's Vista, contained :)Technorati Tag: OpenSolaris
Technorati Tag: VirtualBox
Wednesday May 07, 2008
By tpm on May 07, 2008
I'm thrilled by the latest delivery of the Sun's OpenSolaris distro, and I was happy to see the "party" atmosphere at Community One on Monday. A lot of people have worked very hard to put the thing together, and everyone wanted to mark this event - both as the culmination of a lot of work, and as the beginning of a new phase of building out the distro with more content - both by Sun, our business partners, and by our communities. Thanks to everyone for their efforts.
If you haven't already given it a spin, try it out - it's a live CD image you can download from here.
Monday May 05, 2008
By tpm on May 05, 2008
Hmm. Always wondered if I could write a blog entry "live" from a conference session. Since I'm writing this entry "live", I'll apologise ahead of time for muddling my tenses as I move more into the present tense!
I'm here at community one listening to Joost Pronk and Achim Hasenmueller talking about the latest improvements and coolness in VirtualBox. Achim's team has just released version 1.6, which has an impressive list of features that can be found in the changelog right here.
- Hey, Jonathan just dropped by to say hello before the talk begins.
- Now Joost is explaining the context i.e. the different virtualization technologies that Sun offers and where VirtualBox fits in the picture.
- Next, Achim is introducing the core technology, the history, the design, how it's being used.
- Christoph Schuba is now doing a demo of Trusted Solaris multi-level security with Windows Vista running in VirtualBox, which is a neat combination of the Zones technology on which Trusted Solaris is based, and the VirtualBox type 2 hypervisor.
- Oh and Christoph is also running OpenSolaris native on a MacBook - because, as Christoph says, OpenSolaris runs on lots of machines. Very, very, nice demo.
- Now Achim is walking through configuring and running OpenSolaris on VirtualBox. Oh cool, now he's showing off the latest guest additions for OpenSolaris, which means things like mouse pointer integration and seamless mode now work too.
- Now we're delving into the architecture a little, and Achim's talking about the new WSDL/SOAP interface to VirtualBox that's a part of 1.6 to allow other management tools to manipulate the hypervisor and its guests. The default graphical UI and the command line UI already use that interface; now the API is available to allow developers to do what they want with it.
- Achim's also particularly happy about the virtual SATA controller that's new for 1.6
- VirtualBox now runs on Solaris 10U4 and OpenSolaris too. Some nice integration with zfs and zpools, with integrated iSCSI support. Oh and VirtualBox is Zones aware too, and it can be used simultaneously in multiple zones.
- Mac support is getting better, but there's still more in the pipeline.
Now the Q&A begins
- Q: Virtual Machine migration tools?
- A: Better support for VMDK formats in the pipeline
- Q: Older hardware?
- A: Tend to run out of memory if you want to run modern guests! Otherwise, anything that's Pentium III or above works.
- Q: Compare and contrast with Xen?
- A: Xen is a type 1, VirtualBox is a type 2, Xen emphasis on paravirtualization, VirtualBox focus on usability.
- Q: What kind of shared storage for live migration
- A: No specific restrictions at this point - files, devices, or iSCSI targets
Now some roadmap items that are being worked on
- Memory ballooning
- 64-bit guests
- Live migration
- 3D virtualization
- More portable snapshots
- VMDK support
- VHD support
- Nested paging for AMD-V and VT-x
- Next generation seamless windowing with better desktop integration
- Paravirtualization using VMI and Windows Enlightenments
You can download VirtualBox 1.6 by following the downloads link from virtualbox.org.Technorati Tag: VirtualBox
Thursday May 01, 2008
Wednesday Mar 26, 2008
By tpm on Mar 26, 2008
I gave the OpenSolaris Virtualization Technologies talk at Sun TechDays in Sydney, and there were a couple of slides from that I really wanted to post here as this stuff continues to surprise people. First, let's go back a few years. When we first started exploring the technology that became Solaris Zones, we quickly realized it was a very powerful idea, and while we were creating the basic version for Solaris 10's initial release, Glenn and team went off to implement Trusted Solaris using Zones. A little later, Nils and team went off and implemented "Brand Z" - a general mechanism to allow a Zone to have a completely different system call interface, which in turn allows a zone to have a completely different OS personality, and for example, to allow us to run the userland components of a Linux distribution in a zone.
The most recent application though, is one that has some interesting properties which illustrate some of the properties of virtualization more generally. That's Project Etude, which Dan Price and his colleagues delivered into Solaris 10 a few months ago - the essential concept being a Solaris 8 Zone running on a Solaris 10 system. This slide from marketing shows where the Brand Z personality component for Solaris 8 lives in the usual zones architecture picture.
Other than "because we can" :-) why did we do this? Well, the really compelling point associated with this technology is how we can move our Solaris 8 / SPARC customers forward from their older, under utilized, power-hungry hardware to the latest generation of power efficient SPARC hardware. Like the commodity SPARC microprocessors in the CoolThreads family of servers based on the Niagara processors. Looking from a 2008 vantage point, Solaris 8 seems like quite an old operating system now, and many of the machines it was originally installed on are from the same era. In the meantime Moore's law has advanced, and when combined with the CMT capabilities of Niagara, has enabled us to create very powerful, but extremely power efficient systems that can be used to consolidate multiple machines-worth of workload. There are obvious analogs in the legacy Windows / legacy x86 world that's powering the Windows consolidation move across the industry. But the thing I like about this customer case study below, is the actual numbers involved.
The 8800W and 275W numbers are power consumption comparisons, the BTU numbers are cooling costs. Smaller is better! These are real cost and power savings, and I think it illustrates that maybe there's some interesting comparative work to do here - comparing the efficiency of consolidation using different virtualization technologies e.g. zones vs. hypervisor virtualization technologies. A complex problem to characterize, yes. And interesting trade-offs between isolation, management complexity, business agility and performance. But our repeated experience is that this path is proving very effective for our customers.
Final reflection - I remember wondering why I had to learn all about classical thermodynamics in my engineering courses. What did that have to do with computers and communications technology? And now the irony is that for many of our datacenter customers, their number one concern is the basics of classical thermodynamics and the brutal economics associated with it - getting ever more computational work done while saving power and reducing the need for cooling.Technorati Tag: OpenSolaris
Technorati Tag: Solaris
Wednesday Mar 19, 2008
By tpm on Mar 19, 2008
So, here I am still on the road, starting to write this in Stuttgart airport, after spending a couple of days with the VirtualBox team. We've been introducing them to Sun, and doing a first pass on the roadmap ahead. The usual balance of lots of things we'd like to do, with software physics nipping at our heels - there's always more to do than we have people or time for!
Talking about VirtualBox in Australia was great fun; I introduced it to the audience of my keynote, which as I remarked in my earlier post was one of the largest audiences I've presented to. The full talk should eventually make it to the techdays site, but there are a couple of slides I wanted to review here now for people that can't get them and don't have time to navigate the site to find them. Basically, my talk was about Open Source Virtualization and Project Indiana, and how the two are connected. At first sight, one might imagine that they're not particularly connected at all. VirtualBox is an open source desktop hypervisor, a key component in our virtualization technology portfolio, but otherwise only related to OpenSolaris as a possible host and guest OS. VirtualBox is part of a more general application lifecycle picture involving xVM server and xVM Ops Center, as is nicely captured in this marketing graphic.
On the other hand, Project Indiana is, among other things, about bringing the technology of distribution to the OpenSolaris technology code base. This graphic from Ian Murdock's presentation at the Hyderbad techdays conference captures this notion of the OpenSolaris core surrounded by additional, up-to-date, packages from various open source communities nicely.
And this one summarizes the Indiana project goals:
So what is the connection between them? The connection is in the technology called the distro builder. This is the part the takes the recipe for a distribution that performs a particular set of functions and capabilities, and translates it into the right set of interdependent packages that can perform that function. At first sight, this seems a bit of a niche interest too. After all, relatively few people want to build OS distros .. surely? Well, it's more than you might think. In fact, if you take a look at many IT organizations, they tend to build something very similar for their internal use as a way of reducing costs. They rarely if ever take a vanilla OS install from a vendor. There's always some customization; packages added, removed, administrative pre-configuration of various kinds. So one important target for our distro builder technology are those IT organizations. Another important target are the developers creating virtual appliances to demonstrate their technologies. You can find these virtual appliances on lots of companies web sites these days - usually a file-based image that you can download and quickly instantiate on top of a hypervisor. The advantage being that you -don't- have to install and configure the software you're interested in exploring directly on your desktop operating system - you run it inside a preconfigured OS as a virtual machine. Then when you tire of the demo, you can discard it quickly, and completely, by discarding the virtual machine. So here's the graphic I kludged up to represent how the Indiana project and VirtualBox fit into that picture.
On the left side is the distro builder, on the right side is the developer to deployer flow using VirtualBox on the desktop, and xVM Server and xVM OpsCenter in the data center. If you want to find out more about the IPS packaging system that underpins this picture, then I'd recommend starting on Stephen Hahn's blog.
An open question we've pondered for a while is when virtual appliances will move beyond internal-use and demo-ware to be a mainstream style of SW distribution. There are several vendors who would like this to be true sooner than later. But the question is open for several reasons, performance is a universal concern, licensing of well-known proprietary operating systems in this context is another issue. Another important barrier is how to resolve the support issues for all the software involved that the creator of the application, or facility being demonstrated, didn't write. Let's think about a more concrete example. Assume I'm a developer at a small company, and I create a compelling application that uses various application infrastructure components e.g. an application server, a database, and, e.g. a chunk of ruby-on-rails to do it's work. Then I install and configure it all on an OS image of my choosing, make it into a virtual appliance, then deliver it to my customers that way. That packaging and configuration work I did is a clear advantage to my customer in terms of getting started with the application and exploring the value of the offering - which is one of the reasons vendors are using it more and more for demoware. But having delivered a complete package that way, it seems like as well as being responsible for the defects in my application, I'm now -also- responsible for all the defects in the customized stack I created too. Do I now have to start worrying about all the security patches I might need to include in my virtual appliance? Eeek! That seem really hard. We probably need a better answer to that support question that involves the vendors of the components I used before delivering fully-customized yet fully supported software stacks as virtual appliances really takes off.Technorati Tag: OpenSolaris
Technorati Tag: VirtualBox
Wednesday Mar 05, 2008
By tpm on Mar 05, 2008
Hmm. My first blog entry on the road; writing this in terminal 3 in Sydney airport with my battery running out. Let's see how well this works.
There are quite a number of Sun software engineers in Australia this week. We've been holding one of our TechDays developer conferences here in Sydney. If you get a chance to attend a TechDay in your city or within reach, I do recommend it: lots of information, lots of cool demos and giveaways, and real engineers presenting interesting talks about what they're passionate about. As an example, I heard George Wilson's excellent talk about the work going on in the opensolaris storage community for the first time today - and was intrigued by how interested the audience was in the ZFS-aware installer demo he showed. It's very clear that engineers all over the world are excited by ZFS.
I had a small part in the proceedings too - I did the community keynote on Wednesday morning, and talked about virtualization, Project Indiana, and the connections between them. Probably the largest audience I've ever spoken too; rather daunting; I hope it made some kind of sense to people. And this afternoon I did an opensolaris virtualization technologies talk. In between those events, I've been flying around Australia, visiting universities in Sydney, Melbourne, Canberra and (tomorrow) Brisbane. As you can tell from the title of this post, I'm a little dizzy. But as you'd probably expect, I just can't resist talking about Sun xVM, and VirtualBox and doing demos wherever I go. A few people at UTS had heard of VirtualBox, and were simply pleased to hear about the acquisition, the evolving roadmap, and our investment in this technology. Most of the others hadn't heard of it, but were definitely interested. At the conference, I started seeing it on everyone's desktop I caught sight of.
Don Kretsch, and Liang Chen are also visiting the Universities, talking about HPC and HPC Tools which has sparked a lot of interesting conversations. Josh Marinacci was also with us in Sydney, talking to the students about Dynamic Languages. He showed some fun JavaFX demos - you can find them on Josh's blog.
Though it's been tough to keep track of all the people I've met, it's been wonderful meeting students and faculty across the country. Really smart people who are interested in the technologies Sun is working on around Virtualization, HPC and Dynamic Languages. And we were there to listen to them talk about the technical and scientific problems they're working on - mostly around HPC, but also in other areas e.g. the challenges of scale and parallel programming presented by multicore architectures, complex real-time systems, and more.
As an engineer, it's always been important to me to build real things that other people find useful. That's the ultimate intellectual reward for me, and I think for most of the other engineers at Sun. It's not just about sharing and community in some remote, abstract sense, it's about making positive, real, engineering contributions to a community that can then use them. I think that's fundamentally what makes all engineers tick. So, this morning at ANU, I was surprised and pleased to hear of some work they've been doing, assessing the effectiveness of the MPO subsystem (memory placement optimizations, aka interfaces to describe NUMA machines) we built in OpenSolaris for Opteron-based systems. And in particular, they'd been using the lgroup abstraction, which I had a small hand in the architecture and initial design of a few years ago. So it was great to see the lgroup API being used, the implementation assessed against real needs, and found to be doing well; being used just how we hoped it would. And I'm looking forward to connecting these graduate students and their results with my colleagues Jonathan Chew and Bart Smaalders who put the hard work in designing and implementing the MPO code - I know they'll be interested in the results, and looking for ways to make MPO even better.
I think I'm completely smitten with Australia the country. It's my first visit here, and even though I haven't seen that much beyond CS department seminar rooms, the Darling Harbour Convention center, and the insides of various airports, hotels and taxis, I'm really drawn to the people I've met, and the comfortable feel of the culture. I need to come back again on a family vacation so we can sample the great outdoors too, and imbibe the history (and some of the wine). It's also looks to be a very beautiful place outside of the cities - even though I've only seen tantalizing glimpses on this trip.
Technorati Tag: VirtualBox
Tuesday Feb 12, 2008
By tpm on Feb 12, 2008
It's official. We just announced our intent to acquire innotek - a small company in Germany with (a) some very smart people and (b) some very significant technology, called VirtualBox. What is VirtualBox? Well, if you're a hypervisor engineer, then it's best explained as a high performance type 2 hypervisor that uses a combination of virtualization techniques to run many different unmodified operating systems in x86 virtual machines. It's highly portable across multiple hosts and supports a wide range of guest operating systems.
But perhaps that's a bit dry. And you don't need to be a hypervisor engineer to find it extremely useful.
Think of it this way. If you download and install VirtualBox on your laptop - running Windows, MacOS X, Linux or OpenSolaris, you can then run most any other popular Operating System on the same machine. Or several at the same time, depending on what hardware resources are available. The download is around 25Mbytes on most platforms. And what's truly cool about it for developers is that the download is free for personal use, and the code for VirtualBox is GPLv2 open source. So as well as VirtualBox being a cool product and a powerful set of technologies, it's also a community, and a great fit with Sun's broader open source strategies.
We think this tool is incredibly useful for developers - because most developers want to target multiple operating systems to maximise their audience and return on the time they've invested in their applications, and tools like VirtualBox let them do that by running everything - test environments, debug environments, etc. - on a single laptop. How does VirtualBox stack up against the other laptop and desktop options? Well I think it's great, but you don't have to take my word for it - there's a couple of great reviews here and here.
OpenSolaris and VirtualBox
My first conversation with the innotek engineering team was over a year ago. They told me about the work they'd been doing, what VirtualBox was capable of back then, where they were going, and how they'd just made it be an open source project. I was really impressed. And in many ways we've been working on a closer relationship ever since. Things really started to move quickly when we visited them last September. At that time, builds of OpenSolaris had already been working as guests, but after a marathon effort the night before we arrived at their offices, they managed to demonstrate OpenSolaris as a host for VirtualBox - a pretty significant capability for OpenSolaris. I took this screenshot during the first few hours of it working.
That's IE running inside in Windows XP in the foreground, displaying the opensolaris.org home page. The next day was even more exciting when they showed me seamless mode - applications running under XP sitting on the OpenSolaris desktop in the image below.
Windows Media Player running on the OpenSolaris desktop - whatever next?
For people running OpenSolaris, there's a new beta version of VirtualBox that's just been posted on the virtualbox.org site. Alternatively you can build it from source by following the instructions that Joe Bonasera posted on his blog.
Have fun with it!Technorati Tag: OpenSolaris
Technorati Tag: VirtualBox
Sunday Feb 10, 2008
By tpm on Feb 10, 2008
- I'm an AFOL - an Adult Fan Of Lego. I have far too much Lego at home, but it's impossible for me to resist the new sets every year. I just got a note from amazon.com telling me that a book my sister chose for me for Christmas just shipped - it has the coolest title - "Forbidden Lego: Build the Models Your Parents Warned You Against!" Though I don't recall my parents ever warning me ...
- I grew up in a tourist town on the coast of the north west of England. Pleasant enough, yes, but not really an intellectual hotspot. I still remember the first time I picked up and read Scientific American in W H Smiths - I was 13 and it was literally as if a whole new world opened up. First of all it contained a bunch of exotic-seeming ads for US technology companies and products. But most importantly, Martin Gardner's column, Mathematical Games, gave me a completely different perspective on what, up to that point, had been a difficult subject for me.
- A lot of my time at Cambridge as a graduate student was spent on EM fields - I guess I just got hooked on Maxwell's equations. So I was thrilled in 2002 to visit the Very Large Array in New Mexico with James, Rob Gingell, Jim Mitchell, Josh Simons and others. The VLA was designed in the late 1970's and used very long segments of circular waveguide - basically hollow pipes - to carry the signals from each antenna to the central facility. This particular waveguide was constructed from a single insulated strand of helical wire wound on the inside of an otherwise hollow tube. That construction allows a low-loss TE01 transmission mode to propagate down the guide, and prevent it's conversion to other, lossier modes. Before the trip I'd only ever thought about this in theoretical form, so it was very cool to suddenly be face-to-face with a huge deployment of this idea. But I remember my colleagues thinking I was a little bit strange to be so fascinated by this stuff. Hmm, there on the web you can find a performance evaluation of the system.
- Back in 1987 I was a Fellow of Clare College, Cambridge, and rowed in the Fellows VIII in the May Races. Since I hadn't rowed as an undergraduate, it involved learning how, and I made many early morning outings on the River Cam, getting up at 6am and cycling across town in the cold spring air. At the time I was still living the nocturnal life of the hacker, so the truly surprising part was that I stuck with it through to the day of the competition! Of course I don't remember if we were bumped, or if we bumped the boat in front of us, but I guess it's the taking part that counts.
- When I first started using Sun's software at Cambridge University, one of the things that really impressed me back then was the quality of the documentation - in particular the SunOS 4.1 manpages. We used to use them as a kind of definitive reference work, they certainly were a lot better than the offerings of the other vendors at the time. Nine years after I joined Sun, in a typical twist of geekdom, I married one of the writers that created them.
In the meantime, what else has been happening? Well, since the last entry, Bob Brewin and I have been working for Rich Green as the two CTOs responsible for Sun's Software technology portfolio. We also report to Greg Papadopolous. I like to joke that Bob handles everything beginning with J, and I do the rest - though in reality we work pretty closely, managing the technology portfolio, reviewing what we're doing and how we're doing it, listening to customers, identifying gaps and opportunities. I've also been continuing a special interest in Virtualization and the technology underlying the Indiana program. It has been, and continues to be sometimes frustrating, sometimes fun but almost always interesting. Though the pace is becoming frenetic as more and more opportunity comes our way as Sun's software business expands - the truly cool mySQL acquisition being the most recent example of that.
Friday Jul 14, 2006
By tpm on Jul 14, 2006
New capabilities include OpenSolaris as domain 0 - currently the controlling domain, and I/O domain for Xen-based systems - as well as both 32-bit and 64-bit kernels, diskful and diskless guests, and up to 32-way virtual SMP.
It's been a lot of work by a small team of dedicated people. It's still a work-in-progress, there are bugs, things we haven't finished yet etc., but we thought it was a good time to share where we are so far with the OpenSolaris developer community.
Technorati Tag: OpenSolaris
Monday Feb 13, 2006
By tpm on Feb 13, 2006
There are many bugs still in waiting, many puzzles to be solved, many things left to do.
A true work in progress.
Because we don't believe the developer community only wants finished projects to test. We believe that some developers want to participate during the development process, and now this project can open its doors to that kind of participation.
We wanted to start the conversation with working code. So we have a snapshot of our development tree for OpenSolaris on Xen, synced up with Nevada build 31. That code snapshot should be able to boot and run on all the hardware that build 31 can today, plus it can boot as a diskless unprivileged domain on Xen 3.0. While we were in our final approach to this release, we got live migration to work too, which is one of the key features we've been working on.
Running on Xen, OpenSolaris is reasonably stable, but it's still very much "pre-alpha" compared with our usual finished code quality. Installing and configuring a client is do-able, but not for the faint of heart.
To find out more, see the OpenSolaris on Xen community
Technorati Tag: OpenSolaris
Technorati Tag: Solaris