X

Leaving Oracle

I joined the Solaris Kernel Group in 2001 at what turned out to be a remarkable place and time for the industry. More by luck and intuition than by premonition, I found myself surrounded by superlative engineers working on revolutionary technologies that were the products of their own experience and imagination rather than managerial fiat. I feel very lucky to have worked with Bryan andMike on DTrace; it was amazing that just down the hall our colleagues reinventedthe...

Wednesday, August 18, 2010 | Fishworks | Read More

Fishworks history of SSDs

This year's flash memory summit got me thinking about our use of SSDs over the years at Fishworks. The picture of our left is a visual history of SSD evals in rough chronological order from the oldest at the bottom to the newest at the top (including some that have yet to see the light of day). Early Days When we started Fishworks, we were inspired by the possibilities presented by ZFS and Thumper. Those components would be key building blocks in the enterprise storage solution...

Tuesday, August 17, 2010 | Fishworks | Read More

Farewell to Bryan Cantrill

Bryan Cantrill, VP of Engineering at Joyent, earning $15. I've been expecting this automated mail for a while now, but it was disheartening nonetheless: List: dtrace-discuss Member: bryan.cantrill@eng.sun.com Action: Subscription disabled. Reason: Excessive or fatal bounces. As one of the moderators of the DTrace discussion list, I see people subscribe and unsubscribe. Bryan has, of course, left Oracle and joined Joyent to be their VP of...

Thursday, August 12, 2010 | DTrace | Read More

What is RAID-Z?

The mission of ZFS was to simplify storage and to construct an enterprise level of quality from volume components by building smarter software — indeed that notion is at the heart of the 7000 series. An important piece of that puzzle was eliminating the expensive RAID card used in traditional storage and replacing it with high performance, software RAID. To that end, Jeff invented RAID-Z; it's key innovation over other software RAID techniques was to close the "RAID-5 write...

Wednesday, July 21, 2010 | ZFS | Read More

A Logzilla for your ZFS box

A key component of the ZFS Hybrid Storage Pool is Logzilla, a very fast device to accelerate synchronous writes. This component hides the write latency of disks to enable the use of economical, high-capacity drives. In the Sun Storage 7000 series, we use some very fast SAS and SATA SSDs from STEC as our Logzilla &mdash the devices are great and STEC continues to be a terrific partner. The most important attribute of a good Logzilla device is that it have very low latency for...

Monday, July 19, 2010 | ZFS | Read More

2010.Q1 simulator

On the heels of the 2010.Q1 software release, we've provided a new version of the Sun Storage 7000 simulator that can be found at this new location. As noted previously, the simulator is a terrific way to take the Sun Storage 7000 user interface for a spin; it includes the exact same software as a physical 7000 series system with the same features. The last release of the simulator added support for VirtualBox; this release now removes support for VMware. VMware was...

Wednesday, March 31, 2010 | Fishworks | Read More

The need for triple-parity RAID

When I first wrote about triple-parity RAID in ZFS and the Sun Storage 7000 series, I alluded a looming requirement for triple-parity RAID due to a growing disparity between disk capacity and throughput. I've written an article in ACM Queue examining this phenomenon in detail, and making the case for triple-parity RAID. Dominic Kay helped me sift through hard drive data for the past ten years to build a model for how long it takes to fully populate a drive. I've reproduced a...

Monday, December 21, 2009 | Fishworks | Read More

Logzillas: to mirror or stripe?

The Hybrid Storage Pool integrates flash into the storage hierarchy in two specific ways: as a massive read cache and as fast log devices. For read cache devices, Readzillas, there's no need for redundant configurations; it's a clean cache so the data necessarily also resides on disk. For log devices, Logzillas, redundancy is essential, but how that translates to their configuration can be complicated. How to decide whether to stripe or mirror? ZFS intent log devices Logzillas...

Wednesday, December 9, 2009 | Fishworks | Read More

2009.Q3 Storage Configuration

Today we shipped our 2009.Q3 release. Amidst the many great new features, enhancements and bug fixes, we've added new storage profiles for triple-parity RAID and three-way mirroring. Here's an example on a 9 JBOD system of what you'll see in the updated storage configuration screen: Note that the new Triple parity RAID, wide stripes option replaces the old Double parity RAID, wide stripes configuration. With RAID stripes that can easily be more than 40 disks wide, and resilver...

Wednesday, September 16, 2009 | Fishworks | Read More

Flash Memory Summit 2009

At the Flash Memory Summit today, Sun's own Michael Cornwell delivered a keynote excoriating the overall direction of NAND flash and SSDs. In particular, he spoke of the "lithography death march" as NAND vendors push to deliver the most cost-efficient solution while making huge sacrifices in reliability and performance. On Wednesday, August 12, I'll be giving two short talks as part of sessions on flash-enabled power savings and data center applications: 8:30 - 9:45 Power Saving...

Wednesday, August 12, 2009 | Fishworks | Read More

Triple-Parity RAID-Z

Double-parity RAID, or RAID-6, is the de facto industry standard for storage; when I started talking about triple-parity RAID for ZFS earlier this year, the need wasn't always immediately obvious. Double-parity RAID, of course, provides protection from up to two failures (data corruption or the whole drive) within a RAID stripe. The necessity of triple-parity RAID arises from the observation that while hard drive capacity has roughly followedKryder's law, doubling annually,...

Tuesday, July 21, 2009 | ZFS | Read More

Sun Storage 7310

Today we're introducing a new member to the Sun Unified Storage family: the Sun Storage 7310. The 7310 is a scalable system from 12TB with a single half-populated J4400 JBOD up to 96TB with 4 JBODs. You can combine two 7310 head units to form a cluster. The base configuration includes a single quad-core CPU, 16GB of DRAM, a SAS HBA, and two available PCIe slots for NICs, backup cards, or the Fishworks cluster card. The 7310 can be thought of as a smaller capacity, lower cost...

Wednesday, May 27, 2009 | Fishworks | Read More

Mirroring flash SSDs

As flash memory has become more and more prevalent in storage from the consumer to theenterprise people have been charmed by the performance characteristics, but get stuck on the longevity. SSDs based on SLC flash are typically rated at 100,000 to 1,000,000 write/erase cycles while MLC-based SSDs are rated for significantly less. For conventional hard drives, the distinct yet similar increase in failures over time has long been solved by mirroring (or other redundancy...

Tuesday, May 26, 2009 | Fishworks | Read More

SS 7000 simulator update plus VirtualBox

On the heels of the 2009.Q2.0.0 release, we've posted an update to the Sun Storage 7000 simulator. The simulator contains the exact same software as the other members of the 7000 series, but runs inside a VM rather than on actual hardware. It supports all the same features, and has all the same UI components; just remember that an actual 7000 series appliance is going to perform significantly better than a VM running a puny laptop CPU. Download the simulator here. The new...

Monday, May 4, 2009 | Fishworks | Read More

Sun Storage 7000 simulator upgrade

Today we released version 2009.Q2.0.0, the first major software update for the Sun Storage 7000 series. It includes a bunch of new features, bug fixes, and improvements. Significantly for users of the Sun Storage 7000 simulator, the virtual machine version of the 7000 series, this is the first update that supports the VMs. As with a physical 7000 series appliance, upgrade by navigating to Maintenance > System, and click the + icon next to Available Updates. Remember not to...

Monday, April 27, 2009 | Fishworks | Read More

SSDs for HSPs

We're announcing a couple of new things in the flash SSD space. First, support the Intel X25-E SSD in a bunch of our servers. This can be used to create a Hybrid Storage Pool like in the Sun Storage 7000 series, or as just a little flash for high performance / low power / tough environmentals. Second, we're introducing a new open standard with the Open Flash Module. This creates a new form factor for SSDs bringing flash even closer to the CPU for higher performance and tighter...

Tuesday, March 10, 2009 | Fishworks | Read More

Presentation: Hybrid Storage Pools and SSDs

Today at The First Workshop on Integrating Solid-state Memory into the Storage Hierarchy (WISH 2009) I gave a short talk about our experience integrating flash into the storage hierarchy and the interaction with SSDs. In the talk I discussed the recent history of flash SSDs as well as some key areas for future improvements. You can download it here. The workshop was terrific with some great conversations about the state of solid state storage and its future directions; thank...

Saturday, March 7, 2009 | Fishworks | Read More

Fishworks VM: the 7000 series on your laptop

In May of 2007 I was lined up to give my first customer presentation of what would become the Sun Storage 7000 series. I inherited a well-worn slide deck describing the product, but we had seen the reactions of prospective customers who saw the software live and had a chance to interact with features such as Analytics; no slides would elicit that kind of response. So with some tinkering, I hacked up ourinstaller and shoe-horned the prototype software into a virtual machine....

Friday, March 6, 2009 | Fishworks | Read More

More from the storage anarchist

In my last blog post I responded to Barry Burke author of the Storage Anarchist blog. I was under the perhaps naive impression that Barry was an independent voice in the blogosphere. In fact, he's merely Storage Anarchist by night; by day he's the mild-mannered chief strategy officer for EMC's Symmetrix Products Group — a fact notable for its absence from Barry's blog. In my post, I observed that Barry had apparently picked his horse in the flash race and Chris Caldwell comme...

Monday, March 2, 2009 | Fishworks | Read More

Dancing with the Anarchist

Barry Burke, the Storage Anarchist, has written an interesting roundup ("don't miss the amazing vendor flash dance") covering the flash strategies of some players in the server and storage spaces. Sun's position on flash comes out a bit mangled, but Barry can certainly be forgiven for missing the mark since Sun hasn't always communicated its position well. Allow me to clarify our version of the flash dance. Barry's conclusion that Sun sees flash as well-suited for the server...

Thursday, February 26, 2009 | Fishworks | Read More

HSP talk at the OpenSolaris Storage Summit

The organizers of the OpenSolaris Storage Summit asked me to give a presentation about Hybrid Storage Pools and ZFS. You can download the presentation titled ZFS, Cache, and Flash. In it, I talk about flash as a new caching tier in the storage hierarchy, some of the innovations in ZFS to enable the HSP, and an aside into the how we implement an HSP in the Sun Storage 7410.

Monday, February 23, 2009 | Fishworks | Read More

Flash workshop at ASPLOS

Before this year's ASPLOS conference, I'll be speaking at the First Workshop on Integrating Solid-state Memory into the Storage Hierarchy (WISH2009). It looks like a great program with some terrific papers on how to use flash effectively and how to combine various solid state technologies to complement conventional storage. I'll be talking about the work we've done at Sun on the Hybrid Storage Pool. In addition I'll discuss some of the new opportunities that flash and other...

Thursday, February 19, 2009 | Fishworks | Read More

Casting the shadow of the Hybrid Storage Pool

The debate, calmly waged, on the best use of flash in the enterprise can be summarized as whether flash should be a replacement for disk, acting as primary storage, or it should be regarded as a new, and complementary tier in the storage hierarchy, acting as a massive read cache. The market leaders in storage have weighed in the issue, and have declared incontrovertibly that, yes, both are the right answer, but there's some bias underlying that equanimity.Chuck Hollis, EMC's...

Monday, December 1, 2008 | Fishworks | Read More

Sun Storage 7410 space calculator

The Sun Storage 7410 is our expandable storage appliance that can be hooked up to anywhere from one and twelve JBODs with 24 1TB disks. With all those disks we provide the several different options for how to arrange them into your storage pool: double-parity RAID-Z, wide-strip double-parity RAID-Z, mirror, striped, and single-parity RAID-Z with narrow stripes. Each of these options has a different mix of availability, performance, and capacity that are described both in the...

Wednesday, November 19, 2008 | Fishworks | Read More

Hybrid Storage Pools in the 7410

The Sun Storage 7000 Series launches today, and with it Sun has the world's first complete product that seamlessly adds flash into the storage hierarchy in what we call the Hybrid Storage Pool. The HSP represents a departure from convention, and a new way of thinking designing a storage system. I'vewritten before about the principles of the HSP, but now that it has been formally announced I can focus on the specifics of the Sun Storage 7000 Series and howit implements the...

Monday, November 10, 2008 | Fishworks | Read More

Hybrid Storage Pool goes glossy

I've written about Hybrid Storage Pools (HSPs) here several times as well as in an article that appeared in the ACM's Queue and CACM publications. Now the folks in Sun marketing on the occasion of our joint SSD announcement with Intel have distilled that down to a four page glossy, and they've done a terrific job. I suggest taking a look. The concept behind the HSP is a simple one: combine disk, flash, and DRAM into a single coherent and seamless data store that makes optimal...

Monday, October 20, 2008 | Fishworks | Read More

Apple updates DTrace... again

Back in January, I ranted about Apple's ham-handed breakage in their DTrace port. After some injured feelings and teary embraces, Apple cleaned things up a bit, but some nagging issues remained as I wrote: For the Apple folks: I'd argue that revealing the name of otherwise untraceable processes is no more transparent than what Activity Monitor provides — could I have that please? It would be very un-Apple to — you know — communicate future development plans, but in 10.5.5,...

Saturday, October 4, 2008 | DTrace | Read More

A glimpse into Netapp's flash future

The latest edition of Communications of the ACM includes apanel discussion between "seven world-class storage experts". The primary topic was flash memory and how it impacts the world of storage. The most interesting comment came from Steve Kleiman, Senior Vice President and Chief Scientist at Netapp: My theory is that whether it’s flash, phase-change memory, or something else, there is a new place in the memory hierarchy. There was a big blank space for decades thatis now...

Monday, August 11, 2008 | Fishworks | Read More

Hybrid Storage Pools: The L2ARC

I've written recently about the hybrid storage pool (HSP), using ZFS to augment the conventional storage stack with flash memory. The resulting system improve performance, cost, density, capacity, power dissipation — pretty much evey axis of importance. An important component of the HSP is something called the second level adaptive replacement cache (L2ARC). This allows ZFS to use flash as a caching tier that falls between RAM and disk in the storage hierarchy, and permitshuge...

Wednesday, July 23, 2008 | Fishworks | Read More

Hybrid Storage Pools in CACM

As I mentioned in my previous post, I wrote an article about the hybrid storage pool (HSP); that article appears in the recently released July issue of Communications of the ACM. You can find it here. In the article, I talk about a novel way of augmenting the traditional storage stack with flash memory as a new level in the hierarchy between DRAM and disk, as well as the ways in which we've adapted ZFS and optimized it for use with flash. So what's the impact of the HSP? Very...

Tuesday, July 1, 2008 | Fishworks | Read More

Flash, Hybrid Pools, and Future Storage

Jonathan had a terrific post yesterday that does an excellent job of presenting Sun's strategy for flash for the next few years. With my colleagues at Fishworks, an advanced product development team, I've spent more than a year working with flash and figuring out ways to integrate flash into ZFS, the storage hierarchy, and our future storage products — a fact to which John Fowler, EVP of storage, alluded recently. Flash opens surprising new vistas; it's exciting to see Sun...

Wednesday, June 11, 2008 | Fishworks | Read More

Apple updates DTrace

Back in January, I posted about a problem with Apple's port of DTrace to Mac OS X. The heart of the issue is that their port would silently drop data such that certain experiments would be quietly invalid. Unfortunately, most reactions seized on a headline paraphrasing a line of the post — albeit with the critical negation omitted (the subject and language were, perhaps, too baroque to expect the press to read every excruciating word). The good news is that Apple has...

Saturday, June 7, 2008 | DTrace | Read More

dtrace.conf post-post-mortem

This originally was going to be a post-mortem on dtrace.conf, but so much time has passed, that I doubt it qualifies anymore. Back in March, we held the first ever DTrace (un)conference, and I hope I speak for all involved when I declare it a terrific success. And our t-shirts (logo pictured) were, frankly, bomb. Here are some fairly random impressions from the day: Notes on the demographics at dtrace.conf: Macs were the most prevalent laptops by quite a wide margin, and a ton...

Monday, May 5, 2008 | DTrace | Read More

DTrace and JavaOne: The End of the Beginning

It was a good run, but Jarod and I didn't make the cut for JavaOne this year... 2005 In 2005, Jarod came up with what he described as a jacked up way to use DTrace to get inside Java. This became the basis of the Java provider (first dvm for the 1.4.2 and 1.5 JVMs and now the hotspot provider for Java 6). That year, I got to stand up on stage at the keynote with John Loiacono and present DTrace for Java for the first time (to 10,000 people -- I was nervous). John was then the...

Thursday, April 10, 2008 | DTrace | Read More

Expand-O-Matic RAID-Z

I was having a conversation with an OpenBSD user and developer the other day, and he mentioned some ongoing work in the community to consolidate support for RAID controllers. The problem, he was saying, was that each controller had a different administrative model and utility -- but all I could think was that the real problem was the presence of a RAID controller in the first place! As far as I'm concerned, ZFS and RAID-Z have obviated the need for hardware RAID controllers. ZF...

Monday, April 7, 2008 | ZFS | Read More

pid2proc for DTrace

The other day, there was an interesting post on the DTrace mailing list asking how to derive a process name from a pid. This really ought to be a built-in feature of D, but it isn't (at least not yet). I hacked up a solution to the user's problem by cribbing the algorithm from mdb's ::pid2proc function whose source code you can find here. The basic idea is that you need to look up the pid in pidhash to get a chain of struct pid that you need to walk until you find the pid...

Thursday, March 13, 2008 | DTrace | Read More

Mac OS X and the missing probes

As has been thoroughly recorded, Apple has included DTrace in Mac OS X. I've been using it as often as I have the opportunity, and it's a joy to be able to use the fruits of our labor on another operating system. But I hit a rather surprising case recently which led me to discover a serious problem with Apple's implementation. A common trick with DTrace is to use a tick probe to report data periodically. For example, the following script reports the ten most frequently...

Friday, January 18, 2008 | DTrace | Read More

DTrace/Firefox/Leopard

It's been more than a year since I first saw DTrace on Mac OS X, and now it's at last generally available to the public. Not only did Apple port DTrace, but they've also included a bunch of USDT providers. Perl, Python, Ruby -- they all ship in Leopard with built-in DTrace probes that allow developers to observe function calls, object allocation, and other points of interest from the perspective of that dynamic language. Apple did make some odd choices (e.g. no Java provider,...

Saturday, October 27, 2007 | DTrace | Read More

What-If Machine: DTrace Port

What if there were a port of DTrace to Linux? What if there were a port of DTrace to Linux: could such a thing be done without violating either the GPL or CDDL? Read on before you jump right to the comments section to add your two cents. In my last post, I discussed an attempt to create a DTrace knockoff in Linux, and suggested that a port might be possible. Naively, I hoped that comments wouldexamine the heart of my argument, bemoan the apparent NIH in the Linux knockoff,regr...

Monday, August 6, 2007 | DTrace | Read More

DTrace Knockoffs

Update 8/6/2007: Those of you interested in this entry may also want to check outmy next entry on the legality of a hypothetical port of DTrace to Linux. Tools We Wish We Had -- OSCON 7/26/2007 Last week at OSCON someone set up a whiteboard with the heading "Tools We Wish We Had". People added entries (wiki-style); this one in particular caught my eye: dtrace for Linux or something similar (LIKE SYSTEMTAP?) - jdub (NO, LIKE dtrace) - VLAD (like systemtap, but not crap) DTrace So...

Thursday, August 2, 2007 | DTrace | Read More

DTrace for Ruby at OSCON 2007

I just got back from OSCON, a conference on Open Source that O'Reilly hosts in Portland annually. The conference offered some interesting content and side-shows with some notable highlights (more on those in the next few days). Brendan and I gave a presentation on how a crew from Sun dropped in on Twitter to help them use DTrace to discover some nasty performance problems. Here's the presentation along with the D scripts and load generators we used for the talk.

Tuesday, July 31, 2007 | DTrace | Read More

iSCSI DTrace provider and more to come

People often ask about the future direction of DTrace, and while we have some stuff planned for the core infrastructure, the future is really about extending DTrace's scope into every language, protocol, and application with new providers -- and this development is being done by many different members of the DTrace community. An important goal of this new work is to have consistent providers that work predictably. To that end, Brendan and I have started to sketch out an array...

Wednesday, July 4, 2007 | DTrace | Read More

DTrace @ JavaOne 2007

This year, Jarod Jenson and I gave an updated version of our DTrace for Java (technology-based applications) talk: The biggest new feature that we demonstrated is the forthcoming Java Statically-Defined Tracing (JSDT) which will allow developers to embed stable probes in their code as we can do today in the kernel with SDT probes and in C and C++ applications with USDT probes. While you can already trace Java applications (and C and C++ applications), static probes let the...

Monday, May 28, 2007 | DTrace | Read More

Java/DTrace article

The Texas Ranger himself, Jarod Jenson, has written a nice article about using the new DTrace probes in Java SE 6. If that's up your alley, you should come to the talk Jarod and I will be giving at JavaOne in May. We'll be talking about some of the new features in Java SE 6 and potentially previewing some new features slated for Java SE 7. This will be our third year at JavaOne -- it's great to see how much progress we're making each year. Technorati Tags:

Thursday, March 22, 2007 | DTrace | Read More

Linux Defection

Ian Murdock has left the Linux Foundation to lead the operating systems strategy here at Sun. The last few years have seen some exciting changes at Sun: releasing Solaris 10 (which includes several truly revolutionary technologies), embracing x86, leading on x64, and taking Solaris open source. That a luminary of the Linux world was enticed by the changes we've made and the technologies we're creating is a huge vote of confidence. From my (admittedly biased) view, OpenSolaris...

Monday, March 19, 2007 | OpenSolaris | Read More

gzip for ZFS update

The other day I posted about a prototype I had created that adds a gzip compression algorithm to ZFS. ZFS already allows administrators to choose to compress filesystems using the LZJB compression algorithm. This prototype introduced a more effective -- albeit more computationally expensive -- alternative based on zlib. As an arbitrary measure, I used tar(1) to create and expand archives of an ON (Solaris kernel) source tree on ZFS filesystems compressed with lzjb and gzip...

Wednesday, January 31, 2007 | ZFS | Read More

a small ZFS hack

I've been dabbling a bit in ZFS recently, and what's amazing is not just how well it solved the well-understood filesystem problem, but how its design opens the door to novel ways to manage data. Compression is a great example. An almost accidental by-product of the design is that your data can be stored compressed on disk. This is especially interesting in an era when we have CPU cycles to spare, many too few available IOPs, and disk latencies that you can measure with a...

Monday, January 29, 2007 | ZFS | Read More

It's tested or it's broken

It's amazing how lousy software is. That we as a society have come to accept buggy software as an inevitability is either a testament to our collective tolerance, or -- much more likely -- the near ubiquity of crappy software. So we are guilty of accepting low standards for software, but the smaller we of software writers are guilty of setting those low expectations. And I mean we: all of us. Every programmer has at some time written buggy software (or has never written any...

Wednesday, December 13, 2006 | Software | Read More

DTrace: a history

An unsurprisingly common request on theDTrace discussion forum has been for updated documentation. People have been -- on the whole -- very pleased with theSolaris Dynamic Tracing Guide that we worked hard to produce, but I readily admit that we haven't been nearly as diligent in updating it. OK: we haven't updated it at all. But we have been updating DTrace itself, adding new variables and functions, tacking on new features, adding new providers, and fixing bugs. But unless...

Thursday, November 2, 2006 | DTrace | Read More

DTrace is a web developer's best friend

I have this friend who might be most accurately described as a web developer. When DTrace was able to observe php he was interested. Me: "I should give you a demo some time." Him: "Absolutely..." When DTrace ticked Ruby off its list, he was more enthusiastic. Him: "Cool! I loves me the Ruby!" Me: "Let me know when you want that demo". The other day I got an IM from my friend. Him: "DTrace for JavaScript, eh?" Me: "How 'bout that, huh?" Him: "So when can I get that demo?" Last...

Friday, September 22, 2006 | DTrace | Read More

DTrace user number one

Some people think DTrace was built for developers; others think it was for system administrators; some even think it was a tool designed just for Solaris kernel hackers but was so useful we decided to unleash it on the world. All wrong. The user we always had in mind was Solaris user extraordinaire Jarod Jenson. DTrace let's you explore virtually any element of the system -- it's biggest limitation is the user's own knowledge of the system. Jarod has the most diverse and...

Wednesday, September 13, 2006 | DTrace | Read More

Apple's DTrace team

As Bryan wrote, Apple has ported DTrace to Mac OS X. The Apple kernel team invited us to the WWDC today for the (albeit muted) announcement of their DTrace support, and then for a demo and dinner. It was surprisingly fun to play with DTrace on another OS, and it was a true pleasure to talk to the Apple guys who worked on the port. And it's my pleasure to introduce those engineers to the DTrace community at large: James McIlree, Tom Duffy, Steve Peters, Terry Lambert (DTrace...

Monday, August 7, 2006 | DTrace | Read More

Double-Parity RAID-Z

When ZFS first started, it was just Jeff trying to pair old problems with new solutions in margins too small to contain either. Then Matt joined up to bring some young blood to the project. By the time the project putback, the team had grown to more than a dozen. And now I've been pulled in -- if only for a cameo. When ZFS first hit the streets, Jeff wrote about RAID-Z, an implementation of RAID designed for ZFS. RAID-Z improves upon previous RAID schemes primarily in that it...

Sunday, June 18, 2006 | ZFS | Read More

DTrace on Geek Muse

DTrace was recently featured on episode 35 of Geek Muse. DTrace was brought to their attention because of John Birrell's recent work to port it to FreeBSD (nice work, John!). The plug was nice, but I did want to respond to a few things: DTrace was referred to as "a scripting language for debugging". While I can understand why one might get that impression, it's kind of missing the point. DTrace, concisely, is a systemic observability framework that's designed explicitly for...

Friday, June 16, 2006 | DTrace | Read More

DTrace whatever contest

Calling all college students: do something with DTrace and win this fabulous prize! an XL Solaris fleece of unknown vintage and origin If you use DTrace to solve a problem, help with your work, or adjudicate a bet, enter for a chance to win. Rules: entries must be in the form of a blog post or other web page entries should be submitted as a comment or in email to me submitters must be able to be reasonably described as a college student there is only one prize, but I will pay to...

Sunday, June 4, 2006 | DTrace | Read More

JavaOne DTrace Session 2006

Jarod Jenson and I presented at JavaOne last week. If you're interested, here are the slides we used: Admittedly, a pdf can't capture the scene of Jarod typing in DTrace commands while I futilely try to narrate, but you can run your own demos by checking out this, this, and this. Technorati Tags:

Monday, May 22, 2006 | DTrace | Read More

DTrace caption contest

Two of the DTrace creators (that's me on the right, and Bryan on the left) apparently square off at the recent OpenSolaris user group meeting at JavaOne. You fill in the bubbles; the funniest response wins nothing but my heartfelt amusement.

Wednesday, May 17, 2006 | DTrace | Read More

DTrace at JavaOne 2006

At last year's JavaOne, DTrace enjoyed some modicum of celebrity, being featured in a keynote, a session, and the DTrace challenge. The session was attended by about 1,000 people (out of about 10,000 at the conference), and the DTrace Challenge -- which promised to find a performance win in Java applications brought to us or fork over an iPod -- went an impressive 15 for 16 (and that one outlier was a trivial 200 line program -- we've added a 2,000 line minimum this...

Friday, May 12, 2006 | DTrace | Read More

User-land tracing gets better and better

As I've mentioned in the past, developers can add their own DTrace probes using the user-land statically defined tracing (USDT) mechanism. It's been used to instrument Postgres and Apache, and to add observability into dynamic languages such as Java, Ruby, and php. I recently made a couple of improvements to USDT that I mentioned here and here, but I think deserve a little more discussion. Adding USDT probes (as described in the DTrace manual) requires creating a file...

Monday, May 8, 2006 | DTrace | Read More

DTrace at SCALE 4x

I gave a talk today at SCALE 4X describing how to use DTrace to understand Linux applications by using BrandZ. I admit I was a little leery about how a talk about a Solaris feature would go over at a Linux conference, but it was very well received. Here's my slide deck: For more, check out my blog post on the subject. Technorati Tags:

Sunday, February 12, 2006 | DTrace | Read More

DTrace in ACM Queue

While usually a shameless self-promoter, Bryan has been conspicuously silent about his recent article in ACM Queue. It's quite good, and goes beyond what has previously been written about DTrace in discussing its origins, the landscape of the problem, and some fundamental innovations embodied in DTrace -- information that may have been too obscure in our USENIX paper (according to at least one Stanford professor). Technorati Tags:

Friday, February 3, 2006 | DTrace | Read More

DTrace for Linux

With BrandZ, it's now possible to use DTrace on Linux applications. For the uninitiated, DTrace is the dynamic tracing facility in OpenSolaris; it allows for systemic analysis of a scope and precision unequalled in the industry. With DTrace, administrators and developers can trace low level services like I/O and scheduling, up the system stack through kernel functions calls, system calls, and system library calls, and intoapplications written in C and C++ or any of a host of...

Tuesday, December 13, 2005 | DTrace | Read More

OpenSolaris on LugRadio

The good folks over at LugRadio (that LUG; not the other LUG of course) invited me on their show to answer some of their questions about OpenSolaris. You can find it in the latest episode. Check your volume before you start playing the episode: the first reference to masturbation is approximately 6 seconds in. Understandably, the LugRadio guys didn't have much exposure to OpenSolaris, but they were certainly interested and recognize that it represents a pretty exciting...

Sunday, November 27, 2005 | OpenSolaris | Read More

Fall of Code?

Actually, Fall, Winter and Spring of code. Sun just announced the Solaris 10 Univesity Challenge Content. That's right, it's a challenge and a contest and with three times the staying power of single season of code. Apparently in a modern day treaty of Brest-Litovsk, Google ceded the rest of the year to Sun. Perhaps this was the real beef of the recent Sun/Google agreement. This is actually pretty cool: be a college student, do something cool on OpenSolaris, take a shot at...

Wednesday, October 19, 2005 | OpenSolaris | Read More

OpenSolaris and svk

Today at EuroOSCON, I attended a introductory talk on svn by Chia-liang Kao. I was hopeful that svk might address some of the issues that I thought would prevent us from adopting Subversion for OpenSolaris. In particular, Subversion requires a centralized repository whereas svk, which is built on top of Subversion, provides the distributed revision control system that we'd need. After the talk, my overall impression was that svk seemed to lack a certain polish, but after...

Wednesday, October 19, 2005 | OpenSolaris | Read More

OpenSolaris and Subversion

I just attended Brian W. Fitzpatrick's talk on Subversion at EuroOSCON. Brian did a great job and Subversion looks like a really complete replacement for cvs -- the stated goal of the project. What I was particularly interested in was the feasibility of using Subversion as the revision control system for OpenSolaris; according to the road map we still have a few months to figure it out, but, as my grandmother always said while working away at her mechanical Turing machine,...

Monday, October 17, 2005 | OpenSolaris | Read More

At Euro OSCON

This week I'll be in Amsterdam attending and speaking at Euro OSCON. I'll be running a session on DTrace on Thursday at 1:30. For the uninitiated, DTrace is the systemic observability framework in OpenSolaris that allows for tracing of the kernel and applications on production systems. At OSCON in Portland, Bryan and Wez extended DTrace into php (it already worked on C/C++ and Java); since then, the DTrace community has extended DTrace into Ruby and Perl -- we'll see if we can come up with something new this week. I'll also be tagging in on the OpenSolaris Users' Group Meeting and the OpenSolaris Technologies BoF along with Casper, Alan, Patrick, Simon, and others. I've had a bit of a chance to explore Amsterdam a bit and it's spectacular. My California-thinned blood won't let me forgo a comment on the rather cold weather, but my Northeastern upbringing won't let me complain too loudly. Technorati tags:

Sunday, October 16, 2005 | OpenSolaris | Read More

Too much pid provider

Perhaps it's a bit Machiavellian, but I just love code that in some way tricks another piece of code. For example, in college I wrote some code that trolled through the address space of my favorite game to afford me certain advantages. Most recently, I've been working on some code that tricks other code into believing a complete fiction[1] about what operating system it's executing on. While working on that, I discovered an interesting problem with the pid provider -- code...

Thursday, October 13, 2005 | DTrace | Read More

The mysteries of _init

I hadn't been fully aware that I felt this way, but I recently had a realization: I love the linker. It's a technology that's amazing in both its simplicity and its complexity. I'm sure my feelings are influenced in no small way by the caliber of the engineers working on it -- Rod and Mike are always eager to explain how the some facet of the linker works or to add something new and whizzy if it can't quite do what I need. Over the course of developinguser-levelstatically...

Thursday, September 15, 2005 | OpenSolaris | Read More

DTrace boot camp

I just got back from an extended DTrace world tour -- Prague, Watford, London, San Francisco, Los Angeles, and San Diego. Along the way, I gave a couple of workshops on using DTrace. The slides are far from finished[1], but if you're learning DTrace, they might be a good place to start. The workshop is designed to take you through DTrace gradually and includes many examples and exercises. If you have suggestions for how to improve the talk or content you'd like to see in...

Thursday, August 18, 2005 | DTrace | Read More

At the Unix Users' Group in LA

Next week I'll be continuing the DTrace world tour in LA and San Diego (see Matt's announcement). On Monday, August 15 from 7pm-9pm I'll be talk to the Unix Users' Group here: Sun Office in El Segundo 222 N. Sepulveda Blvd., 10th Floor El Segundo, CA 90245 My plan is to demonstrate DTrace (including the recent advancements in Java and php) and talk a bit about OpenSolaris. If there's anything in particular you'd like to hear about, feel free to leave a comment or just bring...

Wednesday, August 10, 2005 | DTrace | Read More

I didn't mean strace

I admit: some things bug me more than they probably should. One such annoyance which has caused me far more consternation than it was due has been Google's auto-suggest when searching for "DTrace" -- no, I didn't mean "strace", ok?! Perhaps more annoying (but admittedly a less frequent annoyance) was Google's (otherwise fantastic) SMS service: Did you mean "strace"? Because there were no matches for that either, FYI. Great; good to know; thanks for the...

Tuesday, August 9, 2005 | DTrace | Read More

Believe it or not...

I really never thought I'd be sticking a picture of a penguin on my weblog, but here it is... Joined by Bryan and Eric, I'll be speaking (and listening) at the OpenSolaris BoF at LinuxWorld (August 9, 5:30-7:00). We'll be talking about using and developing on OpenSolaris, how you can participate, and most likely comparing Solaris and Linux as well as giving the mandatory DTrace pitch. I expect it will be a little like the times I've been to Yankee stadium in full Red Sox re...

Tuesday, July 26, 2005 | OpenSolaris | Read More

DTrace User-Land

One of the primary motivations for DTrace was the absence of a framework that united observability into all aspects of the system. There were certainly tools for looking at the individual components (iostat(1) for I/O; mpstat(1) and prstat(1) for some basic system monitoring; truss(1), gdb(1), mdb(1) and dbx(1) for examining processes), but correlating the data from the disparate sources was difficult or impossible (tell that to a room of system administrators and they start...

Wednesday, July 20, 2005 | DTrace | Read More

DTrace Presentation at JavaOne

Thanks to everyone who attended Jarod's and my talk this afternoon at JavaOne and especially to those who had to go to the overflow room. The rest of the DTrace team and I were thrilled by the turnout -- we would never have thought that 900+ Java developers would be so interested in DTrace. We spent the next few hours hashing out some ways to get better support for Java in DTrace; we look forward to giving an update at next year's conference. As promised, here are the slides...

Thursday, June 30, 2005 | DTrace | Read More

DTrace in the JavaOne Keynote

In case you missed the DTrace demo during the JavaOne keynote, you can view it in RealPlayer format here. To skip to the DTrace stuff, jump 1:12:30 in. I haven't actually been able to bring myself to watch it -- I'm too self-conscious -- so feel free to comment on it. Things to keep in mind when watching: I can type better when I'm not in front of 7,000 people and I'm well aware that I need a hair cut. That guy next to me is John Loiacono (not to be confused with Laocoon, the...

Tuesday, June 28, 2005 | DTrace | Read More

Open-Sourcing the JavaOne Keynote

This morning I gave a demo of DTrace with the Java agents during the keynote at JavaOne. In the past few hours I've had a lot of great feedback from Java developers -- we've found a bunch of big performance wins already, and I expect we'll find more this week (remember the DTrace challenge). For the demo, I ran /usr/java/demo/jfc/Java2D/Java2Demo.jar with the Java DTrace agents enabled and ran a couple of scripts on it. The first script just gathered a frequency count for each...

Monday, June 27, 2005 | DTrace | Read More

DTrace at JavaOne

I'll be at JavaOne this week demonstrating the new Java/DTrace mash-up. At Monday morning's keynote, I get a 100 second talk-within-a-talk (which itself is a talk-within-a-talk). On Monday evening, Stephen, Mike, Bryan, and I (and possibly some other Solaris folks) will be holding a BoF on OpenSolaris, DTrace and what they mean for Java. We'll be in Golden Gate B1 at the Marriott Hotel from 7:30pm to 8:30pm. I'll be hanging around one of the DTrace or Solaris booths whenever...

Sunday, June 26, 2005 | DTrace | Read More

DTrace Insanity at JavaOne

Something kind of crazy is going on. I got a call from Chris; "Check out what we're going to do at JavaOne," he said. Here's the deal: come to JavaOne, bring your Java application (on a usb device or CD or whatever), and we'll[1] improve your application using DTrace. Or you get a free tchotchke -- possibly one of several iPods. Usually you only see this kind of offer at furniture stores, and there's some catch, but this is the real deal: either your application goes faster...

Friday, June 17, 2005 | DTrace | Read More

Comment out of context

The OpenSolaris launch has been pretty fun -- I've already had some discussions with customers over the source code. Of course, the first thing people seemed to do with the source code is look for references to "shit" and "fuck". This was titillating to be sure. Unsatisfied with the cheap laugh, ZDNet wanted to draw some conclusions from the profanity: The much-vaunted dynamic tracing (dtrace) feature of Sun's system may not be as safe to use as most people think. "This bit me...

Thursday, June 16, 2005 | DTrace | Read More

Debugging cross calls on OpenSolaris

I think the thing I love most about debugging software is that each tough bug can seem like an insurmountable challenge -- until you figure it out. But until you do, each tough bugs is the hardest problem you've ever had to solve. There are weeks when every morning presents me with a seemingly impossible challenge, and each afternoon I get to spike my keyboard and do a little victory dance before running for thetrain. For my first OpenSolaris blog post, Ithought I talk about...

Tuesday, June 14, 2005 | OpenSolaris | Read More

Real Java debugging w/ DTrace

When I was in college one of the rights of passage in the computer science department was the software engineering class which involved a large group project. Fresh from completing that class, my brother turned up the other day in San Francisco (where I live); naturally I wanted to try out the game he and his friends had written. Hogs is a 3-D tank game written in Java -- when it failed to run on my Solaris 10 laptop I decided to use the new DTrace agents for the JVM that I bl...

Sunday, May 29, 2005 | DTrace | Read More

DTrace webchat

A colleague pointed me to a recent online seminar given on the topic of DTrace. If you've been meaning to crack the Solaris Dynamic Tracing Guide to get started with DTrace, but are more of a, um, passive learner, this might be a nice way to get started. Technorati tag:

Friday, May 20, 2005 | DTrace | Read More

DTracing Java

DTrace has cast light on parts of the system that were previously only dimly illuminated by previous tools, but there have been some parts of the system frustratingly left in the dark. The prevalent example is Java. Java has been relatively unobservable with DTrace; the jstack() action has offered a narrow beam of light into interactions between Java code and the rest of the system, but we really need is Java probes in the DTrace framework. DTrace users really want to be able...

Monday, April 18, 2005 | DTrace | Read More

the pid provider and > 10 arguments

A long-time DTrace user was recently examining an ugly C++ application, and this obvious DTrace invocation to trace the 15th argument (zero-indexed) to a particularly ugly function: # dtrace -n pid123::foobar:entry'{ trace(arg15); }'dtrace: invalid probe specifier pid380863:::entry{ trace(arg15); }: in action list: failed to resolve arg15: Unknown variable name As described in the Solaris Dynamic Tracing Guide we actually only provide access to arguments 0-9. I suppose you...

Monday, April 4, 2005 | DTrace | Read More

Beautiful S10/RHEL drag race

Network World Fusion has an article comparing Solaris 10 to Red Hat Enterprise Linux 4.0 (the self-proclaimed Solaris killer. The sweetest part of this article is the graph showing relative performance numbers. Solaris gets beat in 32-bit mode, but edges out Linux in 64-bit mode on identical hardware. AND, we've got a bunch more x86-64 performance through the pipeline. This benchmark is just in one very specific area of the system, but it's exciting nonetheless.

Wednesday, March 2, 2005 | OpenSolaris | Read More

pid provider exposed

In the kernel group, we occasionally give presentations on interesting work we've been doing. Last week, I gave a talk on the implementation the DTrace pid provider, and -- in the spirit of community that the OpenSolaris project has engendered -- I'm posting it on my blog: click to go to the presentation In this presentation I go into the many details of the general technique for arbitrary instruction instrumentation, the implementation on SPARC, x86 and amd64 as well as the...

Tuesday, March 1, 2005 | DTrace | Read More

Sanity and FUD

With all the FUD coming out of Red Hat, I thought the online rags would fall all over themselves to call shotgun on the anti-Solaris bandwagon. But LinuxInsider seems to have planted their flag on rational reporting: Stacey Quandt, Open Source Practice Leader for the Robert Frances Group, agreed. "Although many organizations continue to migrate Solaris workloads from UltraSPARC to Linux on x86 and other platforms, the latest version of Red Hat does not eliminate the need for...

Thursday, February 17, 2005 | OpenSolaris | Read More

DTrace is open

It's a pretty exciting day for the DTrace team as our code is the first part of Solaris to be released under the CDDL. I thought I'd take the opportunity to end my blogging hiatus and to draw attention to some of my favorite corners of the code. Bryan has written an exhaustive overview of the code structure as well as some of his favorite parts of the code. fasttrap The biggest component of DTrace that I was wholly responsible for was the user-level tracing component. The pid...

Tuesday, January 25, 2005 | DTrace | Read More

Solaris 10 in the news: week 1

It's been just over a week since we officially launched Solaris 10 and the reactions from the press have been all over the map. Consider these two declarations: Not many open source aficionados will realize the impact, but by making Solaris 10 free and capable of operating on any kind of hardware, Sun is making a coup in the server market. ...As a result Linux will probably not grow much beyond its current market share of about 10% leaving Red Hat and especially Novell with a...

Thursday, November 25, 2004 | OpenSolaris | Read More

Baby's first DTrace

At the Solaris 10 launch on Monday I was talking to a sysadmin about DTrace. He was clearly very excited about it -- finally he could end a fight between the database guys and the appserver guys about whose stuff was to blame -- but he had one reservation: Where do I start? Since DTrace lets you look at almost anything on the system, it can be hard knowing the first thing to look at, here's what I told him: start with the tools you know You've probably used truss(1) or...

Thursday, November 18, 2004 | DTrace | Read More

Solaris 10 Launch

I was at the Solaris 10 launch for most of Monday, and it was a pretty fantastic day for everyone working on Solaris 10. I spent about two hours helping to answer questions in an online chat -- here's the transcript -- about Solaris 10 in what was dubbed a webchat sweatshop. There were a bunch of us from the Solaris group as well as Scott and some other execs all huddled around laptops while the HR folks beat the drum at a slow, but steady pace: These answers have to be on...

Wednesday, November 17, 2004 | OpenSolaris | Read More

LinuxInsider on Solaris

Check out LinuxInsider's article on Solaris 10. In particular, the last paragraph has high praise for Solaris 10 and Sun's forthcoming hardware ventures: Bring them together in production systems and what do you get? The ability to organize your business around a single physical computer redundantly implemented in processors spread across your network -- meaning that a lot of business processes now limited by technology costs and software complexity can be simplified right...

Thursday, November 11, 2004 | OpenSolaris | Read More

open challenge: vim for DTrace

Inspired by the Ansari X-Prize, I thought I'd start issuing open challenges to the community of DTrace and Solaris users. This isn't because I hate you, or even because I like you necessarily: it's because I'm lazy. Or if not lazy, at least busy. So my first open challenge is this: vim for .d files The following are required of a winning entry syntax hi-lighting of keywords, built-in functions, aggregations, etc. smart commenting (when I hit return I want ' \* ') other fancy stuff e...

Tuesday, November 2, 2004 | DTrace | Read More

DTrace time

The other day I spent some time with a customer tuning their app using DTrace. I hadn't done much work on apps outside of Sun, so I was excited to see what I could do with DTrace other people's code. I'm certainly not the first person to use DTrace on a real-world app (I'm probably the 10,000th), so I had some expectations to live up to. The app basically processes transactions: message comes in, the app does some work and sends out another message. There's a lot...

Friday, October 29, 2004 | DTrace | Read More

breaking with tradition

In this weblog, I've tried to stick to the facts, talk about the things I know about DTrace, Solaris and the industry, and not stray into the excrutiating minutia of the rest of my life. But: The Red Sox Are Going To The World Series!! Post World Series Update:It has been an amazing and unexpected elation I've carried with me this past week since the Sox won their first world series in my and my father's lifetimes. My grandfather was born in 1919 -- we would have loved to have...

Wednesday, October 20, 2004 | Other | Read More

DTrace case study and contest

Jay Danielson has posted a great case study of using DTrace to solve a problem porting a linux driver to Solaris. It's not an incredibly complicated example, but I find this kind of tracking a problem from symptoms to root cause to be very helpful at understanding how to use DTrace. Speaking of using DTrace on sun.com, Sun is sponsoring a contest to come up with the most innovative or creative use of DTrace (or Zones). If you're sitting on some D program that helped you out...

Monday, October 18, 2004 | DTrace | Read More

more on gcore

Trawling through b.s.c I noticed Fintan Ryan talking about gcore(1), and I realized that I hadn't sufficently promoted this cool utility. As part of my work adding variable core file content, I rewote gcore from scratch (it used to be a real pile) to add a few new features and to make it use libproc (i.e. make it slightly less of a pile). You use gcore to take a core dump of a live running process without actually causing the process to crash. It's not completely uninvasive...

Wednesday, October 13, 2004 | OpenSolaris | Read More

back from the DTrace road show

A number of factors have conspired to keep me away from blogging, not the least of which being that I've been on a coast-to-coast DTrace road show. Now that I'm back, I've got some news to report from the road. Step right up! At times it felt a bit like a medicine road show: "Step right up to see the amazing DTraaaaace! The demystifying optimizing tantalizing reenergizing tracing frameworrrrrrrk!" I stopped in the Midwest, and D.C. (Erichelped during that leg as my fellow...

Tuesday, October 5, 2004 | DTrace | Read More

a new view into software

As Bryan has observed the past, software has a quality unique to engineering disciplines in that you can build it, but you can't see it. DTrace changes that by opening windows into parts of the system that were previously unobservable and it does so in a way that minimally changes what you're attempting to observe -- this software "uncertainty principle" has limited the utility of previous observability tools. One of the darkest areas of debugging in user-land has been around...

Tuesday, August 31, 2004 | DTrace | Read More