Wednesday Oct 30, 2013

Back Up to Tape the Way You Shop For Groceries

Imagine if this was how you shopped for groceries:

  1. From the end of the aisle sprint to the point where you reach the ketchup.
  2. Pull a bottle from the shelf and yell at the top of your lungs, “Got it!”
  3. Sprint back to the end of the aisle.
  4. Start again and sprint down the same aisle to the mustard, pull a bottle from the shelf and again yell for the whole store to hear, “Got it!”
  5. Sprint back to the end of the aisle.
  6. Repeat this procedure for every item you need in the aisle.
  7. Proceed to the next aisle and follow the same steps for the list of items you need from that aisle.

Sounds ridiculous, doesn’t it?

Not only is it horribly inefficient, it’s exhausting and can lead to wear out failures on your grocery cart, or worse, yourself. This is essentially how NetApp and some other applications write NDMP backups to tape. In the analogy, the ketchup and mustard are the files to be written, yelling “Got it!” is the equivalent of a sync mark at the end of a file, and the sprint back to the end of an aisle is the process most commonly called a “backhitch” where the drive has to back up on a tape to start writing again.

Writing to tape in this way results in very slow tape drive performance and imposes unnecessary wear on the tape drive and the media, especially when writing small files. The good news is not all tape drives behave this way when writing small files. Unlike midrange LTO drives, Oracle’s StorageTek T10000D tape drive is designed to handle this scenario efficiently.

The difference between the two drive types is that the T10000D drive gives you the ability to write files in a NetApp NDMP backup environment the way you would normally shop for groceries. With grocery shopping, you essentially stream through aisles picking up items as you go, and then after checking out, yell, “Got it!”, though you might do that last step silently. With the T10000D, it has a feature called the Tape Application Accelerator, which prevents the drive from having to stop after each file is written to notify NetApp or another application that the write was successful.

When enabled in the T10000D tape drive, Tape Application Accelerator causes the tape drive to respond to tape mark and file sync commands differently than when disabled:

  • A tape mark received by the tape drive is treated as a buffered tape mark.
  • A file sync received by the tape drive is treated as a no op command.

Since buffered tape marks and no op commands do not cause the tape drive to empty the contents of its buffer to tape and backhitch, the data is written to tape in significantly less time. Oracle has emulated NetApp environments with a number of different file sizes and found the following when comparing the T10000D with the Tape Application Accelerator enabled versus LTO6 tape drives.

Notice how the T10000D is not only monumentally faster, but also remarkably consistent? In addition, the writing of the 50 GB of files is done without a single backhitch. The LTO6 drive, meanwhile, will perform as many as 3,800 backhitches! At the end of writing the entire set of files, the T10000D tape drive reports back to the application, in this case NetApp, that the write was successful via a tape mark.

So if the Tape Application Accelerator dramatically improves performance and reliability, why wouldn’t you always have it enabled? The reason is because tape drive buffers are meant to be just temporary data repositories so in the event of a power loss, there could be data loss in certain environments for the files that resided in the buffer. Fortunately, we do have best practices depending on your environment to avoid this from happening. I highly recommend reading Maximizing Tape Performance with StorageTek T10000 Tape Drives (pdf) to decide which best practice is right for you. The white paper also digs deeper into the benefits of the Tape Application Accelerator. The white paper is free, and after downloading it you can decide for yourself whether you want to yell “Got it!” out loud or just silently to yourself.

Customer Advisory Panel

One final link: Oracle has started up a Customer Advisory Panel program to collect feedback from customers on their current experiences with Oracle products, as well as desires for future product development. If you would like to participate in the program, go to this link at

photo taken on Idaho's Sacajewea Historic Biway by Rick Ramsey

- Brian Zents

Follow OTN on
Blog | Facebook | Twitter | YouTube

Friday Sep 13, 2013

About LTFS - Library Edition

Oracle just launched the T10000D tape drive with its incredible 8.5 TB of native capacity and LTFS-Library Edition (LTFS-LE), which expands the LTFS concept to an entire library. The Oracle T10000D has some neat features that I would like to address in the future, but today I’d like to talk about LTFS-LE since it really is a new concept.


LTFS is an open source specification for writing data to tape on single tape drives. It is supported by Oracle and other tape vendors. The version you can download from Oracle is called StorageTek LTFS, Open Edition (LTFS-OE).

When an LTFS-compatible T10000 or LTO tape is formatted for LTFS, it is split into two partitions. The first partition holds the metadata that tells the user which files are on the tape and where they are located. The second partition holds the files themselves.

Benefits of Using LTFS-LE

There are a few nice benefits for those who utilize LTFS. Most important is the peace of mind that you will always be able to recover your data regardless of your backup application or any other proprietary software because it’s based on an open source specification. It also improves the portability of tape because two parties don’t both need the same application to read a tape. In fact, LTFS has seen tremendous adoption in industries that require the ability to transport large amounts of data.

The limitation with the open source version of LTFS is that it’s limited to just a single drive. Users with even the smallest archives would like to have their entire environment to be LTFS-based. That’s the impetus for StorageTek LTFS, Library Edition (LTFS-LE), but it also serves as a backup application eliminator because of how it’s architected. With LTFS-OE, after you download the driver, a tape looks like a giant thumb drive. LTFS-LE makes the tape library look like a shared drive with each tape appearing as a sub-folder. It’s like having a bucket full of thumb drives that are all accessible simultaneously!

Just as before, you don’t need any additional applications to access files. And end users are almost completely abstracted from the nuances of managing tape. All they need is a Samba or CIFS connection and they have access to the tape library. LTFS-LE is agnostic to corporate security architectures so a system administrator could make some folders (tapes) available to some users while restricting others based on corporate security guidelines.

Security and Performance Considerations

However, security is arguably one of the more straightforward considerations when deciding how to integrate an LTFS-LE implementation into your environment. An additional consideration is to ensure that LTFS-LE can meet your performance expectations. Tape drives are remarkably faster than they are given credit for (the Oracle T10000D can write at 252 MB/sec.), but sometimes networks aren’t designed to handle that much traffic so performance requirements need to be considered accordingly. In addition, it may take some time before a read operation actually starts as the library needs time to mount a tape. As a result, system administrators need to be cognizant of how end user applications will accept response times from any tape storage-based solution.

A final performance consideration is to be aware of how many tape drives are in your library relative to how many users may be accessing files directly from tape. If you have a disproportionately large number of users you may want to consider a more traditional enterprise-level archiving solution such as StorageTek Archive Manager (SAM), which writes files based on the Tape Archive Record (TAR) open source standard.

Ultimately, LTFS-LE provides exciting new opportunities for system administrators looking to preserve files with a format that isn’t dependent on proprietary solutions. It also makes it easy for users who need access to large amounts of storage without a lot of management difficulties. Support for LTFS continues to grow. Oracle is actually one of the co-chairs of the SNIA committee that’s working towards standardizing LTFS. And this is just the start for LTFS-LE as well, as Oracle will continue expanding its capabilities in the near future.

picture of 2008 Harley Davidson FXSTC taken by Rick Ramsey
- Brian Zents

Follow OTN Garage on:
Blog | Facebook | Twitter | YouTube |

Tuesday May 29, 2012

Is Linear Tape File System (LTFS) Best For Transportable Storage?

Those of us in tape storage engineering take a lot of pride in what we do, but understand that tape is the right answer to a storage problem only some of the time. And, unfortunately for a storage medium with such a long history, it has built up a few preconceived notions that are no longer valid.

When I hear customers debate whether to implement tape vs. disk, one of the common strikes against tape is its perceived lack of usability. If you could go back a few generations of corporate acquisitions, you would discover that StorageTek engineers recognized this problem and started developing a solution where a tape drive could look just like a memory stick to a user. The goal was to not have to care about where files were on the cartridge, but to simply see the list of files that were on the tape, and click on them to open them up. Eventually, our friends in tape over at IBM built upon our work at StorageTek and Sun Microsystems and released the Linear Tape File System (LTFS) feature for the current LTO5 generation of tape drives as an open specification.

LTFS is really a wonderful feature and we’re proud to have taken part in its beginnings and, as you’ll soon read, its future. Today we offer LTFS-Open Edition, which is free for you to use in your in Oracle Enterprise Linux 5.5 environment - not only on your LTO5 drives, but also on your Oracle StorageTek T10000C drives. You can download it free from Oracle and try it out.

LTFS does exactly what its forefathers imagined. Now you can see immediately which files are on a cartridge. LTFS does this by splitting a cartridge into two partitions. The first holds all of the necessary metadata to create a directory structure for you to easily view the contents of the cartridge. The second partition holds all of the files themselves. When tape media is loaded onto a drive, a complete file system image is presented to the user. Adding files to a cartridge can be as simple as a drag-and-drop just as you do today on your laptop when transferring files from your hard drive to a thumb drive or with standard POSIX file operations.

You may be thinking all of this sounds nice, but asking, “when will I actually use it?” As I mentioned at the beginning, tape is not the right solution all of the time. However, if you ever need to physically move data between locations, tape storage with LTFS should be your most cost-effective and reliable answer. I will give you a few use cases examples of when LTFS can be utilized.

Media and Entertainment (M&E), Oil and Gas (O&G), and other industries have a strong need for their storage to be transportable. For example, an O&G company hunting for new oil deposits in remote locations takes very large underground seismic images which need to be shipped back to a central data center. M&E operations conduct similar activities when shooting video for productions. M&E companies also often transfers files to third-parties for editing and other activities.

These companies have three highly flawed options for transporting data: electronic transfer, disk storage transport, or tape storage transport. The first option, electronic transfer, is impractical because of the expense of the bandwidth required to transfer multi-terabyte files reliably and efficiently. If there’s one place that has bandwidth, it’s your local post office so many companies revert to physically shipping storage media. Typically, M&E companies rely on transporting disk storage between sites even though it, too, is expensive.

Tape storage should be the preferred format because as IDC points out, “Tape is more suitable for physical transportation of large amounts of data as it is less vulnerable to mechanical damage during transportation compared with disk" (See note 1, below). However, tape storage has not been used in the past because of the restrictions created by proprietary formats. A tape may only be readable if both the sender and receiver have the same proprietary application used to write the file. In addition, the workflows may be slowed by the need to read the entire tape cartridge during recall.

LTFS solves both of these problems, clearing the way for tape to become the standard platform for transferring large files. LTFS is open and, as long as you’ve downloaded the free reader from our website or that of anyone in the LTO consortium, you can read the data. So if a movie studio ships a scene to a third-party partner to add, for example, sounds effects or a music score, it doesn’t have to care what technology the third-party has. If it’s written back to an LTFS-formatted tape cartridge, it can be read.

Some tape vendors like to claim LTFS is a “standard,” but beauty is in the eye of the beholder. It’s a specification at this point, not a standard. That said, we’re already seeing application vendors create functionality to write in an LTFS format based on the specification. And it’s my belief that both customers and the tape storage industry will see the most benefit if we all follow the same path. As such, we have volunteered to lead the way in making LTFS a standard first with the Storage Network Industry Association (SNIA), and eventually through to standard bodies such as American National Standards Institute (ANSI). Expect to hear good news soon about our efforts.

So, if storage transportability is one of your requirements, I recommend giving LTFS a look. It makes tape much more user-friendly and it’s free, which allows tape to maintain all of its cost advantages over disk!

Note 1 - IDC Report. April, 2011. “IDC’s Archival Storage Solutions Taxonomy, 2011”

- Brian Zents

Website Newsletter Facebook Twitter

Tuesday Oct 18, 2011

Solving the Tape Storage Space Problem

Because I came of age professionally in the shorts-and-tshirt smarter-than-you culture of Silicon Valley, I always assumed tape storage was used only by retired British spies with names such as Baratheon and Brewster and Cameron who lived in dank mansions on the rocky coast of Scotland and still dressed in tweed jackets for dinner. They spent their days engrossed in the struggle to keep their Dunhill pipes lit and their hair piece in place against fury of the North Wind. Every few months a carrier pigeon would arrive from MI5, and Baratheon or Brewster or Cameron would slowly descend the stairs to a back room. A week later he would return with a name written in code, and hand it to a man in a dripping wetsuit and spear gun who would jump off the cliff without looking and swim it back to a submarine waiting off the coast.

Turns out I was behind the times. In fact, tape for archiving has several advantages that make it economically feasible in today's digital pack-rat economy. Such as durability. And much, much lower power consumption. You can read about them in this paper by Horison information strategies:

Tape: The Digital Curator of the Information Age (registration required)

If you're a storage admin or IT manager considering tape, there's another paper that may interest you more. Published on OTN in July, it describes very clearly the limitations of data that is written in a stream to tape, and how Oracle technologies overcome them. For instance, once you write a block of data to a stream of tape, that particular bit of data not only becomes inefficient to target for access, but updates to the data become clumsy and cumbersome. And as tape cartridges grow to store a terabyte of data, the problem becomes even more pronounced.

Oracle's StorageTek In-Drive Reclaim Accelerator avoids this problem by simply breaking up the serial data on a tape into smaller, more manageable chunks that are grouped together and managed as logical volumes. Find out how in this well-written white paper:

How it Works: StorageTek Reclaim Accelerator

For more information about Oracle tape drive products, visit OTN's Tape Storage product page.

- Rick

Monday Feb 21, 2011

Red Tape, Part II


As I wrote last week, we recently announced the StorageTek T10000C Tape Drive, and along with it we released a number of related papers. I've been associated with publishing best practices papers for over a decade and it seems like there is no topic with less glamor for the writer – yet more importance for the reader – than the combined topics of backup/archive and restore/retrieve. The challenge has grown exponentially with the growth of disk storage. I remember the first time we sold a terabyte of storage: it was a room full of (big, heat-generating/power-consuming) 250 MB drives, costing millions of dollars. Now, just my home configuration consists of roughly 6 terabytes of disk.

How do you back up all of that storage? Tape: really fast tape. And, lots of it. This creates a whole variety of very interesting challenges today, elevating the topic to – at the very least – glamorous, but I think it qualifies as being downright hot! Fascinating areas include optimizing retrieving information from a vast achive of tape units, making sure that when you have backed up onto tape you can be sure it was done so without errors, and then there is the whole challenge of providing security. We have a paper for that!

Let's start off with the challenge of finding information on a serial, as opposed to random access, device. It can be done with brute force, and there are expensive solutions to assist, but the T10000C has a built in accelerator that relieves your system from the overhead. Learn how to use it by reading the concise Using Oracle's StorageTek Search Accelerator, by Oracle engineer Dwayne Edling.

Consider: you have invested big bucks to archive your priceless information onto potentially thousands of tapes. Of course, the latest tape drives all verify using ECC and CRC. However, these do not protect data that is being moved outside the storage device, resulting in a chance for data corruption as it is migrated across the storage landscape. The T10000C addresses this one step further by validating CRC check-sums generated at the host using Oracle's StorageTek Data Integrity Validation Solution (DIV). The brief article StorageTek Data Integrity Validation for the StorageTek T10000C Tape Drive, by the prolific Dwayne Edling, explains this problem in detail and presents some of the details of DIV.

Finally,we don't think twice about encrypting sensitive data on disks – what about on tape? One important aspect of enterprise security is the physical aspect: if someone stole the compact tape media, they could uncover all of your darkest secrets. The Oracle Key Manager (OKM) working with the Sun Crypto Accelerator 6000 provides a clean and highly efficient solution. You can read about this powerful combination in Oracle Key Manager Version 2.x Security and Authentication White Paper.

I have to be honest: I had never thought of Tape as a hot topic before. My bad!

- Kemer

Monday Feb 07, 2011

No Red Tape Here, Part I.

Old Tape DriveSome of us remember the good old days of computer tape. For example, I remember that the write protect ring on the back of the 6250 tape reels had a perfect balance: grasping the little tab and flinging it at the wall, the ring bounced back to be caught without leaving a mark on the lab. We did this as we waited for long compiles to finish and found other creative amusements with this simple component; I am old enough to escape retribution from my supervisors in making this admission, as they are long retired. But, I digress.

Tape as a backup medium – although in a much more sophisticated form – remains a staple of large enterprise backup. Last week we had a big announcement of Oracle's StorageTek T1000C Tape Drive, which offers 5 TB native uncompressed storage and a data transfer rated at nearly ¼ gigabyte per second. (To keep things in perspective, the 6250 stored 140 MB; I don't remember how slow it was, but it was well under 1 megabyte per second...) As a result of this announcement, we have released a number of technical papers that I thought I would discuss in a couple of blogs.

First of all, in thinking about speed, you won't want to miss Evaluating Tape Drive Performance White Paper, by Oracle engineer Dwayne Edling. Dwayne opens the paper by discussing the three components that limit tape speed:

  1. The speed that the storage application sends data to, or processes data from, the drive.
  2. The speed of the host interface between the drive and the application.
  3. The speed that the tape drive writes or reads data at the head/media interface.

6250 TapeHe then goes on to test the T1000C Tape Drive under a variety of conditions and comes to the important conclusion that in most situations, either the storage applications' speed or the drive throughput speed are the limiting factors in tape drive performance. Typical storage application throughput of 50-60 MB/s falls far short of the 400 MB/s maximum speed of a 4GB SCSI FCP interface. Even with compressed data, current tape drive technologies are also not able to write or read data any faster than a 4 GB (400 MB/s) SCSI FCP interface can push the data.

Back in the Jurassic age of computing, we naively assumed that tape storage was essentially "forever." That proved not to be the case! Another very interesting paper that addresses the issue of longtime archival is Protecting Your Archival Data With Improved Tape Dimensional Stability. This brief paper gives you some insight into the factors that make for longer lasting tape – its dimensional stability. Of course, some of these are environmental, but the substrate used is very important; advances in tape substrates have resulted in significant improvements over the last decade. Oracle's selection of aramid as the StorageTek T10000 T2 substrate has resulted in superior tape dimensional stability performance and long-term archival life.

Finally, don't miss Redefining Tape Usage With StorageTek Tape Tiering Accelerator and StorageTek In-Drive Reclaim Accelerator.The StorageTek In-drive Reclaim Accelerator and StorageTek Tape Tiering Accelerator, an important innovation in the T1000C Tape Drive, provide the capability to randomly access, add or delete physical partitions. This paper goes into the nitty gritty of how this works.

Existing tape media can store more than a terabyte of user data. In the near future, a single tape media will hold tens of terabytes, with 100 TB capacities in the foreseeable future. Managing these multi-terabyte tape cartridges requires a new approach for managing data on tape. Oracle has developed a new tape storage format with the StorageTek T10000C tape drive, using an innovative partitioning architecture, that allows the addition or removal of storage space as needed. With these capacities and efficiencies, tape remains an important medium for enterprise archiving.

- Kemer


Logan Rosenstein
and members of the OTN community


« February 2017
Blogs We Like