Monday Nov 02, 2009

Immutable Service Containers @ Amazon EC2

Just in time for the OpenSolaris Developer Conference, we were able to publish new Immutable Service Containers images directly to the Amazon Web Services Elastic Compute Cloud (EC2) environment. Previously, I talked about creating ISCs using our security enhanced OpenSolaris 2009.06 AMIs. Today, I am happy to announce that we have taken the next logical step by making available AMIs that fully incorporate the ISC changes. If you want to try out this configuration, simply provision an Immutable Service Containers AMI on EC2. We have made AMIs available in both the U.S. (ami-48c32021) and European (ami-78567d0c) regions. As always, we would love to get your feedback on these images and what you would like to see next!

Take care!

Technorati Tag:

Thursday Jul 09, 2009

Encrypted ZFS using Amazon EBS and OpenSolaris 2009.06

Recently, I had the pleasure of exchanging e-mail with István Soós who had contacted our OpenSolaris on EC2 team asking how he could use OpenSolaris along with Amazon's Elastic Compute Cloud (EC2) and Elastic Block Stores (EBS) to create a subversion-based source code control system. Sounds simple, right? Well, István threw us a curve ball. He wanted the revision control system to run on OpenSolaris and be stored on an encrypted, mirrored ZFS file system backed by EBS. Now, you have to admit, that is pretty cool!

This is the point in the story where István met. After going over the requirements, it appeared as though the encrypted scratch space work that had been done for the Immutable Service Container project was a near fit except that persistence was needed. So, I provided István with links to this work which of course linked to Darren's original article on ZFS encryption using LOFI. Just a day later, István replied that his environment was up and running! Talk about speed and agility in the Cloud Computing world!

I would definitely encourage you to check out all of the details on István's blog. I especially want to thank István for sharing this great article that I hope will encourage others to try new things and keep pushing the OpenSolaris envelope forward!

Take care!

Technorati Tag:

Wednesday Jan 28, 2009

Amazon S3 Silent Data Corruption

While catching up on my reading, I came across an interesting article focused on the Amazon's Simple Storage Service (S3). The author points to a number of complaints where Amazon S3 customers had experienced silent data corruption. The author recommends calculating MD5 digital fingerprints of files before posting them to S3 and validating those fingerprints after later retrieving them from the service. More recently, Amazon has posted a best practices document for using S3 that includes:

Amazon S3’s REST PUT operation provides the ability to specify an MD5 checksum (http://en.wikipedia.org/wiki/Checksum) for the data being sent to S3. When the request arrives at S3, an MD5 checksum will be recalculated for the object data received and compared to the provided MD5 checksum. If there’s a mismatch, the PUT will be failed, preventing data that was corrupted on the wire from being written into S3. At that point, you can retry the PUT.

MD5 checksums are also returned in the response to REST GET requests and may be used client-side to ensure that the data returned by the GET wasn’t corrupted in transit. If you need to ensure that values returned by a GET request are byte-for-byte what was stored in the service, calculate the returned value’s MD5 checksum and compare it to the checksum returned along with the value by the service.

All in all - good advice, but it strikes me as unnecessarily "left as an exercise to the reader". Just as ZFS has revolutionized end-to-end data integrity within a single system, why can't we have similar protections at the Cloud level? While certainly it would help if Amazon was using ZFS on Amber Road as their storage back-end, even this would be insufficient...

Clearly, more is needed. For example, would it make sense to have an API layer be added that automates the calculation and validation of digital fingerprints? Most people don't think about silent data corruption and honestly they shouldn't have to! Integrity checks like these should be automated just as they are in ZFS and TCP/IP! As we move into 2009, we need to offer easy to use, approachable solutions to these problems, because if the future is in the clouds, it will revolve around the data.

Technorati Tag:

About

gbrunett

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today