Thursday Feb 21, 2008

Getting the default values of option.dat with imsimta

Every once in a while, I find a need to determine the default settings of parameters not already specified in option.dat. The obscure command makes me spend several minutes looking it up every time:

# imsimta cnbuild -noimage -option_file=/tmp/option.dat.dist
# cd /tmp
# ls -l option.\*
-rw------- 1 mailsrv mailsrv 4862 Feb 21 12:04 option.dat.dist

Wednesday Jan 30, 2008

CalDAV on mobile devices

This week marks the end of the weekly SyncML team call we've been having for nearly two years, which signifies a conclusion to the project. Every Sun employee can now synchronize contacts, calendar, and tasks over-the-air to any device that supports the SyncML protocol. It took much longer than anyone on the team expected, but in the end we pulled it off, and in the process, I learned a lot about the troubles with calendar formats, access protocols and synchronization.

Although I have much respect for SyncML for bringing an open standard to a space which has been mostly proprietary, I'm becoming more convinced that CalDAV with a remote calendar is a better protocol than SyncML with a local calendar for bringing calendaring to mobile devices.

I realize CalDAV is not designed specifically for mobile usage, nor is it capable of synchronization, while SyncML is like an arbitrator who settles differences between two parties engaged in a scheduling dispute. But there wouldn't have been a dispute in the first place, if all parties had read/write access to a master calendar on the network. That's what CalDAV does: it employs a client-server model, intrinsically enabling distributed access to centralized data and thereby eliminating the need to sort out differences.

Due to higher chances of data corruption as the number of synchronization devices increases, we have to warn our users if they use, in addition to SyncML, another synchronization method, such as Outlook Connector, Apple iSync, or Palm Desktop, they're assuming risks on their own. We recognize this is not a realistic proposition, but it is a limitation of calendar synchronization.

Even for those who only use one device, and therefore are not interested in synchronization or distributed access, CalDAV still brings advantages over local calendar. By storing your schedules in the cloud, your data is independent of the device, should it ever gets lost, replaced, or stolen, your calendar is still safe and can be easily transferred to the replacement. Another benefit is the ability to share your calendar with your family, friends or clients, and vice versa.

From a business perspective, mobile network operators and internet service providers looking for ways to retain existing customers could offer CalDAV as another feature to incent subscribers to renew service contracts.

With CalDAV adoption gaining traction among major players, I think we will see CalDAV clients being developed on mobile devices in the near future, supplanting SyncML and local calendaring, just like how IMAP is wiping out POP as the preferred way to access Email.

Sidenote: I'll be attending the CalConnect event in Menlo Park next week, Feb 4-8th, as an observer. If any reader will be at the same event, I'd love to hear your thoughts in person!

[UPDATE Feb 1, 2008] Another downside of local calendar is that most mobile devices do not have enough memory to store every event accumulated over the years. CalDAV could solve that by storing the entire calendar on the server where storage is ample, while the mobile client would simply keep an essential subset based on its memory limitation.

Sunday Dec 23, 2007

Metric system in the U.S.

On this day, in 1975, President Gerald Ford signed the Metric Conversion Act into law thus establishing the U.S. Metric Board to coordinate the conversion to the metric system—also known as SI.

But some Americans refused to give up their familiar measurements. In 1982, President Ronald Reagan disbanded and stopped funding the U.S. Metric Board.

25 years later, metric is standard all over the world, except three countries—the United States, Liberia, and Myanmar.

The other day, I bought a pair of door handles from IKEA, but the screws that came with the package weren't long enough, so I headed to my neighborhood OSH store. Only then did I realize that IKEA screws are measured in millimeters, and the man in the yellow jacket told me although they carry some metric screws, most screws sold in the US are manufactured in inches and since the size I needed was quite unusual, it's quite unlikely I would find it. So I returned the door handles.

Under pressure of globalization, America going metric should be just a matter of when, not if.

Tuesday Dec 18, 2007

Installed punchin on my Mac

Today I went to the installfest at MPK17, and under Dan McDonald's guidance, successfully installed punchin on my Powerbook running OS X 10.4.11. It took less time—about 20 minutes—than I expected, even though we had to re-install the client after realizing what I had the first time wasn't the latest version (Punchin 2.1.0 for OS X).

Punchin is a secure remote access solution at Sun that is currently used by over 1,000 employees. Unlike Cisco's VPN client, punchin uses the native TCP/IP stack with IPsec and IKE on a platform, so there's no need for third-party kernel extension, which leads to better stability.

These are the commands I learned (punchctl is found under /usr/local/bin so make sure it's in $PATH):

~$ punchctl configure (gathers preferences and generates keys)
~$ punchctl start
~$ punchctl stop
~$ punchctl cleanup_local (is like stop but doesn't talk to the remote gateway)

The only difference I've noticed is that punchin connection persists (or perhaps re-establishes is the more correct term) when the laptop awakes from sleep. Per Dan, the maximum idle timeout is four hours. This is a time-saver for me as I often close the lid on my laptop before carrying it to another part of the house, only to realize upon wake up that VPN got disconnected, and have to make another trip to get my token card.

On the flip side, establishing punchin connection takes noticeably longer (3-4 times that of Cisco's solution). According to others, it's faster once the tunnel is up, but I haven't noticed any significant difference in speed myself.

Punchin is available on Solaris, Mac OS X and Linux.

[UPDATE Dec 19, 2007] The slower startup is due to quirks in IPsec on Mac OS X.

Friday Dec 14, 2007

A way to stop Email Cc abuse?

Do you frequently add others to the Cc: distribution?

Carbon copying (Cc for short) in Email is undoubtedly one of the most useful features. Alice sends Bob and Charlie an Email, then Bob invites Dave to the discussion by adding him in the Cc: field.

[Growing Inboxes]

As it happens, however, Cc is also one of the most abused features. WSJ has an article titled Email's Friendly Fire which says:

Email overload is now considered a much bigger workplace problem than traditional email spam. Inboxes are bulging today partly because of what some are calling "colleague spam—that is, too many people are indiscriminately hitting the "reply to all" button or copying too many people on trivial messages, like inviting 100 colleagues to partake of brownies in the kitchen."

If you're Bob, the person who adds others, Cc is great. But if you're Dave, the person who is being added, sometimes you may wonder why you're on the distribution at all and silently curse Bob for contributing to the "colleague spam" you receive in your INBOX.

One difference between Facebookmail and Email I've noticed is that in Facebook, once the sender defines the distribution scope, it becomes fixed and cannot be expanded or shrunk. In other words, Alice, Bob or Charlie may not invite Dave into the discussion, nor could Bob respond in private to Alice or Charlie without starting a new thread. The upside is that whatever is said between Alice, Bob and Charlie remains private to them, but the downside is that others cannot chime in or add value.

In contrast, Facebook event scheduling system allows participants to invite more friends as long as the event is open. A bigger party is always a better party, I suppose. :)

Can we think of a way to prevent Cc abuse yet maintain the flexibility of it at the same time?

Thursday Dec 13, 2007

Plaintext vs HTML Email

A new group called Email Standards Project was recently started to improve the state of web standards support in Email clients by working with both client developers and the designer community. I mentioned in passing that HTML Email doesn't always appear to the recipient the way its sender intended. One reason is that different clients render HTML Emails differently, so it's good to see an effort towards standardization.

That being said, however, I used to be strongly against HTML Email. I had my mail client set to display plaintext alternative when available. I had, whenever a choice is given by the mailing list or online merchant, set preference to receive plaintext rather than HTML. I felt that the sender should not dictate the font size and background color I like to read my Emails, especially on mobile devices.

But over the years, my view gradually shifted. I now consider HTML Email to be complementary to plaintext. Like a presentation or a speech, HTML Email is structured, well-prepared and unidirectional; whereas plaintext is more informal, friendly and dialog-like. Both are important forms of communication, so it would be a bit naive to write off either one. How should one choose one form over the other? Here's a guideline I recommend:

Plaintext Email

  • Good for discussions or two-way exchanges.
  • Preferred on mailing lists.
  • Not suitable where formatting of information is essential to the recipient.

HTML Email

  • Good for announcements, newsletters, press releases, status reports, e-bills, coupons and generally Emails to which you don't expect a reply.
  • Must always include a plaintext alternative.

Reason being that there are reasons for an organization to send professional-looking Emails to its clients, but it's very difficult for both humans and mail client software to respond to and properly quote formatted text, and you never know if someone in your audience may not care for visual appeal, such as an eyesight impaired person. A plaintext alternative ensures graceful degradation.

I wish there was a mail client that would auto-select between plaintext and HTML based on previous habit and context...

Friday Dec 07, 2007

9th anniversary

Today marks my ninth year at Sun. I'm grateful to have the opportunity to work with the smartest and most driven people in their respective fields and to work for many first-class managers (miss you Bob). I look forward to meeting more great people as tenth year begins. Happy anniversary to me!

Tuesday Nov 27, 2007

Mulberry is back and now open source

mulberry_logoWhile I have never used it, Mulberry is an Email client praised by many power users for its large set of features and standards compliance. The company that made it went bankrupt in 2005, but it's back again and released as open source. I should give it a test drive and write a review.

Sidenote: Mulberry is developed by a very talented Cyrus Daboo who now works at Apple on their CalDAV server.

Essential things everyone should know about email

  1. Anyone can send Email to anyone pretending to be from anyone else.
  2. Email is not guaranteed to arrive quickly or at all.
  3. Your Emails can be read by other people.
  4. Email wasn't designed to handle large files.
  5. Any Email that asks you to forward it on to other people is a chain letter, and should be deleted.
  6. You can't trust any Email from anyone.
  7. If you put your Email address on the public internet even once, you will forever get spam.
  8. Never reply to junk mail or scams.
  9. Emails do not convey emotion and can easily be misunderstood.


I'd like to add a few more:

  1. Once you click 'send', there's no 'undo'.
  2. There's no way of knowing if and when the recipient has read your Email.
  3. The danger with composing HTML Email is that it may not look the same on the recipient's end.

Wednesday Nov 21, 2007

Troubleshooting iChat AV

Apple's iChat AV is a very useful application, but sometimes it fails to establish connection with the other end. Most of the time, the failure is due to firewall settings, but on which end? To test, both parties can try connecting to an iChat robot run by Apple; add one of the following test accounts as a new person of Account Type AIM and if the connection establishes successfully, you should see the a video ad.

  • appleu3test01
  • appleu3test02
  • appleu3test03

Tuesday Nov 20, 2007

Finally, no more Gmail blank page

For months, I couldn't use Gmail on my PowerBook. Instead of my INBOX, a blank page would load in Firefox 2 and Safari 2. Clearing cache and related cookies made no difference. Here's what it looked like:

My workaround had been to use a second computer but it's inconvenient. Then Gmail IMAP came along and at least I could use But Gmail IMAP hasn't been very reliable so it would be nice to be able to use the web version from time to time.

Then when Gmail 2.0 came out, the blank page problem was gone from Firefox, but Safari still showed constant "Loading...".

Today, I installed the OS X 10.4.11 update, which includes Safari 3, and now it's usable with Gmail again.

Messaging 2.0 brainstorm session

It's becoming clear and present that a re-architecture of Email is overdue.

Last week, a few aspiring folks got together in New York City to discuss this very topic. In attendance were Tom Evslin, Fred Wilson, Matt Blumberg, Brad Feld, Jeff Pulver and Phil Hollows (click on the names to read thoughts and summaries of the event on their respective blogs).

It appears the goals are pretty well-defined over 3 years ago by the Internet Mail Consortium. There's also consensus on ingredients that will make next generation of messaging a successful one:

  • Social graph enabled
  • Give control to users, not spammers
  • Accessible through an open API
  • Must be secure and trustworthy
  • One interface to manage your online communications

When these elements are put together into a solution, we'll have a reliable, productive and fun way to communicate again.

p.s. Does anyone in the San Francisco Bay Area want to meet up and talk about messaging 2.0? Drop me a note in the comments, Email, IM, Twitter or Facebook (now see why a single interface is attractive?) :)

Thursday Nov 15, 2007

Email getting lots of attention

A quick rundown:

Nov 13, 2007:
  • Marshall Kirkpatrick from wrote "it took Facebook to introduce people to RSS in a way that was really compelling." and "The social network of the future will be populated by the RSS feeds of the activities of your friends and your friends will be determined by email."
  • "The Death of E-Mail" on Slate said "Those of us older than 25 can't imagine a life without e-mail. For the Facebook generation, it's hard to imagine a life of only e-mail..." and "Instant-messaging, mobile text-messaging, blogging, micro-blogging, and social-networking profiles all help compensate for e-mail's shortcomings." and "It's not hard to imagine a future communications command center where, on a single screen, you'll be able to choose between sending an e-mail, instant message, status note, or blog post—or sending all of them at once—and then have all those bits of text neatly and securely archived."
  • Nick O'Neill wrote on "Email Becomes Center of Social Networks?" "I’m not quite sure how I feel about using my email for the center of my social network but maybe my feelings will change once it launches."
  • "Inbox 2.0 isn’t coming, it is here." on Xobni blog said "We realized after building Xobni analytics and playing with email data for 6 months that the most interesting data in email revolved around relationships."
  • Mathew Ingram wrote skeptically on Can getting social make email better? "for me, email is pretty close to broken."
  • Don Dodge wrote "Email contacts - the natural social network" and said "Email is where [people] naturally communicate and collaborate. Social networks are another isolated island of information."
  • Steven Hodson wrote "For me using email as my social network hub doesn’t depend on integration with outside environments or the need to be widgetized in some fashion or other. Instead my Inbox needs tools like Xobni and some serious attention to dealing with spam." and "We already have been using the original social network for a very long time - it just needs some fixing up and new tools to make it better."
  • Google announced the new Google Apps Email Migration API for customers whose existing mail systems don't support IMAP.
  • Sachin Balagopalan thinks it’s time to phase out email from the work place. "The profound difference [...] is that social networks enable you to participate in a virtual team or community and IMO that is conducive to the business environment - we all belong to a team at work and we interact with our team members. Email on the other hand was not designed with the community in mind rather it’s based on an “account” and you can send and receive emails from any account."

I shared my Messaging 2.0 idea with Han today and she seems to like it. I'm hoping to find some time during Thanksgiving holiday to refine and formalize the architecture.

Wednesday Nov 14, 2007

Yahoo and Google to turn Email into a social network

One of the clever things that Facebook does is how it gives users an option to initialize their social graphs from their address book data on Yahoo! Mail, Hotmail and Gmail to see which of their friends are already on Facebook. I didn't bite, in fear of giving my credentials to Facebook (even though they promise to discard them after data is pulled), but a thought struck: isn't social graph basically a more fashionable way of saying address book 2.0?

Then yesterday I read this blog on NYTimes that Yahoo! is working to turn existing user profiles and address books into a social network and they're calling it INBOX 2.0. Google is allegedly doing something similar. Makes sense.

I have been thinking for years that address book should be consulted during spam detection to minimize false positives, and the only systems which have your address book in their possession and also handle your Email are webmail providers. Extending it to create social graphs seems like the logical next step.

Tuesday Nov 13, 2007

Pubsub sender-side Email storage concept

After posting my thoughts on Twitter and Email and reading comment on it, I've been thinking a lot about how outdated the store-and-forward mechanism is for modern asynchronous communication, and how Email could be advanced to meet current and future needs.

From the very beginning, Email delivery has been modeled after postal mail: a sender creates some content and ships it away, where it gets relayed with every step closer to the destination, until it arrives at the designated mailbox. This technique is referred to as "store-and-forward," and SMTP was designed this way 25 years ago because the network back then was unreliable and nodes were not online all the time.

Today, these are no longer the cases. Recent trends such as blogging, podcasting and twittering take advantage of these changes and make use of a completely opposite paradigm—publish-and-subscribe, also known as pub/sub, in which writers publish content on a server, and readers subscribe to those content via feeds. Similar to radio, there are all kinds of broadcast signals in the air but as listeners you pick which station to tune into.

To a certain degree, an Email sender is the same as a blogger in that both publishes content. The difference is a blogger doesn't send what she wrote to others because they, as subscribes of her blog feed, will come to her website to pick it up. Could the same concept be applied to Email whereby a sender publishes an Email on a website which creates a permalink and updates the feed, then the named recipient pulls down the Email through subscription?

Although I came up with the idea independently, I wasn't the first. Upon research, I found out that a sender-based storage concept coined IM2000 was originally proposed by professor Daniel J. Bernstein (djb) about 10 years ago. Where IM2000 differs from mine is that it's not truly pub/sub because tiny notifications are sent to the recipient.

There appears to be many merits to a pub/sub sender-based storage Email concept. For example, it puts the burden of storage on the sender, not the recipient who may have a disk quota from her service provider. Should someone send you a 20MB attachment and you're already at 95% of your 100MB quota, you don't have to worry about the message bouncing because there's not enough space for it. But if you're the sender and your quota's at 95%, it's your responsibility to upgrade your account to one with bigger quota, and in the meantime you can still receive messages.

Pay for what you do, not what others do to you. Seems fair to me.

[UPDATE Nov 21, 2007] Note to self: there is a few issues with sender-based storage:

  • how should searching be handled?
  • shared folder won't work
  • offline mode won't work
  • what if sender-side data is missing?


I currently live in San Francisco Bay Area. For the past seven years, I have been designing and building messaging solutions for Sun.


« July 2016