Tuesday Sep 11, 2007

Breaking News: Messaging Server 6.3 and ZFS

It's official! Messaging Server 6.3 now supports the ZFS file system, with the following caveats:

  • You must patch Messaging Server 6.3 to 6.3p3 (6.3-4.01)). This patch has been released to SunSolve and should show up the week of September 10. See below for patch IDs.
  • You must be running a minimum of Solaris OS 10 U3 for Messaging Server 6.3 to support ZFS.
  • ZFS support does not apply to Linux platforms.

Other requirements:

  • Do not perform a ZFS snapshot when the message store database snapshot is running. Schedule ZFS snapshot to run 30 minutes after a message store database snapshot is run. This will allow the database snapshot to complete before the ZFS snapshot. Message store database snapshot frequency is controlled by local.snapshot.interval.
  • Before restoring a ZFS snapshot make sure all Messaging Server processes are stopped.

Patch Information

Platform
32 Bit
64 Bit
SPARC
120228-23
126479-04
x86
120229-23
126480-04

More Details

If you want to backup your Message Store for disaster recovery, you can use a file system snapshot tool such as ZFS snapshot to capture a snapshot of the file system. The file system snapshot must be an atomic point-in-time snapshot (as is the case of ZFS), otherwise, the backup can have inconsistent data.

It is best to capture all message store data (message store partitions and database files) with the same file system snapshot instance. However, if this cannot be done, then capture the data in separate file system snapshots at the same time. And if this can't be done, capture the message store data in separate file system snapshots in the following order:

  1. Message store database snapshots. Note that database snapshots (configured with local.store.snapshotpath) are different from file system snapshots.
  2. Database files (data-root/store/mboxlist)
  3. Message store partition paths (store.partition.\*.path)
  4. Message store partition message paths (store.partition.\*.messagepath)
  5. Configuration data (data-root/config)

If the partitions and the database files are not backed up with the same point-in-time snapshot, run reconstruct -m after restoring the file system snapshots. This will synchronize the database and store partitions. Again, it is best to capture all message store data with the same file system snapshot instance to avoid doing a reconstruct.

If the message store database snapshot (db snapshot) is running when the file system snapshot is taken, the file system snapshot can contain an incomplete db snapshot. Although an incomplete db snapshot is harmless, it is best to schedule the file system snapshot to run 30 minutes after the db snapshot. This will ensure that a complete db snapshot is included in the file system snapshot.

Friday Aug 17, 2007

New Messaging Server 6.3 Patch Available

Reminder: The following Messaging Server 6.3 patch (MS 6.3p3 (6.3-3.01)) is now available on Sunsolve:

Platform
32 Bit
64 Bit
SPARC
120228-22
126479-03
x86
120229-22
126480-03
Linx
120230-22
N/A

The experts tell me that, among other things, this patch fixes the quota warning problem noted in BugID 6574627, and also provides VCS support.

Friday Aug 10, 2007

Messaging Server 6.3: Troubleshooting the Message Transfer Agent

I recently worked with a couple of our Messaging Server gurus to come up with an indepth article titled "Troubleshooting the Message Transfer Agent for Sun Java System Messaging Server."

Synopsis: This article describes how to troubleshoot the Message Transfer Agent (MTA) for Sun Java System Messaging Server, specifically message buildup in channels, including the TCP channels (tcp_local and tcp_intranet) and the ims-ms channel. This article treats troubleshooting as a process of learning about the MTA so that you can identify signs of problems and can then respond with possible solutions.

Get the article.

Wednesday Aug 01, 2007

Comms Suite 5 Q&A: Setting up Single Sign-On

Q. I need to set up SSO for Communications Express. What has changed between Messaging Server 6 2005Q4 (Java ES 4) and Messaging Server/Communications Express 6.3 (Comms Suite 5)?

A. Jhawk writes:

In the prior releases, we needed SSO to maintain authenticated sessions
between two of our HTTP servers. The two HTTP servers are the web
container (web server/app server) for the UWC servlets and the WebMail
server (mshttpd). Going into our Comms Suite 5 (UWC 6.3), we moved the
WebMail server behind the web container.

prior to 6.3
|------- web container(uwc)
web browser ----| (sso between)
|-------webmail server (mshttpd)

in 6.3
web browser----web container(uwc) --- webmail server(mshttpd)

Now in 6.3, we do not require SSO to maintain authenticated sessions
between our internal services. However, we do still support SSO to
co-exist authentication with 3rd Party web applications such as Portal
Server.

Tuesday Jul 31, 2007

Welcome Guillaume!

New to me: Sun GDD blogger. Give him a visit.

Monday Jun 25, 2007

Messaging Server, Message Store, and ZFS: Hang On...

I've been trying to get an answer on whether the lastest version of Messaging Server, 6.3, supports ZFS yet. Unfortunately, it looks like the answer is to hang on for the next release.

In the meantime, for more information, I found this thread on the Messaging Server software forum to be about as informative as anything else we've said publically on the matter.

Friday Jun 22, 2007

Messaging Server HA: Two Nodes Are Better Than One


Two Heads from the Banquet of the Officers (after Frans Hals), by John Singer Sargent

One of our Messaging Server developers just released a feature article that I know will go a long way towards filling a long standing info gap on this subject: Configuring Sun Java System Messaging Server 6.3 With Sun Cluster 3.1 Software. If you want to know how to definitively install and configure a two-node, asymmetric, high availability cluster for Messaging Server 6.3 with Sun Cluster 3.1 software, get the article here.

Tuesday Jun 12, 2007

Extra, Extra, Read All About It: New Messaging Server 6.3 Administration Guide

The Sun Java System Messaging Server 6.3 Administration Guide has just been re-released with many fixes and enhancements, specifically to documenting what we refer to as "MTA Minor Features." Get the book here:

Wednesday Jun 06, 2007

Communications Suite: Confirming That Upgrade Procedure from 5.2

re: upgrading directly from iPlanet Messaging Server 5.2 to Communications Suite 5 (that is, Messaging Server 6.3), the Sun Java Communications Suite Upgrade Guide is clear on the upgrade path:

http://docs.sun.com/source/819-7561/planning.html

To quote:

While it is possible to upgrade all previous releases of Communications Suite software to Communications Suite 5, the only certified upgrades are from Java Enterprise System 2005Q4, Java Enterprise System 2005Q1, and Java Enterprise System 2004Q2. Upgrades from earlier releases are not documented in this Upgrade Guide.

Thus, upgrading directly from iPlanet Messaging Server 5.2 is not a certified way to go. From 5.2, you're supposed to go to 2005Q1 and then from 2005Q1 you can upgrade to 6.3. It's a two-step process. Headache, yes. But that's the certified route to take.

Friday Jun 01, 2007

Messaging Server: Cross Site Scripting (CSS or XSS) Vulnerability

The Sun Security Blog has a new entry on Messaging Server, Sun Alert 102909 "Cross-site Scripting Vulnerability in Sun Java System Messaging Server."

Two points of clarification:

  • This does not occur in iPlanet Messaging Server 5.
  • The issue also applies to the HP-UX and Windows platforms.

Thursday May 31, 2007

Diagnosing and Recovering Calendar Server Database Corruption

Thanks to the Calendar Server engineering team and one of the Comms writers, there's a new article up on BigAdmin titled "Recovering Sun Java System Calendar Server Databases." This article discusses how to diagnose Sun Java System Calendar Server calendar database corruption and describes the best ways to recover a database in various situations. The example at the end offers a cookbook list of steps for recovering a database after it has been corrupted.

Get the article.

Wednesday May 16, 2007

Messaging Server Upgrade: The Missing Link

Heads up: For those of you patiently awaiting the Messaging Server 6.3 upgrade patch (that is, the patch that will take you from pre-6.3 releases to 6.3), there is news: it should be appearing on SunSolve in the next few weeks. Once it is made available, I'll be sure to alert the Comms community, as well as update the Comms Hub Updates tab.

One other heads up: The patch number will be a different revision than the one initially specified in the C5 Upgrade Guide.

Monday May 14, 2007

Comms Suite: Pull or Push, It's Up to You

Doctor Doolittle's pushmi-pullyu (pronounced "push-me-pull-you")

Traditional email access (dial-up) was and still is "pull" based. You log in to your mail server, your mail client polls the server to see if there is new mail, and if so downloads it to a mailbox in your home directory. The same process happens at regular intervals afterwards as well.

The IMAP protocol, in effect, introduced clients to "push" email. Through support for polling and monitoring of the server, the IMAP protocol enables clients to become aware of new messages, fetch message data, and choose to dowload the message. Wireless devices were next to become 'instant-on' email clients, but used proprietary protocols to achieve that state of bliss.

Now the IETF, in the form of the 'Lemonade Profile,' has provided a standard way to use the existing IMAP IDLE command along with SMTP modficiations for push email.

Communications Suite 5 (released March 2007) supports IMAP IDLE (aka push email). Support for the IMAP and SMTP extensions described in the Lemonade Profile, RFC 4550, is planned for the next major Communications Suite release.

The advantages of IMAP IDLE are:

  • Mail clients do not have to poll the IMAP server for incoming messages.
  • Eliminating client polling reduces the workload on the IMAP server and enhances the server's performance. Client polling is most wasteful when a user receives few or no messages; the client continues to poll at the configured interval, typically every 5 or 10 minutes.
  • A mail client displays a new message to the user much closer to the actual time it arrives in the user's mailbox. A change in message status is also displayed in near-real time.
  • The IMAP server does not have to wait for the next IMAP polling message before it can notify the client of a new or updated mail message. Instead, the IMAP server receives a notification as soon as a new message arrives or a message changes status. The server then notifies the client through the IMAP protocol.

To configure IMAP IDLE in Messaging Server 6.3, see To Configure IMAP IDLE.

Monday May 07, 2007

Messaging Server: More on the War Against Spam

Here's a useful tip for configuring Sun Java System Messaging Server to combat spam: delay sending the SMTP banner for a brief time (half a second, say), then clear the input buffer, and finally send the banner. The reason this works is that many spam clients are not standards-compliant and start blasting SMTP commands as soon as the connection is open. Spam clients that do this when this capability is enabled will lose the first few commands in the SMTP dialogue, rendering the remainder of the dialogue invalid.

This feature has now been implemented in Messaging Server 6.3. You can enable the feature unconditionally by setting the BANNER_PURGE_DELAY SMTP channel option to the number of centiseconds to delay before purging and sending the banner. A value of 0 disables both the delay and purge.

The PORT_ACCESS mapping can also be used to control this capability. Specifying $D in the template causes an additional argument to be read from the template result, after the mandatory SMTP auth ruleset and realm and optional application info addition. This value must be an integer with the same semantics as the BANNER_PURGE_DELAY value. Note that any PORT_ACCESS mapping setting overrides the BANNER_PURGE_DELAY SMTP channel option.

Thursday May 03, 2007

Part II, Troubleshooting MTA Message Queues

As I mentioned previously, the topic of message build-up in the MTA message queues prompted this writeup. Here's the second part of this installment, providing some details on what you can do if you discover that your deployment is having an actual problem with message build-up.

Before You Begin

At a minimum, you need to enable logging for the channels that you want to troubleshoot. The amount of logging that you set (log level) depends on your situation. To enable logging on a channel and learn about other options, see Managing MTA Message and Connection Logs in Sun Java System Messaging Server 6.3 Administration Guide for more information.


Troubleshooting TCP Channels

This section describes a general approach to troubleshoot build up of messages in TCP message queues to determine if there is an actual problem.

  1. When you suspect that there is something happening that is more than the store-and-forward aspect of messages building up in a message queue, begin by using the imsimta qm summarize command.

    For more information, see imsimta qm in Sun Java System Messaging Server 6.3 Administration Reference.

  2. The imsimta qm summarize command can greatly impact your system if you have a large backlog of messages. Instead of running this command frequently, consider using the imsimta qm messages channel command instead.

    This command lists the destination hosts for which messages are queued in the specified channel. This command also lists how many messages are waiting for their next schedule retry (the delayed messages column) versus how many are ready to be retried now (active now). When you use the imsimta qm messages command, you must specify a channel name; a wildcard is not valid input. For example:

    # imsimta qm
    qm.maint> messsages tcp_local
    host                             active messages  delayed messages
                             example.com           0                 2000
                             sesta.com             0                 3000

    In this example, 2,000 messages are waiting for their next scheduled retry to be delivered to example.com, and 3,000 are waiting for their next scheduled retry for delivery to sesta.com.


    Note - The Messaging Server 6.3 release removed the imsimta qm messages command. However, Messaging Server 6.3 does contain a new, useful command—imsimta qm jobs—to help understand why messages are not being delivered.


    This information is also useful for the situation where messages have failed and are waiting to be retried. If you see there are many messages in a channel queue, but most of them are delayed, this probably indicates the problem is with the remote domain. See Destination Host Problems for information on how to workaround this problem.


    Note - A message can be in process of being tried by a job, or on a channel waiting to be tried, or on a channel waiting to be retried. The messages command active messages column includes messages which have not been tried yet and those which were previously delayed and are now ready to be tried again. That is why you might see a zero (0) in this column. In Messaging Server 6.3, you can see the messages being retried with the new jobs command.


    iPlanet Messaging Server 5. Use top -to channel or top -domain_to channel to analyze what is going on in that channel.

  3. Look for trends on your system. For example, when most of the mail is all destined for one remote domain, check the status of that remote domain.

    Additionally, look in the mail.log_current file to determine what has happened in recent history when you tried to send mail to that remote domain.

  4. Use the imsimta qm dir -to address command to select a group of messages. Then use this information to look at the delivery attempt history of some of the messages. (You use the sequence numbers from the dir listing). Often, you will find that these messages are all non-delivery notifications for spam, which was not deliverable. If this is the case, determine how those original spam messages got into the system in the first place. Verify that the messages are spam by using the imsimta qm subcommands dir, read, and history. If this is indeed the case, think about routing the non-delivery notifications through a different outbound channel, thereby preventing them from choking the normal tcp_local channel queue.

    For example, use the notificationchannel and dispositionchannel keywords to specify an alternate process channel to queue delivery status notifications and modify status notifications, respectively. Then you use source-specific rewrite rules to direct messages from these process channels to a particular tcp_\* channel set to only use a few processes/threads. For more information, see Source-Channel-Specific Rewrite Rules ($M, $N) in Sun Java System Messaging Server 6.3 Administration Guide.

  5. Verify that the master process for the channel is started. The tcp_\* channels all use the smtp_client process. To find out which process is associated with which channel, in respect to dequeuing, see the master_command parameter in the associated channel block in the job_controller.cnf file.


Destination Host Problems

When you have determined that messages are queued to an unavailable remote host, you have two options:

  1. Create a new channel for the host. If this host is consistently a problem, all future email will go to this new channel. For existing messages that are enqueued, you can either wait for the problem with the destination host to be resolved or delete the messages from the queue.

  2. Increase the number of delivery threads for the channel, or set a ceiling on the number of queued messages that will trigger a new thread or process to start. See the max_client_threads parameter in the channel option file and the threaddepth channel keyword, respectively.


Troubleshooting the ims-ms Channel

This section describes a general approach to troubleshoot build up of messages in the ims-ms channel. The four general cases where the ims-ms channel shows a build-up of messages in the queue are:

  • IMAP_MAILBOX_LOCKED. While you might see this error in a message file that is briefly in the queue area, typically such a message file doesn't remain for long. The error only repeats in a message file in the queue area if the mailbox is remaining locked for an extended period. The job controller retries delivery of these messages after short delays until either the message gets delivered or a different error is encountered.

  • IMAP_MAILBOX_BADFORMAT, IMAP_MAILBOX_NOTSUPPORTED. The mailbox is most likely corrupted. This case rarely occurs. You might want to use the reconstruct command for these cases.

  • IMAP_IOERROR. The message store is most likely corrupted or otherwise inaccessible. This case occurs even more rarely.

  • IMAP_QUOTA_EXCEEDED. The user or users are over quota. This is the most common case, which this technical note discusses below.


Note - If the channel gets a permanent delivery failure error, then the message is immediately bounced and does not remain in the ims-ms queue area.


To troubleshoot the ims-ms channel, use the following high-level approach:

  1. Perform a similar investigation as you would for tcp_\* channels by using the imsimta qm summarize command to view what is happening on the system.

  2. Use the imsimta qm history command to examine the message IDs to detect if there are different sorts of messages. For example, you might see:

    Message id: 800
    Filename: /opt/SUNWmsgsr/data/queue/tcp_local/001/ZZf0b0KaNZykG.00

    A message's file name starting with ZZ indicates that it has not been tried yet. The message file name is a counter starting at ZZ and decremented (ZY, ZX, and so on) each time the message is tried, fails, and is reenqueued for later retry. Thus, a ZZ\* file name has not been tried yet, and there is no history.

    In general, but not always, when you have non-ZZ\*, non-.HELD files in the queue area, you have the IMAP_QUOTA_EXCEEDED case. (The frequency with which you see IMAP_MAILBOX_LOCKED conditions probably depends upon user and email client characteristics. This condition is more common with users who like to receive and move around lots of large attachments but it should typically occur rarely.)

    For a site that enforces quota, probably most of the non-ZZ\*, non-.HELD messages in the ims-ms queue area are there because of the recipient user being over quota. Verify this is the case by running the imsimta qm command with the history subcommand. You should see “over quota" in the history of the over-quota messages.


    Note - In iPlanet Messaging Server 5.2, the imsimta qm top command was enhanced to have more sorting options.


  3. Are there any Q status messages in the mail.log_current file pertaining to the ims-ms channel? When you see “mailbox is busy” and Q status in the mail.log_current file, then the message is put back on the queue to be retried later as per the job controller's scheduling and the backoff keyword on the channel.

  4. If not, check that the ims_master process is running. Are there any errors in its log file (the imta file)? The ims_master process could be hung. Use the imsimta process command to verify running processes.

Use the following strategies for users who become over quota:

  1. Inform users of the need to perform mailbox maintenance to return to under quota status, or increase their quota.

  2. Reduce the time that mail is queued for over quota accounts before being bounced back as over quota. See the store.quotagraceperiod configutil parameter. If you don't want to queue email for over quota accounts (and bounce the message straight back), set this parameter to 0 (that is, no grace period). This parameter is available in iPlanet Messaging Server 5 as well.

  3. For Messaging Server 6, you can enable the local.store.overquotastatus configutil parameter. This enables quota enforcement before messages are enqueued in the MTA and prevents the MTA from filling up.


More About the ims-master Process

At times you might see the ims-master process shutting down and starting up in the log file:

# > imta /opt/SUNWmsgsr/logs
# grep "Sun Java" imta
[30/Aug/2006:17:05:05 -0400] learn ims_master[19736]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) shutting down
[30/Aug/2006:17:05:20 -0400] learn ims_master[28310]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) starting up
[30/Aug/2006:17:07:24 -0400] learn ims_master[28310]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) shutting down
[30/Aug/2006:17:07:32 -0400] learn ims_master[28380]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) starting up
[30/Aug/2006:17:19:31 -0400] learn ims_master[28380]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) shutting down

This is normal operation and does note indicate a problem. This is a “notice” message (not an “error” or “critical” level message). As with all channel jobs, ims-ms channel jobs shut down from time to time based on either having nothing to do, or based on “timing out” (getting old). Then the job controller restarts new jobs as needed.


Configuration Issues

You might want to increase the number of processes the job controller can start for the tcp_local, tcp_intranet, or other tcp_\* channels, or increase the number of threads each of those processes will start. You might also want to give the tcp_local channel its own pool. If you observe queued messages (total across all queues) to be greater than 100,000, increase the value of MAX_MESSAGES for the job_controller.cnf setting. See Job Controller Configuration File in Sun Java System Messaging Server 6.3 Administration Reference for more information.


Additional Information

This section contains additional information to help you understand MTA operations.


What Are .HELD Messages?

If the MTA detects that messages are bouncing between servers or channels, delivery is halted and the messages are stored in a file with the suffix .HELD in the msg-srv-base/data/queue/channel directory. Typically, a message loop occurs because each server or channel thinks the other is responsible for delivery of the message. You need to manually fix these .HELD messages with the imsimta process held command.

There is an unfortunate collision of terminology and concepts between .held messages and the hold channel. And worse still, the command to process .held messages is called release, whereas the command to process messages on the hold channel is called process_held.

You use the hold channel to hold messages of a recipient temporarily prevented from receiving new messages. For example, you might be moving a user's mailbox and want to hold new incoming messages. The hold channel is located in the msg-svr-base/queue/hold directory. Messages are written to this queue as ZZxxx.held files. Because the job controller doesn't “see” these .held files, they are not dequeued for delivery. You release these files with the imsimta qm release command, and the reprocess daemon reprocesses them.

See To Temporarily Hold Messages Using the Hold Channel in Sun Java System Messaging Server 6.3 Administration Guide for more information.


How Does a Message Become a .HELD Message?

Messaging Server makes use of MAX_\*_RECEIVED_LINES options that you set in theoption.dat file to determine when a message is put into the .HELD state. The most relevant options and their default values are:

  • MAX_LOCAL_RECEIVED_LINES=10

  • MAX_RECEIVED_LINES=50

  • MAX_TOTAL_RECEIVED_LINES=100

Once a message has looped through the MTA enough to accumulate MAX_RECEIVED_LINES header lines indicating the local MTA, then the message becomes .HELD. You can cause the MTA to immediately recognize that it has connected to itself, rather than waiting to accumulate MAX_LOCAL_RECEIVED_LINES local Received: headers, by specifying the loopcheck keyword on the appropriate channel(s) in the imta.cnf file.


For More Information on Troubleshooting the MTA

Use the following to aid in troubleshooting the MTA:

About

Reporting about Unified Communications Suite Documentation, including news, Comms 101, documentation updates, and tips and tricks.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today