Wednesday May 02, 2007

Comms 101: Troubleshooting MTA Message Queues

I've been working with two of our Messaging Server experts to come up with an article on Messaging Server's Message Transfer Agent (MTA) and how it handles the build-up of messages in its message queues. This has been a fairly frequent topic of inquiry from the Comms community that presents itself as, 'I'm noticing a build-up of messages in my queues, should I be worried about it.'

The short of it is, not necessarily, because of the way the MTA is designed to work as a store-and-foward message system. The long of it is, maybe, but there's more to consider than just number of messages in the queue.

So, here's part one (of two) on my take on this situation, based on the experiences of two of our support engineers. I'd be interested in hearing from others and their experiences, to see if this article should be expanded/corrected.

Troubleshooting Sun Java System Messaging Server MTA Message Queues

Part I

This article describes how to troubleshoot the Sun Java System Messaging Server Message Transfer Agent (MTA), specifically message build-up in channels, including the TCP channels (tcp_local and tcp_intranet) and ims-ms channel.

Products covered by this article are:

  • Sun Java System Messaging Server 6

  • iPlanet Messaging Server 5


Note - This technical note assumes you are running Sun Java System Messaging Server 6. Where appropriate, iPlanet Messaging Server 5 commands are also mentioned.


This article contains the following topics:

  • About the MTA and Channels

  • When Is a Message Queue Having a Problem?

  • Main Causes of Backed-up Message Queues

  • Before You Begin<

  • Troubleshooting TCP Channels

  • Destination Host Problems

  • Troubleshooting the ims-ms Channel

  • Configuration Issues

  • Additional Information


About the MTA and Channels

To begin understanding why messages build up in a channel queue, and whether an actual problem might be occurring on your system, you must first understand how mail messages flow through the MTA.

The MTA can be thought of as Messaging Server's central brain, responsible for message routing. The MTA takes in messages via SMTP sessions from other systems and decides what to do with those messages. The first stop for a message is the MTA SMTP server, which executes programs to handle the SMTP session. Based on numerous configuration possibilities, the SMTP server processes the message, which could include message blocking, address changing, or channel enqueueing.

Actually, the sequence of events is a bit more complex. The dispatcher spawns the SMTP server by listening on port 25 (or whichever port is defined for the SMTP server in the dispatcher.cnf file). When the dispatcher detects an attempt to connect to port 25, it starts an SMTP process to handle the incoming connection. The SMTP server typically decides whether to accept or reject the message based on numerous configuration possibilities. During the SMTP dialogue, the MTA machinery kicks in to decide what to do with the message. However, the PORT_ACCESS mapping table works with the dispatcher rather than the SMTP server to allow or block access to certain ports such as the SMTP port (port 25).

The focus of this technical note is the decision to route the message to a channel where the message is to be enqueued. A channel is a message connection with another system or destination. Once enqueued to a channel, the message's destination could be another server (either on the Internet or on your company's intranet), a remote message store, a specific domain name, a channel for extra processing, such as virus filtering, or a local message store.

When a channel contains messages but is not delivering them, the messages build up in the channel's message queue. The message queues are directories, located by default in the msg-svr-base/data/queue/channel/ directory. In this way the MTA holds the messages for future delivery when whatever situation preventing their delivery is resolved.

The specific channels discussed in this technical note are the TCP channels (tcp_local and tcp_intranet) and the ims-ms channel. The tcp_local channel is responsible for routing messages to the Internet, while the tcp_intranet channel is responsible for delivering messages to remote message stores on your company's intranet. The tcp_intranet channel also routes messages to any intermediary internal systems on their way to another system. The ims-ms channel is responsible for delivering messages to the local message store.

For a complete description of the MTA architecture and message flow, see MTA Architecture and Message Flow Overview in Sun Java System Messaging Server 6.3 Administration Guide.


When Is a Message Queue Having a Problem?

Though this might be counter-intuitive, in general, it is not a problem for messages to build up in a message queue. Indeed, the MTA is designed to handle this situation. Internet mail (SMTP) is a store-and-forward mail system. Internet mailers are designed to store the messages that cannot yet be delivered. Keep in mind that it is perfectly normal to not be able to deliver outgoing SMTP messages immediately. Network problems, problems on remote hosts, problems with remote users' mailboxes, and so forth, are common cases that cause the MTA to hold and not deliver messages. MTAs, including Messaging Servers', therefore can store and retry outgoing messages. An often encountered case has to do with message store users being over quota and hence their messages are waiting for retry. From the MTA design point of view, this exactly the same case as being unable to deliver a message because of a network problem. The MTA handles the over quota situation with exactly the same underlying mechanisms: the messages are stored in queues where they will be retried for delivery. Thus, just because you are seeing a build-up of messages in queues is not necessarily cause for alarm.

The real issue about backed-up queues is when should you have cause to worry and so try to troubleshoot the source of the problem. The next sections explain the main causes of queue backups and how to go about troubleshooting such situations.


Main Causes of Backed-up Message Queues

In general, four situations can cause messages to back up in the MTA message queues:

  • Performance problems. Simply put, this occurs when your system is unable to keep up. Performance problems manifest themselves by high system loads and processes running flat out. In such situations, you need to perform an in-depth look at the system to be able to tune it and reduce the load.

  • Destination host problems. Regardless of what is causing problems on the destination host (networking problems, system problems, and so on), this situation can cause all delivery threads to become “stuck,” stopping valid email from getting through, or causing a lot of queued emails. The “stuck” problem presents itself as a lot of active messages waiting to be processed but with a low load. The “queued” problem shows as lots of queued email with long delivery attempt history.


    Tip - To contact the administrator of a domain that is causing you problems, use the whois command or http://www.netsol.com/cgi-bin/whois/whois.


  • Problems with Messaging Server itself. An example of a Messaging Server problem is when the stored process has an orphan lock for a user account, resulting in message build up for users in the ims-ms channel.

  • Configuration issues. Two common configuration issues include the job controller ignoring queued email or queued email due to overquota accounts, or queued email due to slow directory response, common for big mailing lists.

Part Two of this article will discuss courses of action to the above situations, and provide more information on troubleshooting. Stay tuned.

Tuesday May 01, 2007

Calendar Protocol Faceoff

For those interested in calendar developments, head over to arnaudq's blog, where you'll find a great comparison of WCAP (Web Calendar Access Protocol,Sun's proprietary calendar protocol), and the proposed standard calendar protocol CalDAV. As Jim Parkinson pointed out here, we're looking at a "a calendar server based on the latest CalDAV standard."

Wednesday Apr 25, 2007

New in Calendar Server 6.3: Event Organizers Can Receive Reply Notifications

The question came up recently about whether the new version of Calendar Server, 6.3, could email notifications to organizers when an attendee replies to an invitation. The answer is YES.

How do you enable this feature? By configuring the ics.confparameter ine.reply.enable. Set it to "y" to enable the feature for the entire system. Set it to "n" to disable the feature. The feature is enabled by default.

The three reply types are: accept, decline, tentatively accept. The notification indicates whether the reply is to a single invitation or to an recurring event. The following new message format file parameters were added. The corresponding format files were also added:

  • calmail.imipeventacceptnotification.fname= "mail_eventacceptnotification.fmt"

  • calmail.imipeventdeclinenotification.fname= "mail_eventdeclinenotification.fmt"

  • calmail.imipeventtentativeacceptnotification.fname= "mail_eventtentativeacceptnotification.fmt"

  • calmail.imipeventacceptnotificationrecur.fname= "mail_eventacceptnotificationrecur.fmt"

  • calmail.imipeventdeclinenotificationrecur.fname= "mail_eventdeclinenotificationrecur.fmt"

  • calmail.imipeventtentativeacceptnotificationrecur.fname= "mail_eventtentativeacceptnotificationrecur.fmt"


Note –

This feature is not a user preference. That is, it is a system wide configuration parameter, so it applies to all users who send invitations.


For more information about configuring Calendar Server for email notifications, see To Enable Email Notifications in Sun Java System Calendar Server 6.3 Administration Guide.

Tuesday Apr 24, 2007

Calendar Server 6.3: csstored Now a Required Process

A heads-up to those of you who will be running Sun Java System Calendar Server 6.3: the csstored process has changed in the new 6.3 release. Whereas it used to be a PERL script that could be enabled and disabled at will to implement automatic backups, and could be stopped and started at will, now in version 6.3, csstored is a required daemon that is started and stopped automatically by the start-cal and stop-cal scripts, along with other required daemons.

The csstored daemon manages the various Calendar Server databases. Because each service that accesses the store depends on the successful start of this store service, it should remain running on all servers, both front-end and back-end, whenever the Calendar Server system is running.

It is no longer optional to run this daemon, whether you decide to configure automatic backups or not. Previously, you could disable the script from running by setting the ics.conf parameter local.store.enable to "no", but now the local.store.enable parameter must be set to "yes", which is the default.

The Calendar Server 6.3 Administration Guide will be re-released shortly with this updated information (Chapter 9).

Friday Apr 20, 2007

Sun Java System Messaging Server 6.3: New White Paper

Per a previous post, I alluded to a new white paper out on Messaging Server 6.3. Astute readers alerted me to the fact that I may have, uh, jumped the gun: link not found.

I'm here to report that as of today, this paper is available:

http://www.sun.com/software/products/messaging_srvr/wp_messagingsrvr63.pdf

Thursday Apr 12, 2007

Configuring SpamAssassin with Messaging Server

I gather that you don't need any brains to be a spammer, as evidenced by the above email. Yeah, dealing with this crud is a fact of the Info Age. Coincidentally, I just came across a post to help. Check it out: Air's Blog has a nice step-by-step entry of installing SpamAssassin with Sun Java System Messaging Server.

Monday Apr 09, 2007

Messaging Server: When Things Go Bump, er, Bounce in the Night

There was an interesting conversation on the Info-iMS alias last week concerning "bounced email" that caught my eye. Ah, another learning opportunity! To recap, then, here is another installment of Comms 101, courtesy of our Comms experts.

The specific error sited was:

nsmail Illegal host/domain name found (Too many failures to this host during this run; skipping this host: No such host/domain)

The short reply as to what's going on is fairly intuitive: the remote domain in question failed to resolve to a legal result, meaning that almost certainly this is a DNS issue (and not a Messaging Server MTA issue). Note that "other mail" will continue to make it through to your system (expected behavior), so don't be mystified by that activity. In addition, the error message is communicating that the problem has happened multiple times during the current operation (run) of the channel. The message bouncing is thus occurring because of an underlying DNS problem. To troubleshoot this problem, you would need to enable debugging (such as on LOG_CONNECTION) to determine the specific issue. Very likely, such a problem is transitory in nature but it is clearly not an MTA issue. Of final note: fancy configuration workarounds to such a situation are typically more work than troubleshooting and fixing the real problem.

The full detail on this thread can be found here:

http://lists.balius.com/pipermail/info-ims-archive/2007-April/027757.html

Friday Apr 06, 2007

More on Installing a Comms 5 Calendar Server 6.3 Deployment (Front End and Back End)

Update, 4/9/07:

A co-worker has correctly pointed out in the comments that I have this not quite right. It is the configurator.sh script itself that is causing the problem. You do not have the option while running the script to choose between IP address and hostname. The script basically choose the IP address rather than the hostname. But again, the fix is afterwards to manually update the ics.conf file for the parameter mentioned below.

- Factotum Management

4/6/07:

As I mentioned in a previous post this week, there's a Calendar Server 6.3 install issue that I want to make folks aware of.

When you use the configurator.sh script to configure your Calendar Server front end, in a front end/back end deployment, as described here, your are supposed to use a fully qualified domain name for the back-end host the front end is pointing to that you will use DWP to commmunicate with. If by mistake you put in an IP address of the host instead, which evidently the script lets you do, the configuration will not work. And then you'll have to manually go back and enter the FQDN of the host instead of the IP address for caldb.dwp.server.backend.ip in the ics.conf file.

And yes, we do have a bug (CR) filed for this.

Sun Security Blog

I just stumbled across this in the Sun blogosphere: Sun Security Blog

There was a recent alert and workaround posted (March 30) on NSS and SSL clients and servers, with a potential affect on Messaging Server:

Security vulnerabilities in the Network Security Services (NSS) implementation of SSL2 may affect both SSL clients (such as browsers) and SSL servers which make use of this library. As a result, the client or server may exit unexpectedly, which is a type of Denial of Service (DoS). For servers running on Microsoft Windows, they may present a remote code execution vulnerability.

In any case, this seems like a potential blog to follow on a regular basis.

Thursday Apr 05, 2007

Communications Suite Odds and Ends

As usual, lots to scrape off the Comms windshield. Here are some interesting factoids that have turned up in the past few days:

  • Installing a Comms 5 Calendar Server deployment, front end and back end (where the front end is using the DWP protocol to communicate with the back end): when you use the csconfigurator.sh program on the front end to configure Calendar Server, you are prompted to use the IP address of the back end to point to. Unfortunately, this needs to be the host name, so you'll have to correct that manually. (Bug has been filed.)
  • When deploying Connector for Outlook with Calendar Server/Communications Express, realize that Connector needs to access calendar in WCAP (a calendar protocol encapsulated in HTTP) and communicate directly to a calendar process that 'speaks' WCAP (cshttpd). In a multi-tier deployment, that means installing a Calendar Server front-end on your host running Communications Express. This calendar front-end prowides the WCAP services necessary for Connector, and then communicates to the back-end host using the DWP protocol (port 57779 by default; thus, be carefull with you firewall rules between front-end and back-end host). For more information on this kind of deployment:

    http://docs.sun.com/app/docs/doc/4439/6n4udv3em?a=view
    http://docs.sun.com/app/docs/doc/4439/6n4udv3cg?a=view

  • I pushed a new tech note to docs.sun.com today, should be available by tomorrow: Using Sun StorageTek 53xx NAS with Messaging Server Message Store.
  • Our deployment team says that you need an expert to install Comms, and even then, don't expect a problem-free time. Question: Is this news to anyone? I do hope, at least, that our customers have been seeing some improvement in this area from at least the docs standpoint. If not, you know you can always drop me a line and let me know.
  • Found this interesting tech note from our Access Manager brethren: Sun Java System Access Manager ACI Guide. Personally, I'd rather have to eat peanut butter and crackers stranded in the desert without water than to have to deal with AM ACIs, but here you have a new peek into what the heck's going on there.

Tuesday Apr 03, 2007

The Secret Life of Calendar Groups in Calendar Server 6.3 and Communications Express 6.3

We've been having an internal discussion about a 'new' feature in Calendar Server 6.3, namely, Calendar Groups, and how all this fits in with Calendar Server and Communications Express. According to the Calendar Server Release Notes:

Support for LDAP Groups in Calendar Server 6.3

It is now possible to create LDAP groups using Delegated Administrator. Groups have the following functionality:

  • A group is a list of users. The group does not “contain” the listed users. It is not a container.

  • A group can have a group calendar.

  • Invitations sent to a group reside on all the members' calendars, as well as the group calendar.

  • All members of the group share the same access rights to the group calendar.

  • There is no primary owner for a group calendar.

    - This should be corrected to: The group calendar does have an owner, and can have co-owners too.

Clarifying these points, then, with some discussion that took place today:

  • The group calendar that you create through Delegated Administrator is a Corporate Group--different from your usual Personal Group (created through the Communications Express Addressbook).
  • When subscribing to a group calendar through the Communications Express interface, you subscribe to the Personal not Corporate group. That is, the CE interface does not access the group calendar created in the LDAP directory (by using DA).
  • The cscal command manages calendars in the Calendar Database not the LDAP directory. Thus, if you go looking for your LDAP group calendar with cscal, you won't find it initially (see below for more info).
  • When you invite the LDAP group calendar, member calendars added to the group get the events. There is no 'physical' calendar for the group calendar but the individual user's (member's) calendars.
  • To manage a group calendar, you actually manage the member's calendars (in Comms Express).
  • Calendar Groups can be used for limited functionality, including invite to event, check availability, compose mail to group, and view calendar from Addressbook (in Comms Express).
  • Currently, when you create the group calendar, you must then access it the first time (use it in an invite, for example) before it 'shows up' using the cscal command. Unfortunately--and this is a big unfortunately--currently there is no client today with which you an invite the calendar.
I hope this explanation helps in how to use (or not use) the current implementation of group calendars in Calendar Server and Communications Express. Questions anyone?

Monday Apr 02, 2007

Minor Discrepancy: Clarifying Calendar Attachments in Comms Suite 5

We noticed this slight discrepancy in the Communications Suite 5 Release Notes:

In the Communications Express 6.3 section:

The Calendar component of Communications Express allows users to include attachments to an event or task.

However, in the Calendar Server 6.3 release section, we have:

While Communications Express, the Web user interface, does not support attachments yet, users of the Connector for Microsoft Outlook can now put attachments in their events and tasks, and can send attachments with invitations.

The answer is that the Communications Express 6.3 interface does indeed support calendar attachments, but the older Calendar Express interface does not. In fact, with respect to the Calendar Express interface, we are now saying (quote from Releaase Notes):

"The Calendar Express graphical user interface (GUI) has been deprecated in favor of the Communications Express GUI and will be removed from the distribution in the the next major feature release. Move to Communications Express as soon as possible.

Tuesday Mar 20, 2007

Calendar Server Pop-ups: Not Just For Breakfast Anymore

Ya know, it's actually kinda cool when we use our own products. I'm talking about getting pop-up reminders on your desktop for your Calendar Server appointments. As long as you're running both Calendar Server and Instant Messaging, you can make this happen. And believe it or not, we actually do just that at Sun. Here's how.

To get pop-up reminders, you must have Instant Messaging installed on your network and you must be logged into Instant Messenger. (Instant Messenger can be iconified if you like). With Sun Java Communications Suite 5, the IM configuration wizard configures the iim.conf file for pop-up notifications and there is nothing you need to add to the iim.conf file, unless you want to define an optional password. You still need to modify the Calendar Server ics.conf file and enable calendar reminders in the IM client.

How to do it:

  1. Edit the Instant Messaging iim.conf file as shown below, and restart Instant Messaging. (Only required for versions of IM prior to IM 7.2. You can skip the iim.conf configuration steps if running IM 7.2 or later.)
    ! calendar event config
    jms.consumers=cal
    
    jms.providers=ens
    jms.provider.ens.broker=.domain:57997
    jms.provider.ens.factory=com.iplanet.ens.jms.EnsTopicConnFactory
    !
    jms.consumer.cal.destination=enp:///ics/customalarm
    jms.consumer.cal.provider=ens
    jms.consumer.cal.originator=.domain
    jms.consumer.cal.type=topic
    jms.consumer.cal.factory=com.iplanet.im.server.JMSCalendarMessageListener
    jms.consumer.cal.param="eventtype=calendar.alarm"
    iim_agent.enable=true
    iim_agent.agent-calendar.enable=true
    agent-calendar.jid=calimbot..domain
    agent-calendar.password=
    iim_server.components=agent-calendar
    
    \*Note: .domain is the fully qualified hostname of the calendar server such as megademo.demo.iplanet.com
    
  2. Edit the Calendar Server ics.conf as show belown, and restart Calendar Server.
    caldb.serveralarms = "yes"
    caldb.serveralarms.dispatch = "yes"
    caldb.serveralarms.url = "enp:///ics/customalarm"
    caldb.serveralarms.contenttype = "text/xml"
    caldb.serveralarms.dispatchtype = "ens"
    
  3. Click the "Show Calendar Reminders" checkbox in the Instant Messenger client. This is contained in the "Alerts" tab in the Instant Messenger "Settings" popup.

Monday Mar 19, 2007

Messaging Server News: Telstra Completes Upgrade

Interested in improving your mail system, and refreshing your aging software and hardware. (Man, if that doesn't sound like a marketing intro, I don't know what is.)

Read about how Telstra (with a 300,000-strong ISP customer base) consolidated onto Messaging Server, T1000/T2000 servers, and Solaris 10 OS:

Sun completes TelstraClear mail upgrade
TelstraClear completes successful email systems consolidation

As I understand it, the Sun Groupware Migration Toolkit played a large part in this upgrade. You can read more about SGMT here:

Escaping Vendor Lock-in: Life After Microsoft Exchange

Thursday Mar 15, 2007

Calendar Server: Saving the Daylight Update

Update to Download the Calendar DST Fix Tool
Get this Sun tool (and instructions) to correct timings of events and tasks in Sun Java System Calendar Server, related to the DST switchover in US and Canadian timezones.

There is now a new version of this tool, which was released to SunSolve on 3/14. Per the Change Log on the DST Hub:

[2007-03-14] Added updates for DST Fix Tool Version 2, released 3/14.

About

Reporting about Unified Communications Suite Documentation, including news, Comms 101, documentation updates, and tips and tricks.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today