Messaging Server: MTA Preferred Practices

As I previously wrote about here, I've been collaborating with one of our Messaging Server experts on a white paper that we hope clarifies Sun's position on how to best deploy AV/AS services with the Messaging Server MTA. I'm posting the draft paper below in hopes of soliticing comments and feedback from the Comms community.

DRAFT: Deploying Sun Java System MTA Preferred Practices

Wojtek Chemijewski and Joe Sciallo, December 2007

In general, a typical Sun Java System Messaging Server deployment consists of front-end and back-end services. The front-end services enable users to access the system, while the back-end services store and provide access to data. In the past, we've heard disputes about how to deploy the MTA on the front end: either as an entrance gateway to the system, or “behind” an anti-spam/anti-virus service that also provides an MTA functionality. This article describes the recommended way to deploy your MTA with respect to anti-spam/anti-virus services.

This article contains the following sections:

Executive Summary

Sun Java System Messaging Server provides a highly reliable, high performance, full featured SMTP implementation in the form of its Mail Transfer Agent (MTA). This MTA enables you to integrate anti-virus/anti-spam (AV/AS) solutions through the use of plugins. These plugins provide message content to the AV/AS engine for its evaluation and scanning, while allowing the MTA to retain full responsibility for the message at the SMTP level. The MTA does not provide the AV/AS engine itself: Messaging Server relies on third-party vendor experts to provide these solutions. Such a best-of-breed architecture, built in to Messaging Server, provides you with the greatest flexibility of integration. Thus, you use the Messaging Server MTA for the SMTP message processing and an AV/AS solution of your choice for virus and spam processing.

Messaging Server Architecture Overview

A two-tiered messaging architecture provides the optimum design for scalability and reliability. Instead of having a single host run all the components of a messaging system, a two-tiered architecture separates the components onto different machines. These separate components perform specific specialized functions. As the load for a particular functional component increases—for example, more Message Storage is required, or more outbound relaying is needed—you can add more servers to handle the larger loads.

The Messaging Server two-tiered architecture consists of an access layer and a data layer.

The access layer is the portion of the architecture that handles delivery, message access, user login, and authentication. (The access layer is also referred to as the front-end layer or tier.) For a Messaging Server deployment, the access layer comprises the MTA, Messaging Multiplexor (MMP), and Webmail server. AV/AS services are also part of the access layer.

The data layer is the portion of the architecture that holds all the data. (The data layer is also referred to as the back-end layer or tier.) This includes the LDAP master servers and Messaging Server machines that are configured to store user messages.

For more information on Messaging Server architecture, see

More About the MTA and AV/AS Services

The MTA is one of the Messaging Server access layer services. The MTA receives, routes, transports, and delivers mail messages using the SMTP protocol. An MTA delivers messages to a local mailbox or to another MTA. The anti-virus/anti-spam check is performed on these mail messages along their entry to the system. The preferred way of integrating AV/AS services with the MTA is by accepting messages from the users and the Internet, providing them to the AV/AS software for a check, and then relaying these messages further (to other users or to the Internet) as applicable.

AV/AS software vendors usually provide some MTA functionality in their products. Furthermore, AV/AS software vendors tend to recommend that you deploy their software as the entry point to the system, and that their software rely messages to the MTA after they have been checked for spam or viruses. As this paper shows, deploying your AV/AS software in such a manner—in front of the MTA—is a suboptimal method. Indeed, if you opt for such a deployment, there are certain MTA features that you cannot even access. Furthermore, there are MTA features that, if used in such an architecture, cause performance degradation.

Deploying AV/AS Services with Messaging Server

To begin with, let's understand the two fundamental ways of developing a Messaging Server architecture for AV/AS such that the MTA is the component to first accept messages:

  • MTA Plugin (Integrated Approach). You deploy the AV/AS services through an integrated MTA plugin, which is dedicated for the specific AV/AS vendor. Such plugins usually work through a proprietary protocol for providing messages to the AV/AS scanner, and that scanner provides its verdict back to the MTA. The MTA then uses this verdict for further processing of the message.

  • SMTP Relaying. If there is no AV/AS plugin available, you can configure the MTA to accept the message first, and then relay it to the AV/AS service using SMTP. The AV/AS service should then relay the message back, adding information with the result of the scanning into the message headers. The MTA can then access that information and use it as the verdict for further process of the message.

Whenever possible, use the integrated approach when developing your AV/AS deployment for Messaging Server. The integrated approach provides more flexibility in the way messages are chosen for scanning. In addition, with the integrated approach, the MTA retains all responsibility for the message all the time it is processed. The integrated approach also provides only the content of messages to the AV/AS service for evaluation and verdict. Thus, no SMTP relaying is involved in providing the message for AV/AS processing, and so the AV/AS scanner does not need to manage queues nor generate non-delivery reports. Finally, with the integrated approach, the set of ESMTP extensions supported by the AV/AS' MTA does not influence the extensions that the clients can use. When an MTA relays messages to another MTA, it first queries for the protocol extensions that other MTA supports. In the case of the Messaging Server MTA, when a client first submits a message to the Messaging Server MTA, it queries for the set of supported extensions. Next, the Messaging Server MTA uses the DSN extension to request a successful delivery report. Finally, the Messaging Server MTA relays that message to the AV/AS service and discovers that the AV/AS MTA does not support the DSN extension. Thus, a request for a successful delivery report cannot be relayed any further. In case of the MTA plugin, the responsibility for message processing remains with the Messaging Server MTA, so no such effect happens.However, the integrated approach is possible only with the AV/AS entities, for which respective plugins exist.

MTA Functionality Lost Due to Lack of Source IP Address Information

When you deploy the AV/AS services “in front” of the MTA, the MTA does not have access to the IP address of the sending client. Instead, it has access only to the IP address of the AV/AS box. The result is that the MTA cannot make make any decisions (that is, possibly reject mail) based on the client IP address. The following situations describe how knowing the source IP address of the sending client is useful:

  • Connection Throttling, Possibly Across Multiple MTAs. The MTA can throttle connections based on the source IP address. Throttling means limiting the number of connections that a particular IP address can open to the MTA within a certain time period. You can configure the MTA to allow only a certain number of connections to be open within one minute, and reject all connections exceeding that limit. You can also penalize an IP address that opens more connections than allowed. The MTA will continue to reject such an IP address for some extended time, even if the rate drops below the limit. In Messaging Server 6.3, you can also deploy the Metermaid server, which hosts a throttling database, that you can use commonly across multiple MTA servers. With Metermaid, you can control how many connections a certain IP address opens, even if the IP address tries to connect to multiple MTAs (which is very possible, due to the the front-end tier's usual horizontal scalability).

  • Sender Policy Framework (SPF). Spam producers and email scammers often forge email by using false domain names and email addresses or by using legitimate domain names and email addresses in order to fool users into thinking that a message is from someone or some company they know. For example, a spammer could send email from an address such as and the user could be fooled into thinking the mail was actually from this address. Forging email may fool users into opening the unsolicited message, or worse, provide information to a false authority. Also, spammers prefer to send their email from legitimate domains that are not on an RBL list.

    Sender Policy Framework (SPF) is a technology that can detect and reject forged email during the SMTP dialogue. Specifically, SPF is a protocol that allows a domain to explicitly authorize the hosts that may use its domain name. In addition, a receiving host may be configured to check this authorization. SPF can thus significantly reduce the instances of forged email. If the IP address is not known, the SPF check cannot be performed. For more information on SPF, see

    The Messaging Server MTA supports SPF in a flexible way, enabling it to influence the action which is performed upon the SPF result or to incorporate the SPF result into further (for instance, per-user) processing. Messaging Server provides this capability so that you do not have to depend on the AV/AS, which may or may not be as flexible in handling complex scenarios.

  • Other DNS-based Checks. The Mail Abuse Protection System's Realtime Blackhole List (MAPS RBL) is a dynamically updated list of known unsolicited bulk email (UBE) sources identified by source IP address. The Messaging Server SMTP server supports use of the RBL and can reject mail coming from sources identified by the RBL as originators of UBE or spam. You an use the result of such a check to reject mail. You can also use this check to influence how the MTA further processes the message. For example, you can increase the likelihood of the message to be spam through a positive RBL check, and you can use that value in a user-based SIEVE filter.

  • IP-based Channel Switch. The MTA can switch the source channel based on the client IP address. Typically, all mail arriving on port 25 of the server is enqueued to the tcp_local channel, and characteristics of that channel are applied to the connection and to the mail submitted on that connection. However, it is possible to switch the connection to another channel based on the client IP address and apply different characteristics if the client's IP address belongs to certain pool. Examples of these characteristics include:

    • Whether encryption is required

    • Whether sender authentication is required

    • Whether the source channel should be further switched based on the fact that connection is encrypted or authenticated

    • Whether ETRN is allowed, the maximum message size, the recipient count limits, and so on

    You can also deploy source-channel-specific rewrite rules so that addresses in messages being switched to a specific channel are rewritten differently than the rest. You can use the source channel in mapping tables, which enable you to deploy regular-expression-based ways of processing messages. The most common example of this functionality is the default behavior of mail coming from clients whose IP address belongs to the value specified in the INTERNAL_IP mapping table: the mail is switched to the tcp_intranet and allowed to go out to the Internet.

  • Using IP Address in Mappings. The client IP address can be directly used in mapping tables, which the MTA accesses at various stages of mail processing. You can use the IP address in PORT_ACCESS (to make decisions based on IP addresses), FROM_ACCESS (to make decisions based on the IP addresses, application information, source channel, envelope From: address, and authentication information), MAIL_ACCESS, and ORIG_MAIL_ACCESS (to make decisions based on IP addresses, application information, source and destination channels, and envelope From: and To: addresses).

  • Logging. You can configure the MTA to log the sending client IP address. Not being able to capture and log this IP address because of locating the AV/AS service in front of the MTA complicates analysis of the MTA behavior and troubleshooting problems.

Functionality Lost Due to Lack of Authentication Information

It may be possible to configure an AV/AS entity to provide means for SMTP authentication (using LDAP bind for instance), and to relay mail to the MTA without authenticating. In such cases, the MTA does not have access to the authentication information. Therefore, you cannot use MTA functionality that requires users to authenticate or to modify the MTA's behavior based on the authentication information. Examples of lost functionality in this case include:

  • Authentication Enforcement. You can enforce authentication to send mail out to the Internet (this is the typical setup). You can also enforce authentication on certain source channels. Knowing the sending client's IP address enables switching the source channel based on the client IP, and knowing that the users authenticated enables you to enforce authentication on messages from certain IP address ranges.

  • Per-user Channel Switch. If you provision a user with the mailSMTPSubmitChannel LDAP attribute, the source channel can be switched to a channel specific for that user. For the MTA to do so, the user must authenticate. Once the switch is done, this particular channel's characteristics will be applied for all mail originating from that user.

  • List Submission Authorization. You can provision a mailing list with attributes that specify who can send mail to that list, for example, mgrpDisallowedBroadcaster, mgrpAllowedBroadcaster, mgrpDisallowedDomain, and mgrpDisallowedDomain. The authentication information can be used in these checks.

  • Message Headers. The fact that the user authenticated is put into the Received: header of a message. Also, you can configure the MTA to put the Sender: header line, containing information who authenticated in order to send mail, into the message. This behavior can be channel-specific, so the ability to switch the channel based on an IP address of the client may be of use here, as well.

  • Logging Who Authenticates. The authentication information (information who authenticated) can be recorded in log files.

Other Functionality Lost

Other information lost when the AV/AS relays messages to the Messaging Server MTA after having accepted them first can be the claimed EHLO/HELO name. You can use the EHLO/HELO name in various mapping tables that the MTA consults during message processing as part of the application information (this applies to FROM_ACCESS, MAIL_ACCESS, and ORIG_MAIL_ACCESS).

The behavior of the MTA can be modified based on the fact that the client used ESMTP, or that the encryption (TLS/SSL) was used. This information is also lost if the AV/AS connects to the MTA and uses ESMTP and no encryption all the time.

Support for SMTP Extensions

The Messaging Server MTA supports a rich set of SMTP extensions. Clients can use these extensions to influence the way that messages are processed by the MTA, or to increase their efficiency. If an AV/AS service supports a lesser number of ESMTP extensions than the Messaging Server MTA, clients lose out on not being able to access the MTA extensions when the AV/AS service is placed in front of the MTA.

The significance of SMTP authentication (which is one of the SMTP extensions) has already been discussed. Other interesting SMTP extensions are:

  • SIZE (RFC 1870). Enables clients to declare the size of a message, and give the server opportunity to reject messages greater that it is willing to accept before the message data is even sent. This avoids wasting network bandwidth.

  • DSN (RFC 3461). The server generates standards-compliant notifications. It can also be used by clients to specify conditions in which notifications should be sent back, and whether notifications should contain entire original messages.

  • STARTTLS (RFC 3207). Enables more secure transfer of messages.

  • ETRN (RFC 1985). Enables clients to request remote queue starting.

  • CHUNKING (RFC 3030). Enables transferring binary messages, which can reduce the bandwidth use.

  • PIPELINING (RFC 2920). Enables more asynchronous processing, hence it can improve network throughput.

  • 8BITMIME (RFC 1652). Enables transferring unencoded 8-bit data, hence can improve throughput.

  • NO-SOLICITING (RFC 3865). Reduces undesirable messages.

Rejecting Mail

According to the SMTP standards, if an MTA is trying to relay mail to another MTA and the destination responds with a rejection, the sender is mandated to perform certain actions, including:

  • If the rejection is permanent, sending a Delivery Status Notification (DSN) message to the original sender of the mail.

  • If the rejection is temporary, storing the message in its queue and retrying it later. After a certain amount of unsuccessful retries, the MTA is supposed to send the DSN message to the original sender.

The potential problem with this is that the DSN messages might not be deliverable themselves, and retrying to send messages, which are not deliverable, wastes MTA resources (computing power, bandwidth, and outgoing threads). Therefore, the MTA should do all it can to ensure that it does not accept messages that will be undeliverable in the first place.

The problem worsens if you have multiple levels of MTAs in your deployment. The MTA that is exposed to the Internet might accept certain messages, which the next relay MTA would reject. Then, if the rejection is permanent, that first MTA is supposed to send the DSN messages to the original senders, which would possibly waste its resources. If the rejection is temporary, the first MTA should retry sending the message for some time, and then send back a DSN.

Thus, the preferred practice is for the first MTA (the one exposed to the Internet) to reject more mail than any subsequent MTA in the environment. That is, the first MTA is “less liberal” in accepting mail. In the case of an architecture that places the AV/AS MTA service exposed to the Internet, and the Messaging Server MTA is the next relay hop, the AV/AS service should be able to reject mail that the Messaging Server MTA would reject.

This type of deployment, in turn, may not be possible. The reason for this is that the Messaging Server MTA would reject mail in many cases that cannot be detected by the AV/AS MTA. First, the LDAP support in Messaging Server MTA is quite complex, and therefore flexible. The Messaging Server MTA does not just check the existence of a user, or a domain in LDAP. Instead, the MTA fetches series of attributes of that user, and makes decisions based on those attributes. Some of these decisions could include, but are not limited to the following situations:

  • Domains and users might have different statuses, some of them resulting with permanent rejections, some with temporary rejections (like inactive, or overquota).

  • The maximum size of messages accepted is computed based on domain, group and user limits. Messages that exceed that dynamically computed limits are rejected.

  • Recipient count limits can be computed dynamically based on domain attributes.

  • Groups might have submission authorization rules imposed. If the sender of a message fails to meet these rules, the message can be rejected.

Also, the Messaging Server MTA can reject messages with temporary errors in case of transient problems that it detects during message processing.

Flexibility of AV/AS Integration

The Messaging Server MTA provides the means to integrate with various AV/AS vendors in a flexible and efficient way through plugins that are dedicated to AV/AS entities. This flexibility and efficiency is present in various areas:

  • It is possible to scan the same messages multiple times, with various AV/AS entities (up to eight different entities from a single MTA). You can choose which AV/AS entity the message should be provided to based on per-channel, per-domain, and per-user basis. Thus, you can provide premium, commercial-based scanning to some users, and other means of scanning to different users.

  • It is also possible to integrate with various AV/AS entities (for instance from different vendors), and act on verdicts provided by these various entities in complex ways. For example, this enables you to rely more on one AV/AS verdict while still making use of another.

  • It is possible to provide per-channel, per-domain, or per-user optins to the scanning engine, if supported by the engine. Thus, the engine understands what kind of checks it is supposed to perform for each user.

  • The AV/AS box can return per-recipient verdicts. These verdicts can be later used to construct per-channel, per-domain, or per-user filters acting on messages based on these verdicts. Thus, specific users can define custom actions that should be performed if a message is a spam suspect. Examples of custom actions include discarding likely spam messages, moving messages to an alternate folder that might be spam but the user would like to read, and tagging a message's subject as potential spam but passing along to the user's inbox.

Using a Single User Repository

The previous section described how the Messaging Server MTA may reject mail based on the content of LDAP attributes, as well as the ability of providing per-user and per-domain optin values to the AV/AS based on the LDAP directory content. The AV/AS entity may support the functionalities previously described (rejecting mail on per-domain and per-user criteria, or ability to provide per-domain and per-user optins), although it may not support using LDAP as the repository for that information. In this case, when you want to take advantage of these functionalities while deploying the AV/AS entity in the front, you must provision per-user and per-domain data into both the configuration repository of that AV/AS service (so that it can use it for providing the above-mentioned functionalities), and into the LDAP (so that the MTA can use it for its operation).

Note - Having to provision two different types of repositories at the same time causes unnecessary complexity to the provisioning tools and creates possibility of data inconsistency between the directory and AV/AS service databases.

Enhancing Performance

Positioning an AV/AS box in front of the MTA can have a negative impact on the overall performance of your messaging system. This impact may show up in directory performance and AV/AS performance.

Impact on the Directory Performance

The following two situations can impact the directory performance:

  • If an AV/AS box is facing the Internet, it must check recipient's existence in LDAP. The way this check is designed can influence the performance of the directory environment:

    1. First, the MTA checks whether the domain is local (and caches that result, both negative and positive result).

    2. If the domain is determined to be local, the mail address is searched for in the directory. This saves the directory service from searching non-local addresses. The domain-related search is usually not costly, as the number of local domains is usually small as compared to the number of users.

  • If the Messaging Server MTA is facing the Internet, it performs the LDAP check to find the domain- and user-related information. This includes, but is not limited to, verification if the mail address exists in the database. As described previously, the MTA fetches a significant amount of information that influences how to process messages. If the AV/AS LDAP integration is limited to verification of the user existence in the database, the MTA check ensures that what the AV/AS requires is known. Therefore, if the MTA is performing its LDAP check first, the AV/AS-related LDAP check is not needed anymore. If the Messaging Server MTA passes the message to the AV/AS entity, that AV/AS entity may be certain the user exists and can refrain from performing the LDAP check. This saves directory server cycles. If, however, the AV/AS service is facing the Internet, and performs its LDAP check to determine if the user is local, the Messaging Server MTA that is deployed further in the processing chain has to fetch the domain-related, group-related, and user-related information from the directory anyway. So it must perform its checks on the directory regardless if it is facing the Internet, or not. Thus, the overall number of LDAP searches is always greater if the AV/AS entity is facing the Internet, compared to the other way round.

Impact on the AV/AS Performance

If you deploy the AV/AS entity so that it faces the Internet, it has to check all messages that are determined to be accepted by this entity. If the Messaging Server MTA, which is deployed further in the relay chain rejects a message that has already been checked, because of the group or user-related parameters, the cycles required for the check have been lost. The AV/AS service is utilized better if the MTA rejects such messages before passing them to the AV/AS service. Thus, the architecture where the Messaging Server MTA faces the Internet and rejects messages before passing them to the AV/AS entity may be more efficient from this perspective.


I'm very pleased to see this document. It has a clear structure and is easy to read.

If time permits I'll post a more descriptive review. One of the first things that I noticed is that SPF is mentioned, but DKIM is not. From a marketing perspective it is a good idea to mention SPF, but from a technical point of view it's more important to mention DKIM, and the way it can be used in such a configuration.


Posted by Rolf E. Sonneveld on December 06, 2007 at 07:58 AM MST #

Post a Comment:
  • HTML Syntax: NOT allowed

Reporting about Unified Communications Suite Documentation, including news, Comms 101, documentation updates, and tips and tricks.


« August 2016