As I previously wrote about here
, I've been collaborating with one of our Messaging Server experts on a white paper that we hope clarifies Sun's position on how to best deploy AV/AS services with the Messaging Server MTA. I'm posting the draft paper below in hopes of soliticing comments and feedback from the Comms community.
DRAFT: Deploying Sun Java System MTA Preferred Practices
Wojtek Chemijewski and Joe Sciallo, December 2007
In general, a typical Sun Java System Messaging Server deployment consists of front-end
and back-end services. The front-end services enable users to access the system,
while the back-end services store and provide access to data. In the past,
we've heard disputes about how to deploy the MTA on the front end:
either as an entrance gateway to the system, or “behind” an anti-spam/anti-virus service
that also provides an MTA functionality. This article describes the recommended way to
deploy your MTA with respect to anti-spam/anti-virus services.
This article contains the following sections:
Sun Java System Messaging Server provides a highly reliable, high performance, full featured
SMTP implementation in the form of its Mail Transfer Agent (MTA). This MTA
enables you to integrate anti-virus/anti-spam (AV/AS) solutions through the use of plugins. These
plugins provide message content to the AV/AS engine for its evaluation and scanning,
while allowing the MTA to retain full responsibility for the message at the
SMTP level. The MTA does not provide the AV/AS engine itself: Messaging Server
relies on third-party vendor experts to provide these solutions. Such a best-of-breed architecture,
built in to Messaging Server, provides you with the greatest flexibility of integration.
Thus, you use the Messaging Server MTA for the SMTP message processing and
an AV/AS solution of your choice for virus and spam processing.
Messaging Server Architecture Overview
A two-tiered messaging architecture provides the optimum design for scalability and reliability. Instead
of having a single host run all the components of a messaging system,
a two-tiered architecture separates the components onto different machines. These separate components perform
specific specialized functions. As the load for a particular functional component increases—for example, more
Message Storage is required, or more outbound relaying is needed—you can add more
servers to handle the larger loads.
The Messaging Server two-tiered architecture consists of an access layer and a data
The access layer is the portion of the architecture that handles delivery, message
access, user login, and authentication. (The access layer is also referred to as
the front-end layer or tier.) For a Messaging Server deployment, the access layer
comprises the MTA, Messaging Multiplexor (MMP), and Webmail server. AV/AS services are also
part of the access layer.
The data layer is the portion of the architecture that holds all
the data. (The data layer is also referred to as the back-end layer
or tier.) This includes the LDAP master servers and Messaging Server machines that
are configured to store user messages.
For more information on Messaging Server architecture, see http://docs.sun.com/app/docs/doc/819-4439/6n6jehs1d?a=view.
More About the MTA and AV/AS Services
The MTA is one of the Messaging Server access layer services. The
MTA receives, routes, transports, and delivers mail messages using the SMTP protocol. An MTA
delivers messages to a local mailbox or to another MTA. The anti-virus/anti-spam check
is performed on these mail messages along their entry to the system. The
preferred way of integrating AV/AS services with the MTA is by accepting messages
from the users and the Internet, providing them to the AV/AS software for
a check, and then relaying these messages further (to other users or
to the Internet) as applicable.
AV/AS software vendors usually provide some MTA functionality in their products. Furthermore, AV/AS
software vendors tend to recommend that you deploy their software as the entry
point to the system, and that their software rely messages to the MTA
after they have been checked for spam or viruses. As this paper shows,
deploying your AV/AS software in such a manner—in front of the MTA—is a
suboptimal method. Indeed, if you opt for such a deployment, there are certain
MTA features that you cannot even access. Furthermore, there are MTA features that,
if used in such an architecture, cause performance degradation.
Deploying AV/AS Services with Messaging Server
To begin with, let's understand the two fundamental ways of developing a Messaging
Server architecture for AV/AS such that the MTA is the component to first
MTA Plugin (Integrated Approach). You deploy the AV/AS services through an integrated MTA plugin, which is dedicated for the specific AV/AS vendor. Such plugins usually work through a proprietary protocol for providing messages to the AV/AS scanner, and that scanner provides its verdict back to the MTA. The MTA then uses this verdict for further processing of the message.
SMTP Relaying. If there is no AV/AS plugin available, you can configure the MTA to accept the message first, and then relay it to the AV/AS service using SMTP. The AV/AS service should then relay the message back, adding information with the result of the scanning into the message headers. The MTA can then access that information and use it as the verdict for further process of the message.
Whenever possible, use the integrated approach when developing your AV/AS deployment for Messaging
Server. The integrated approach provides more flexibility in the way messages are chosen
for scanning. In addition, with the integrated approach, the MTA retains all responsibility
for the message all the time it is processed. The integrated approach also
provides only the content of messages to the AV/AS service for evaluation and
verdict. Thus, no SMTP relaying is involved in providing the message for AV/AS
processing, and so the AV/AS scanner does not need to manage queues nor
generate non-delivery reports. Finally, with the integrated approach, the set of ESMTP extensions supported
by the AV/AS' MTA does not influence the extensions that the
clients can use. When an MTA relays messages to another MTA, it first
queries for the protocol extensions that other MTA supports. In the case of
the Messaging Server MTA, when a client first submits a message to the
Messaging Server MTA, it queries for the set of supported extensions. Next, the
Messaging Server MTA uses the DSN extension to request a successful delivery report.
Finally, the Messaging Server MTA relays that message to the AV/AS service and
discovers that the AV/AS MTA does not support the DSN extension. Thus, a
request for a successful delivery report cannot be relayed any further. In case
of the MTA plugin, the responsibility for message processing remains with the Messaging Server
MTA, so no such effect happens.However, the integrated approach is
possible only with the AV/AS entities, for which respective plugins exist.
MTA Functionality Lost Due to Lack of Source IP Address Information
When you deploy the AV/AS services “in front” of the MTA, the MTA
does not have access to the IP address of the sending client.
Instead, it has access only to the IP address of the AV/AS box.
The result is that the MTA cannot make make any decisions (that is,
possibly reject mail) based on the client IP address. The following situations describe
how knowing the source IP address of the sending client is useful:
Connection Throttling, Possibly Across Multiple MTAs. The MTA can throttle connections based on the source IP address. Throttling means limiting the number of connections that a particular IP address can open to the MTA within a certain time period. You can configure the MTA to allow only a certain number of connections to be open within one minute, and reject all connections exceeding that limit. You can also penalize an IP address that opens more connections than allowed. The MTA will continue to reject such an IP address for some extended time, even if the rate drops below the limit. In Messaging Server 6.3, you can also deploy the Metermaid server, which hosts a throttling database, that you can use commonly across multiple MTA servers. With Metermaid, you can control how many connections a certain IP address opens, even if the IP address tries to connect to multiple MTAs (which is very possible, due to the the front-end tier's usual horizontal scalability).
Sender Policy Framework (SPF). Spam producers and email scammers often forge email by using false domain names and email addresses or by using legitimate domain names and email addresses in order to fool users into thinking that a message is from someone or some company they know. For example, a spammer could send email from an address such as
firstname.lastname@example.org and the user could be fooled into thinking the mail was actually from this address. Forging email may fool users into opening the unsolicited message, or worse, provide information to a false authority. Also, spammers prefer to send their email from legitimate domains that are not on an RBL list.
Sender Policy Framework (SPF) is a technology that can detect and reject forged email during the SMTP dialogue. Specifically, SPF is a protocol that allows a domain to explicitly authorize the hosts that may use its domain name. In addition, a receiving host may be configured to check this authorization. SPF can thus significantly reduce the instances of forged email. If the IP address is not known, the SPF check cannot be performed. For more information on SPF, see http://docs.sun.com/app/docs/doc/819-4428/gdpno
The Messaging Server MTA supports SPF in a flexible way, enabling it to influence the action which is performed upon the SPF result or to incorporate the SPF result into further (for instance, per-user) processing. Messaging Server provides this capability so that you do not have to depend on the AV/AS, which may or may not be as flexible in handling complex scenarios.
Other DNS-based Checks. The Mail Abuse Protection System's Realtime Blackhole List (MAPS RBL) is a dynamically updated list of known unsolicited bulk email (UBE) sources identified by source IP address. The Messaging Server SMTP server supports use of the RBL and can reject mail coming from sources identified by the RBL as originators of UBE or spam. You an use the result of such a check to reject mail. You can also use this check to influence how the MTA further processes the message. For example, you can increase the likelihood of the message to be spam through a positive RBL check, and you can use that value in a user-based SIEVE filter.
IP-based Channel Switch. The MTA can switch the source channel based on the client IP address. Typically, all mail arriving on port 25 of the server is enqueued to the
tcp_local channel, and characteristics of that channel are applied to the connection and to the mail submitted on that connection. However, it is possible to switch the connection to another channel based on the client IP address and apply different characteristics if the client's IP address belongs to certain pool. Examples of these characteristics include:
Whether encryption is required
Whether sender authentication is required
Whether the source channel should be further switched based on the fact that connection is encrypted or authenticated
Whether ETRN is allowed, the maximum message size, the recipient count limits, and so on
You can also deploy source-channel-specific rewrite rules so that addresses in messages being switched to a specific channel are rewritten differently than the rest. You can use the source channel in mapping tables, which enable you to deploy regular-expression-based ways of processing messages. The most common example of this functionality is the default behavior of mail coming from clients whose IP address belongs to the value specified in the
INTERNAL_IP mapping table: the mail is switched to the
tcp_intranet and allowed to go out to the Internet.
Using IP Address in Mappings. The client IP address can be directly used in mapping tables, which the MTA accesses at various stages of mail processing. You can use the IP address in
PORT_ACCESS (to make decisions based on IP addresses),
FROM_ACCESS (to make decisions based on the IP addresses, application information, source channel, envelope From: address, and authentication information),
ORIG_MAIL_ACCESS (to make decisions based on IP addresses, application information, source and destination channels, and envelope From: and To: addresses).
Logging. You can configure the MTA to log the sending client IP address. Not being able to capture and log this IP address because of locating the AV/AS service in front of the MTA complicates analysis of the MTA behavior and troubleshooting problems.
Functionality Lost Due to Lack of Authentication Information
It may be possible to configure an AV/AS entity to provide means
for SMTP authentication (using LDAP bind for instance), and to relay mail to
the MTA without authenticating. In such cases, the MTA does not have access
to the authentication information. Therefore, you cannot use MTA functionality that requires users to
authenticate or to modify the MTA's behavior based on the authentication information. Examples
of lost functionality in this case include:
Authentication Enforcement. You can enforce authentication to send mail out to the Internet (this is the typical setup). You can also enforce authentication on certain source channels. Knowing the sending client's IP address enables switching the source channel based on the client IP, and knowing that the users authenticated enables you to enforce authentication on messages from certain IP address ranges.
Per-user Channel Switch. If you provision a user with the
mailSMTPSubmitChannel LDAP attribute, the source channel can be switched to a channel specific for that user. For the MTA to do so, the user must authenticate. Once the switch is done, this particular channel's characteristics will be applied for all mail originating from that user.
List Submission Authorization. You can provision a mailing list with attributes that specify who can send mail to that list, for example,
mgrpDisallowedDomain. The authentication information can be used in these checks.
Message Headers. The fact that the user authenticated is put into the Received: header of a message. Also, you can configure the MTA to put the Sender: header line, containing information who authenticated in order to send mail, into the message. This behavior can be channel-specific, so the ability to switch the channel based on an IP address of the client may be of use here, as well.
Logging Who Authenticates. The authentication information (information who authenticated) can be recorded in log files.
Other Functionality Lost
Other information lost when the AV/AS relays messages to the Messaging Server MTA
after having accepted them first can be the claimed
EHLO/HELO name. You
can use the
EHLO/HELO name in various mapping tables that the MTA consults
during message processing as part of the application information (this applies to
The behavior of the MTA can be modified based on the fact
that the client used ESMTP, or that the encryption (TLS/SSL) was used. This
information is also lost if the AV/AS connects to the MTA and uses
ESMTP and no encryption all the time.
Support for SMTP Extensions
The Messaging Server MTA supports a rich set of SMTP extensions. Clients can
use these extensions to influence the way that messages are processed by the
MTA, or to increase their efficiency. If an AV/AS service supports a lesser
number of ESMTP extensions than the Messaging Server MTA, clients lose out on
not being able to access the MTA extensions when the AV/AS service is
placed in front of the MTA.
The significance of SMTP authentication (which is one of the SMTP extensions) has
already been discussed. Other interesting SMTP extensions are:
SIZE (RFC 1870). Enables clients to declare the size of a message, and give the server opportunity to reject messages greater that it is willing to accept before the message data is even sent. This avoids wasting network bandwidth.
DSN (RFC 3461). The server generates standards-compliant notifications. It can also be used by clients to specify conditions in which notifications should be sent back, and whether notifications should contain entire original messages.
STARTTLS (RFC 3207). Enables more secure transfer of messages.
ETRN (RFC 1985). Enables clients to request remote queue starting.
CHUNKING (RFC 3030). Enables transferring binary messages, which can reduce the bandwidth use.
PIPELINING (RFC 2920). Enables more asynchronous processing, hence it can improve network throughput.
8BITMIME (RFC 1652). Enables transferring unencoded 8-bit data, hence can improve throughput.
NO-SOLICITING (RFC 3865). Reduces undesirable messages.
According to the SMTP standards, if an MTA is trying to relay
mail to another MTA and the destination responds with a rejection, the sender
is mandated to perform certain actions, including:
If the rejection is permanent, sending a Delivery Status Notification (DSN) message to the original sender of the mail.
If the rejection is temporary, storing the message in its queue and retrying it later. After a certain amount of unsuccessful retries, the MTA is supposed to send the DSN message to the original sender.
The potential problem with this is that the DSN messages might not
be deliverable themselves, and retrying to send messages, which are not deliverable, wastes MTA
resources (computing power, bandwidth, and outgoing threads). Therefore, the MTA should do all
it can to ensure that it does not accept messages that will be
undeliverable in the first place.
The problem worsens if you have multiple levels of MTAs in your
deployment. The MTA that is exposed to the Internet might accept certain messages,
which the next relay MTA would reject. Then, if the rejection is permanent,
that first MTA is supposed to send the DSN messages to the original
senders, which would possibly waste its resources. If the rejection is temporary, the
first MTA should retry sending the message for some time, and then send
back a DSN.
Thus, the preferred practice is for the first MTA (the one exposed
to the Internet) to reject more mail than any subsequent MTA in the
environment. That is, the first MTA is “less liberal” in accepting mail. In the
case of an architecture that places the AV/AS MTA service exposed to
the Internet, and the Messaging Server MTA is the next relay hop, the
AV/AS service should be able to reject mail that the Messaging Server MTA
This type of deployment, in turn, may not be possible. The reason
for this is that the Messaging Server MTA would reject mail in many
cases that cannot be detected by the AV/AS MTA. First, the LDAP support
in Messaging Server MTA is quite complex, and therefore flexible. The Messaging Server MTA
does not just check the existence of a user, or a domain
in LDAP. Instead, the MTA fetches series of attributes of that user, and
makes decisions based on those attributes. Some of these decisions could include, but
are not limited to the following situations:
Domains and users might have different statuses, some of them resulting with permanent rejections, some with temporary rejections (like inactive, or overquota).
The maximum size of messages accepted is computed based on domain, group and user limits. Messages that exceed that dynamically computed limits are rejected.
Recipient count limits can be computed dynamically based on domain attributes.
Groups might have submission authorization rules imposed. If the sender of a message fails to meet these rules, the message can be rejected.
Also, the Messaging Server MTA can reject messages with temporary errors in case
of transient problems that it detects during message processing.
Flexibility of AV/AS Integration
The Messaging Server MTA provides the means to integrate with various AV/AS vendors
in a flexible and efficient way through plugins that are dedicated to AV/AS
entities. This flexibility and efficiency is present in various areas:
It is possible to scan the same messages multiple times, with various AV/AS entities (up to eight different entities from a single MTA). You can choose which AV/AS entity the message should be provided to based on per-channel, per-domain, and per-user basis. Thus, you can provide premium, commercial-based scanning to some users, and other means of scanning to different users.
It is also possible to integrate with various AV/AS entities (for instance from different vendors), and act on verdicts provided by these various entities in complex ways. For example, this enables you to rely more on one AV/AS verdict while still making use of another.
It is possible to provide per-channel, per-domain, or per-user optins to the scanning engine, if supported by the engine. Thus, the engine understands what kind of checks it is supposed to perform for each user.
The AV/AS box can return per-recipient verdicts. These verdicts can be later used to construct per-channel, per-domain, or per-user filters acting on messages based on these verdicts. Thus, specific users can define custom actions that should be performed if a message is a spam suspect. Examples of custom actions include discarding likely spam messages, moving messages to an alternate folder that might be spam but the user would like to read, and tagging a message's subject as potential spam but passing along to the user's inbox.
Using a Single User Repository
The previous section described how the Messaging Server MTA may reject mail based
on the content of LDAP attributes, as well as the ability of
providing per-user and per-domain optin values to the AV/AS based on the LDAP
directory content. The AV/AS entity may support the functionalities previously described (rejecting mail on
per-domain and per-user criteria, or ability to provide per-domain and per-user optins), although
it may not support using LDAP as the repository for that information. In
this case, when you want to take advantage of these functionalities while deploying
the AV/AS entity in the front, you must provision per-user and per-domain data
into both the configuration repository of that AV/AS service (so that it can
use it for providing the above-mentioned functionalities), and into the LDAP (so that
the MTA can use it for its operation).
Note - Having to provision two different types of repositories at the same time causes
unnecessary complexity to the provisioning tools and creates possibility of data inconsistency between
the directory and AV/AS service databases.
Positioning an AV/AS box in front of the MTA can have a
negative impact on the overall performance of your messaging system. This impact may
show up in directory performance and AV/AS performance.
Impact on the Directory Performance
The following two situations can impact the directory performance:
If an AV/AS box is facing the Internet, it must check recipient's existence in LDAP. The way this check is designed can influence the performance of the directory environment:
First, the MTA checks whether the domain is local (and caches that result, both negative and positive result).
If the domain is determined to be local, the mail address is searched for in the directory. This saves the directory service from searching non-local addresses. The domain-related search is usually not costly, as the number of local domains is usually small as compared to the number of users.
If the Messaging Server MTA is facing the Internet, it performs the LDAP check to find the domain- and user-related information. This includes, but is not limited to, verification if the mail address exists in the database. As described previously, the MTA fetches a significant amount of information that influences how to process messages. If the AV/AS LDAP integration is limited to verification of the user existence in the database, the MTA check ensures that what the AV/AS requires is known. Therefore, if the MTA is performing its LDAP check first, the AV/AS-related LDAP check is not needed anymore. If the Messaging Server MTA passes the message to the AV/AS entity, that AV/AS entity may be certain the user exists and can refrain from performing the LDAP check. This saves directory server cycles. If, however, the AV/AS service is facing the Internet, and performs its LDAP check to determine if the user is local, the Messaging Server MTA that is deployed further in the processing chain has to fetch the domain-related, group-related, and user-related information from the directory anyway. So it must perform its checks on the directory regardless if it is facing the Internet, or not. Thus, the overall number of LDAP searches is always greater if the AV/AS entity is facing the Internet, compared to the other way round.
Impact on the AV/AS Performance
If you deploy the AV/AS entity so that it faces the Internet,
it has to check all messages that are determined to be accepted by
this entity. If the Messaging Server MTA, which is deployed further in the
relay chain rejects a message that has already been checked, because of the
group or user-related parameters, the cycles required for the check have been lost.
The AV/AS service is utilized better if the MTA rejects such messages before
passing them to the AV/AS service. Thus, the architecture where the Messaging Server
MTA faces the Internet and rejects messages before passing them to the AV/AS
entity may be more efficient from this perspective.