MailerQ 5.0 has been released

We are happy and excited to announce MailerQ version 5.0. This new major release offers many, many changes and improvements over the previous 4.4.* versions. Our colleagues from SMTPeter and Copernica have been using this new version for the last couple of months and are very happy with the improved stability, speed and the better controls of the user interface, which makes us confident to bring this new version out to the general public as well.

All users are encouraged to upgrade to this new version. However, since this is a dot-zero release of a new major version, and because it is a good practice in general, we do advise to upgrade with care.

New management console

The most striking change in MailerQ 5.0 is the massively improved management console: MailerQ 5.0 comes with a very rich user interface, with a dashboard full of real time charts and widgets to monitor what is going on inside the MTA. You can keep an live look on incoming and outgoing mails, and adjust the delivery throttles on the fly.

We have changed the architecture of the management console from using pull technology to push technology. This means that the front end no longer refreshes its data every couple of seconds, but that a constant data stream is sent from the MTA to the browser while the management console is open. As a result, the management console now shows the data in real time and it can also show much more than before. And despite the fact that we now send much more data to the browser, the switch to push technology even reduced the load on the core MTA process because the process is no longer interrupted every couple of seconds. In fact, we now have so much data to show that it was quite a challenge for our front-end developers to handle the incoming stream of data from the MTA in such a matter that it did not take up all the resources of the browser.

The best way to find out what exactly has been changed (a lot!) is by experiencing this yourself. The new management console is mostly self-explanatory (well, that’s what we tried to achieve), so you can simply download the latest MailerQ version and try it out.

Improved support for message tags

In the previous MailerQ version we already had support for message tags, but in this new version of MailerQ we’ve made these tags much more usable. Tags allow you to associate messages that flow through MailerQ with customer names, campaign identifiers or anything else to, well, tag messages. In MailerQ 5.0 these tags are much better integrated in the core MTA processes, so that you can now use tag filters on all sort of levels to monitor or influence the behavior of MailerQ.

Whether you’re looking at statistics, pausing or rewriting mails or monitoring log files or SMTP traffic: you can always use tags to limit the data that you’re looking at or the consequences of your actions.

New way to fetch a license

You need a license file to run MailerQ 5.0. But unlike previous versions, this license file can not be downloaded from the website, but has now to be fetched by starting MailerQ with the “--fetch-license” command line argument. When you start MailerQ, the program will automatically explain how you can get access to your license file: it is much simpler than before.

Changes to throttle settings

A couple of new options have been added to the delivery throttle settings. These extra properties are accessible through the management console if you edit a capacity.

We now allow users to set the minimum number of messages that are sent over SMTP connections. This setting makes sure that no new SMTP connections are opened if there still is an existing connection that does not yet have reached this minimum. This new setting prevents that MailerQ opens a lot of connections over which only a small number or even a single message gets delivered.

When MailerQ discovers that an IP address is offline, it used to mark that IP as offline for a fixed (hardcoded) period of time. This setting is now configurable via the throttle settings. You can also configure for how long received SSL certificates get cached. This cache is used to decide whether a mail can indeed be sent over the connection. The cache prevents that MailerQ has to set up a SMTP connection to find out if the SSL certificate is valid if it already found out that the certificate was invalid.

MailerQ 5.0 also makes it possible to set a timeout for the STARTTLS handshake. If the remote server fails to complete the STARTTLS handshake within this period of time, the connection is now closed. This setting prevents that unresponsive servers cause MailerQ to run out of resources.

The SMTP protocol officially only allows servers to send back a 220 or 554 response on connection setup. However, in practice there are also servers that send back 4** replies. With MailerQ 5.0 you can now better configure how this should be treated: should MailerQ proceed to the secondary MX (this was the old behavior), or respect the error code as if it was a valid greylisting error and retry the mail a couple of minutes later.

When an 4** error is encountered later in the SMTP protocol (after the welcome message), a message is considered to be “greylisted” and will be retried in a couple of minutes time. The SMTP protocol however does not say whether that retry should be sent from and to the same IP address, or that it is valid to send the same to a different IP instead. By default, MailerQ sends from and to the same IP, but with version 5.0 you can allow MailerQ to be more flexible and use different IP's for the retried attempt.

New rewrite module replaces delegation

We’ve added a rewrite feature to MailerQ 5.0. You can now install rewrite rules to modify the messages right after they’ve been read from RabbitMQ and before they are processed and sent out. With this rewrite feature you can for example change the IP address from which a message will be sent, or change the smarthost to which it is sent.

This feature replaces the old delegation system that we had in MailerQ 4.4 and before. With delegation it was possible to send out messages from a different IP than was set in the JSON. With the rewrite module you can achieve exactly the same, and it is more powerful at the same time because it allows other rewrites too.

Instance-only pauses and errors

If you run a cluster of MailerQ instances that all connect to the same database (which is nice, because the delivery throttles are then shared between all instances), you used to have the side effect that the pause and error settings were also shared on the entire cluster: if you pressed the pause button on one MailerQ instance, all other instances were paused too. With the introduction of MailerQ 5.0, this cluster-wide pausing is now optional: if you press the pause button you can choose whether to apply this to the entire cluster or to just a single MailerQ instance.

Changes in result states

In the JSON result objects that are published to RabbitMQ we now also identify the state “process”, which is used to report errors that MailerQ ran into when processing the message, but before an actual SMTP connection is set up. Such errors were previously reported in states like “amqp”, but that did not really make sense because the errors were not amqp related (amqp is the message queue protocol that is used to communicate with RabbitMQ). The “amqp” state is no longer reported.

Watch out: you may have to update your scripts that read out the result queues and process these result objects, because this new state=process setting will not also appear in the result queue.

Database related changes

We have made some changes to the config file. The old setting “database” is for example different now: it now is named “database-address”. This is just a rename to improve consistency: all the database related config file variables now start with a “database-” prefix. The old variable, “database” is still supported for backwards compatibility and is checked if you have not assigned a value to “database-address”.

We have a new variable “database-threads” which can be set to the number of threads that are started to load all data from the database into main memory. This setting is especially useful if you’ve stored a lot of DKIM keys in your MailerQ database, because every DKIM key is normally verified against DNS on database (re)load. We do this check to prevent that a mail gets signed with a private key that does not match the public key found in DNS. This DNS check is a blocking operation (meaning: it blocks the thread that reads out the database, not he core MTA process) and it gets especially slow if there are many DKIM records in your database. By setting the “database-threads” setting to a higher value (the default is 1, so you can for example set it to 5 or 10), you can speed up loading the data into memory because a single DNS lookup will no longer block the entire reload thread.

There is an extra change to the stored DKIM keys: you can now also store the algorithm (sha1 or sha256) to be used for the signature (tip: always use sha256, sha1 is not safe enough!). In the previous version the algorithm was fetched from DNS, but it can now be set manually.

The config file variable “database-validate” has been introduced to check whether the data in the database is in a consistent state. This setting is only used on startup. When MailerQ is started, it first checks if all the tables in the database are in the right format (do all columns exist, are the indexes created, et cetera). If you also set the “database-validate” setting to “yes” or “true”, MailerQ will also check if the data inside the tables in a consistent state. This setting has been introduced because up to now, MailerQ was not consistent in storing “unlimited” and “fallback to default” values in the database: sometimes the value 0 was used for this, while at other times -1 was used.

We further more discovered an issue with UTF8 encoding in MySQL and Postgresql, which has been fixed. We also fixed some issues with Sqlite that sometimes caused concurrent queries to go wrong, and we now truncate the Sqlite journal file, so that MailerQ can switch its user ID and still be able to connect to the database.

Storage options

We’ve received a feature request from users who sometimes experience small hickups or delays in their message store environment. We have therefore created two new config file variables that can be used to set a max time to wait for a storage fetch operation: “storage-timeout” and “storage-reschedule”. If you use an external storage environment for the message content (like MongoDB), you can use the “storage-timeout” setting to set the max number of seconds that MailerQ is allowed to wait for a fetch operation. If MailerQ does not succeed in loading the message within this time, the message will be retried later on (hoping that by then the load on the storage environment has been normalized). The “storage-reschedule” contains the number of seconds to wait before the retry. This solves the issue that a temporary issue with the storage environment results in mails not being delivered.

If you’re using a mongodb:// backend, we now support a “maxQuerySize” parameter to limit the number of lookups that are combined into a single fetch operation that is sent to MongoDB. You can experiment with this setting if you notice that it sometimes takes too much time to load data from MongoDB. Normally, MailerQ tries to combine multiple fetch operations into a single MongoDB query, to bring down the number of queries that are sent to MongoDB. However, these combined queries do take longer to process.

If you’re using a directory:// backend for the message store, you may have noticed that MailerQ usde to create a directory structure of four directories deep. We received some complaints from users about this: too many directories were created and it was, according to them, a waste of inodes. We see their point and have therefore changed the directory depth to only 2 directories, with a backwards compatibility check: if a resource cannot be found at depth two, MailerQ also checks the four-deep directory. With an optional parameter you can set the depth manually and use the old depth: “directory:///path/to/dir?depth=4”.

DNS lookups

MailerQ runs all its DNS operations in seperate threads to prevent that one slow DNS operation blocks deliveries to other domains. The management console now has a special widget that allows you to monitor these threads. If you see that there are regulary more DNS operations in progress than the number of DNS threads that you have configured in the config file, it is recommended to increase your threads by changing the “dns-threads” config file variable. This very often leads to immediate performance improvements.

On top of this, we have added a special config file setting “dns-getaddrinfo” to enable or disable the use of the “getaddrinfo” library function. If you want to reduce the number of DNS lookups, you can set this config setting to “no”. MailerQ will then start communicating directly with the DNS server, and bypass the functions from the system library. The downside is that it will also no longer check /etc/hosts files for IP addresses.

Internal web server

We’ve completely rewritten the internal web server of MailerQ. We sometimes received reports about crashes caused by the old web server that we could not reproduce in our own development environment, and that were probably caused by race conditions between threaded file operation. The new web server has a much simpler design and is less vulnerable for these sort of issues. At the same time this allowed us to make some improvements.

The HTTP responses that are sent by MailerQ do normally not contain a “cache-control” header, so that browsers can decide all by themselves if and for how long they cache the resources. This is not so much of a problem of course, since the source code of the management console does not change that often (only if you install a new version of MailerQ), but it was inconvenient for our front end development team that was working on the application. For them we have added a “www-cache-control” config file setting. If you set it to “no-cache, no-store, must-revalidate”, you can be pretty sure that your browser will not cache the HTML and javascript files of the management console.

A more useful new feature is that all the incoming requests to the webserver can now be logged. We have new config file variables “www-log-directory”, “www-log-prefix”, “www-log-history”, “www-log-maxsize”, “www-log-maxage” and “www-log-compression” to facilitate this. The variables can be used in exactly the same way as the other logging variables (for example the variables to log the deliveries and errors).

If you’re using a cluster of MailerQ instances, the instances expose their URL to each other so that each management consoles can link to the other instances. To be able to expose the URL, each instance must have a way to find out its URL. MailerQ normally does this by simply constructing a string with the hostname of the local server and the port number. If you access the web interface via a different URL however (for example via a proxy), you can now use the “www-url” config file variable instead.

Download cache

The config file settings “cache-size” and “cache-dimensions” have been removed. They are no longer supported because MailerQ no longer maintains an in-memory cache of downloaded resources (for example images when a response email message is being generated). Instead, MailerQ now has the following variables: “download-cache”, “download-ttl”, “download-blacklist” and “download-whitelist”.

The “download-cache” setting holds a storage url, with a similar syntax as the storage address for the message store (so it can be set to a string like “mongodb://...” or “directory:///...”). The “download-ttl” setting should be set to the max number of seconds a resource gets cached. The “download-blacklist” and “download-whitelist” variables can be used to prevent that resources are downloaded from private addresses (like from the local network).

Other JSON library

Plugin developers watch out: we switched from libjson-c to libfastjson for the JSON handling in MailerQ. We ran into a couple of issues with libjson that were not fixed, that we decided to switch to libfastjson instead. Libfastjson is a fork of libjson-c, so the API is almost the same. However: you may have to update your existing plugin.

Many, many small changes

Besides all the above changes, we have also made many, many small changes and bug fixes all around the code. That list of fixes is long, and often too technical to mention here. For now, we recommend everyone to upgrade to this 5.0 release and try out the fixes and improvements. Please contact us if you run into any issues so that we assist you in the upgrade process.