To upgrade MailerQ from version < 5.7 to 5.7+
Debian/Ubuntu:
sudo apt update
sudo apt install -y mailerq
sudo mailerq --repair-databaseCentOS/RHEL:
yum update
mailerq --repair-databaseTo downgrade MailerQ from version 5.7+ to 5.6
Debian/Ubuntu:
sudo apt-get install -y --allow-downgrades mailerq-5.6
sudo mailerq --repair-databaseCentOS/RHEL:
yum remove mailerq-5.7
yum install mailerq-5.6
mailerq --repair-databaseThe easiest way to try out MailerQ is using Docker. The MailerQ docker image is completely self-contained, and allows you to run MailerQ without the hassle of setting up its dependencies in under a minute.
First, to install the 5.7 version of the MailerQ image, run
docker pull mailerq/mailerq:5.7Now, create a trial if you have not already done so. Then, the simplest method is to copy your license key from here and supply it to Docker inside an environmental variable. To do this, you can run
docker run -e LICENSE_KEY=<your_license_key> -it mailerq/mailerq:5.7with your license key filled in. This way, the image will automatically download a short-lived license on each startup, ideal for testing.
Alternatively, you can manually download the license.txt file, and bind the image license
file to the host file. To accomplish this, you can run
docker run -v `pwd`/license.txt:/etc/mailerq/license.txt -it mailerq/mailerq:5.7with the license.txt in the current working directory.
That is it! Now you can access the MailerQ management console here and the RabbitMQ management console here.
By default, the image runs on its own network managed by Docker, which means that the
default username and password guest in RabbitMQ do not work. Therefore, on startup,
an extra account mailerq is created with password set to mailerq.
MailerQ runs on Linux. We distribute the software in binary form for Debian/Ubuntu based systems and for Red Hat based systems. Please drop us a message if you need a version for a different type of system.
Most libraries that are needed by MailerQ are statically linked. This means that most of the required libraries are embedded into the binary code, so MailerQ runs straight out of the box with no need for you to fix dependencies and install other software. However, when MailerQ starts, it does do a scan of your system to detect which libraries are available. If MailerQ happens to find one or more of the following libraries, it does load them to use specific features from these libs:
| Tables | Description |
|---|---|
| libopenssl | Used for TLS encryption, license checking and base64 encoding |
| libz | Used for compression algorithms |
| libmagic | Used to detect the mime-type of files |
| libuuid1 | Used for generating globally unique identifiers |
| libidn | Used for parsing international domain names |
| libxml2 | Used for parsing and modifying XML/HTML code |
| libimagemagick | Used to find out the dimensions of images |
| libmysqlclient | Used to connect to a mysql/mariadb database |
| libmariadbclient | Used to connect to a mysql/mariadb database |
| libpq | Used to to connect to a Postgresql database |
| libsqlite3 | Used to process sqlite3 database files |
| libmongo-c-driver | Used to connect to a MongoDB NoSQL server |
| libcouchbase | Used to connect to a Couchbase server |
From the above list, the openssl, libz and libidn libraries are required. All other libraries are optional and MailerQ can run without them. If a library is missing, MailerQ will either fall back on its own implementation, or will run without the features from the specific library.
Most of the MailerQ runtime settings are stored in a relational database. This means that you must have at least one of the mysql, mariadb, postgresql or sqlite3 libraries installed on your system.
Sqlite3 is a local solution. Mysql or Mariadb can run on a different servers, but then you need libmysqlclient on the mailerq box. Postgresql 9.5+ which can be on a different server too, but then you need libpq >= 9.5 to use it.
If you use Swift Mailer as your main mail engine in your project you can download MailerQ Transport Plugin to extend functionality of the Swift Mailer. Below you can find an example showing how to combine those two projects together.
example.php<?php
// load required dependencies
require_once 'swiftmailer/lib/swift_required.php';
require_once 'MailerQPlugin/MailerQConnection.php';
require_once 'MailerQPlugin/MailerQTransport.php';
// an array with additional options
$options = array(
'ips' => ['1.2.3.4','1.2.3.5'],
'maxattempts' => 5
);
// set up a connection to AMQP server
$connection = new MailerQConnection('localhost', 'outbox', 'guest', 'guest', '/');
// create the message using Swift_Message
$message = Swift_Message::newInstance();
$message->setSubject('This email was sent using Swift MailerQ Transport Plugin');
$message->setBody('Example message');
$message->setFrom('example@domain.com', 'Example');
// deliver the message to
$message->setTo('example@domain.com');
// create a new MailerQTransport instance
$transport = new MailerQTransport($connection, $options);
// create a new SwiftMailer instance
$mailer = Swift_Mailer::newInstance($transport);
// send the actual message using the MailerQTransporter
$mailer->send($message);
?>
MailerQ can monitor a directory on the file system for new files: the spool directory. You can simply inject email into MailerQ by copying files to this directory.
The files that you drop in the spool directory must be correctly formatted MIME messages. Invalid files will not be picked up. MailerQ opens the file, and filters out all the recipients from the "to", "cc" and "bcc" header fields. For each recipient a JSON formatted message is copied to the "inbox" queue (set with the "rabbitmq-inbox" variable).
The envelope address is also extracted from the message: the "x-mq-envelope" header is preferred. If this header is missing, the "return-path", "from" and "sender" headers are checked; the first valid email address that is recognized in these headers will be used as the envelope address.
The mail is published to the "inbox" queue. If you've assigned the outbox queue to the "rabbitmq-inbox" setting, the email is immediately delivered to the recipient(s).
Although using the spool directory is very convenient for sending out mails it is also very slow. This is caused by the fact that every mail needs to be written to and read from disk. We therefore encourage you to use the other methods to send mails to MailerQ.
In the config file the following options are relevant for the spool directory:
spool-directory: /path/to/directory
spool-fail-directory /path/to/directory
spool-delay: 0 (default: 0)
spool-remove: true (default: true)
spool-extract: true (default: true)
spool-open-files: 10 (default: 10)
spool-threads: 1 (default: 1)The "spool-directory" is the most important one. It contains the path to the directory that should be monitored. All files that you copy to this directory are immediately picked up, processed and removed. This happens in a blink of an eye. If you need a small delay before messages are picked up, you can use the "spool-delay" variable. Set this to a numeric value holding the number of seconds to wait before files are picked up.
After publishing the message to the inbox queue, MailerQ removes the input file. If you do not want this, you can set "spool-remove" to false. However, it is not recommended to disable removing files, because emails might get redelivered if MailerQ is restarted and starts scanning the spool directory again.
If the contents of a file cannot be processed, MailerQ can move the file to another directory. The path of the directory can be set using the "spool-fail-directory" option. If this setting is left empty, files are not moved from the spool directory.
All mails that you drop in the spool directory are scanned for headers that start with "x-mq-*". Such headers are filtered out and converted to JSON properties that control the delivery of the mail. If you do not want to scan for such headers, you can use the "spool-extract" variable to disable this.
MailerQ supports processing multiple files in the spool directory simultaneously. The variable "spool-open-files" sets the maximum number of files MailerQ has open at any given time. To use more threads for processing files, you can set "spool-threads".
MailerQ can open one or more SMTP ports on which it accepts incoming mail. All these incoming messages are published to RabbitMQ. The incoming messages are published to the inbox queue, as set with the "rabbitmq-inbox" setting in the config file.
Remember that MailerQ sends out message from the outbox queue - which is a different queue than the inbox queue to which incoming messages are published. By default, incoming message are therefore not immediately sent out again. To overcome this, most MailerQ users give the "rabbitmq-inbox" setting the same value as the "rabbitmq-outbox" setting. All incoming emails are then automatically published to the outbox queue, from which they are then immediately picked up again and scheduled for immediate forwarding.
However, you can also configure MailerQ to use different inbox and outbox queues. MailerQ then stores all incoming messages in the inbox queue first. You can add your own scripts that process these messages and forwards them to the outbox queue. Would you like to add a script that does additional processing or filtering before an incoming message is forwarded? Configure MailerQ to publish received messages in an inbox queue and let your scripts read these messages from this inbox queue. After processing, post the message to the outbox queue where MailerQ picks them up to deliver them.
If you use MailerQ to handle incoming email, you can also use scripts that pick up messages from the inbox queue, and store them in mailboxes or maildirs.
In the config file there are a number of variables that you can use to set the ports on which MailerQ should listen, and from which ips MailerQ should accept incoming SMTP connections:
smtp-ip: 1.2.3.4
smtp-port: 25,2525,1024-1026
smtp-secure-port: 2526The "smtp-port" variable holds the port or ports that MailerQ opens and on which incoming connections are accepted. You can assign a single port, but comma-separated values and port-ranges are also accepted. The default SMTP port is 25, which is the one that you probably want to use.
Normally, MailerQ opens this port on all IP address that are available on the server. This means that if a server has multiple IP addresses, it does not matter to which of its IP addresses you connect: MailerQ listens on all of them. If you want to limit this, you can assign an explicit IP address using the "smtp-ip" variable. If you set this, MailerQ only accepts incoming connections to that specific IP.
The "smtp-port" setting contains the port number for the normal SMTP protocol. The SMTP protocol starts as a non-secure connection. The client and server can start a STARTTLS handshake to secure the connection. Besides normal SMTP connections, it is also possible to open an already secured SMTP socket. Use the "smtp-secure-port" setting for this. A secure port uses TLS right from the start, and no STARTTLS handshake is necessary. This is slightly faster (the initial handshake can be skipped) and is also more secure (the initial EHLO message can not be intercepted). However, such encrypted connections are not part of the SMTP standard, and regular SMTP clients do not expect this. However, if you write your own SMTP handshake code, it sometimes is simpler and faster to have access to a connection that is already encrypted without using "STARTTLS".
The IP address to which you send a message to MailerQ, is normally the same as the address from which the mail is going to be forwarded. Thus, if you send an email to MailerQ listening on IP address 5.6.6.5, the message will also be sent out from this IP. If you want to send out the message from a different IP instead, you should either add a special MIME header to your mail ("x-mq-ip"), or you can use the following config file variables:
smtp-default-ips: 1.2.3.4;4.5.6.7
smtp-unmappable-ips: 1.2.3.28/32;1.2.3.5/32
smtp-mappable-ips: 1.2.3.0/24;4.5.6.0/24The above config file settings tell MailerQ that the default outgoing IP addresses are 1.2.3.4 and 4.5.6.7. All messages that have not an explicit IP address from which they should be sent, will be sent from one of these two IP addresses.
Messages that are submitted over SMTP, are sent out from the same IP address as to which they were submitted if the IP address is mappable. Thus, if you send a message to 1.2.3.100, it will also be sent out from IP addresses 1.2.3.100. An IP address is considered to be mappable if it matches with the IP ranges inside the smtp-mappable-ips variable and does not match the ranges in the smtp-unmappable-ips variable. Thus, messages submitted to 1.2.3.100 are also sent out from 1.2.3.100 because that IP is mappable, while messages to 1.2.3.28 get sent out from the default IP's, because this IP address is not mappable.
To secure your SMTP traffic, you can assign a private key and certificate. You can use a self-signed certificate, but it is much better to use a private key and a certificate from a certificate authority. However, using a self-signed key is still better than using no key at all, as most SMTP clients will still accept the self-signed key.
smtp-certificate: /etc/mailerq/your.domain.com.crt
smtp-key: /etc/mailerq/your.domain.com.key
smtp-ciphers: !aNULL:!eNULL:!LOW:!SSLv2:!EXPORT:!EXPORT56:FIPS:MEDIUM:HIGH:@STRENGTHThe paths to the certificate and private keys for your domain should be added to the config file, with the list of ciphers that you'd like to support.
By default, the whole universe can connect to the inbound SMTP server. You may want to restrict this. If you set a username and password in the config file, all inbound connections must first authenticate before they can inject emails. With the "smtp-range" setting you can also restrict the IP addresses from which you want to allow incoming email.
smtp-range: 192.168.0.0/16
smtp-username: my_user_name
smtp-password: secret_passwordYou can assign a semicolon separated list of IP ranges to the "smtp-range" variable.
Keep in mind that if you set a username and password, the SMTP handshake becomes a little slower (incoming connections first have to authenticate before a message can be delivered). If you do not have to, it is therefore ofter faster to set up a firewall or an IP range to restrict access, than to require authentication.
If you have a username and password installed, messages from unauthenticated connections are normally rejected. However, if an incoming message is sent to an address that is marked as a local address, the message is accepted anyway. MailerQ checks each incoming message and compares the email address with the addresses listed on the "local address list". This list can be managed from the management console.
In the config file you can also assign a special message queue for such local email messages. Every time a message is accepted that is meant for a local address, it will be published to the queue that is assigned to the "rabbitmq-local" config file variable.
If you have also set a "rabbitmq-reports" variable in the config file, MailerQ parses all incoming local messages, and if it detects that it holds a report message (for example a Delivery Status Notification or a DMARC report), it will post the message to this reports queue.
MailerQ can run SPF, DKIM, DMARC and spam checks on incoming messages. The results of these checks are added to the JSON that is published to the inbox queue. You can also instruct MailerQ to modify the incoming messages and add an extra "authentication-results" header to the mail.
smtp-check: spf,dkim,dmarc,spam
smtp-auth-results: local,nonlocalThe "smtp-check" variable in the config file can be set to a comma separated list of checks that have to be performed on incoming messages. Supported values are "spf", "dkim", "dmarc" and "spam". The results of the checks are added to the JSON.
If you also want to add the check results to the "authentication-results" field in the MIME header, you can use the "smtp-auth-results" config file variable. This variable can be set to a comma separated list holding the possible values "local" and "nonlocal". If you set it to "local", the authentication-results header will only be added to incoming emails that were sent to a local email address. The "nonlocal" value does exactly the opposite.
The DKIM, DMARC and SPF checks are all performed by MailerQ ifself. MailerQ has its own embedded algorithms to test whether incoming messages are valid. In practice, it comes down to extracting information from the e-mail, and doing a couple of DNS lookups to check whether the information in the mail matches the settings published in DNS.
Besides comparing mails with settings in DNS, MailerQ can also run a spam-check on incoming messages. For this, MailerQ uses the open source "SpamAssassin" application. Each incoming message is sent to SpamAssassin to test whether it is spam or ham. If you add the "spam" option to the "smtp-check" config file variable, you therefore also need a running SpamAssassin daemon (which is normally called "spamd"). The address where this daemon can be reached must be set in the config file.
spamassassin-host: localhost
spamassassin-port: 783The above config file settings tells MailerQ that the the SpamAssassin daemon is running on the same machine as MailerQ, and that it can be reached on port 783 (which is SpamAssassin's default port).
It is possible to run MailerQ behind a HAProxy server. If you do this, external clients do not directly connect to MailerQ, but to the proxy server instead. This proxy server then forwards the incoming traffic to one of the MailerQ backend servers.
All incoming traffic is thus first passed through the proxy server. From the perspective of MailerQ, the remote IP address of each incoming TCP connection is the IP address of the proxy, and not the (much more interesting) client address. To allow MailerQ to see the client IP address too, the PROXY protocol can be enabled on the HAProxy server. By enabling this protocol, the proxy server adds a small header in front of all forwarded TCP connections, with some meta information of the connection (most importantly, the client IP address).
With the "smtp-proxy" setting you tell MailerQ that when a connection comes in from a specific IP address, it should be treated as a connection from a proxy server, and the first bytes that are received should not be treated as the SMTP protocol, but as the initial PROXY header:
smtp-proxy: 192.168.0.0/16The "smtp-proxy" settings accepts a list of semicolon separated IP ranges. All incoming connections that match one of these ranges will be treated as proxied connections, and the first bytes will be treated as PROXY header, and not as SMTP traffic.
For the JSON, email address validation can be loosened,
so that attempts will be made despite an email address not being fully RFC compliant. To allow
injection of such messages, the smtp-validate-address option should be set. This option is for the
email supplied at the MAIL FROM and RCPT TO stage. Valid values are strict and loose.
By enabling injection for invalid emails, yet requiring RFC compliant emails during the sending phase, the mails can be initially accepted and processed by the rest of the infrastructure via RabbitMQ, instead of having to acount for the possibility that the injection goes wrong.
smtp-validate-address: strictTo prevent that MailerQ exhausts the max number of open TCP connections that is allowed by the OS, you can set a "smtp-connections" setting. There will never be more than this number of total incoming and outgoing connections.
smtp-connections: 100
smtp-connections-in: 50
smtp-connections-out: 75To prevent an excessive amount of either outgoing or incoming connections to block their counterparts from creating connections, the limits can also be adjusted separately. In the example above, the maximum amount of incoming connections are set to 50, but the maximum amount of outgoing connections are 75. When MailerQ is mostly sending, that means that MailerQ only allows 75 outgoing connections at the same time, which means that there are 25 connections reserved for the incoming connections. This guarantees that always at least 25 clients can be connected at the same time. If sending slows down, this number can grow up to 50, at which point it hits the incoming connection limit, preventing it from taking up the connection space for outgoing connections.
If only the global limit would be set in the example above, if MailerQ is connected to 100 endpoints, incoming connections would be queued until the connection limit dropped again. If there are already 100 incoming connections, MailerQ would be unable to send any email, because the global connection limit is already exhausted.
Usually, you want all three values to be set, to guarantee that ever side has a reserved space for the amount of connections that they can make.
The following variables may also be useful when you set up an SMTP server:
smtp-maxsize: 100MB
smtp-threads: 1In the initial SMTP handshake the client advertises its capabilities (like the max message size to accept). The config file setting "smtp-maxsize" contains the maximum allowed input size for message.
All SMTP traffic is handled by separate threads. The number of threads that are started for SMTP traffic can be set with the "smtp-threads" setting. Increasing this value can give a real boost to MailerQ's performance and we recommend to set it close to the number of CPU's that you have in your machine.
Important! The "smtp-threads" variable is meaningful for outgoing connections as well!
If you have not used the "smtp-ip" setting and you run MailerQ on a server with multiple IP addresses, the SMTP server is available on all the server's IP addresses. A client can therefore choose to which IP address to send your mail. MailerQ recognizes the IP address to which the mail was originally submitted, stores that information in the JSON message, and when the mail is finally forwarded to the internet, it will be sent out from exactly the same IP address. You should be aware that this can cause problems if you deliver your email to an IP address from which no external connections can be made (like 127.0.0.1).
If you want MailerQ to send out the mail from a different IP address than that you originally sent it to, you can include an extra MIME header field that instructs MailerQ to use a different IP instead.
All incoming messages are converted into JSON messages and published to RabbitMQ. The minimal JSON message contains the envelope address, the recipient and the mime data, which is all extracted from the SMTP protocol.
If you want to add more settings to the JSON, you can do so with special headers in the MIME message. Every header that starts with "x-mq-" is treated as a special MailerQ header, and will be turned into a JSON property that controls how the mail is going to be handled.
This feature to automatically extract meta data from the MIME header can be disabled in the config file. If you use MailerQ as an internet facing receiver, you probably do not want to allow users to control the deliverability by adding "x-mq-*" these headers, so in that case you can better disable this feature in the config file:
smtp-extract: true (default: true)The list below is a standard list with the SMTP reply codes according to the SMTP standard. Beware however that this is just a standard, and that certain SMTP servers might return their own status, codes and descriptions that differ from the list below.
| Code | Status (SMTP Error) | Description |
|---|---|---|
| 211 | 2.1.1 | System status, or system help reply |
| 214 | 2.1.4 | Help message |
| 220 | 2.2.0 | |
| 221 | 2.2.1 | |
| 250 | 2.5.0 | Requested mail action okay, completed |
| 251 | 2.5.1 | User not local; will forward to <forward-path> |
| 354 | 3.5.4 | Start mail input; end with <CRLF>.<CRLF> |
| 421 | 4.2.1 | <domain> Service not available, closing transmission channel |
| 450 | 4.5.0 | Requested mail action not taken: mailbox unavailable |
| 451 | 4.5.1 | Requested action aborted: local error in processing |
| 452 | 4.5.2 | Requested action not taken: insufficient system storage |
| 500 | 5.0.0 | Syntax error, command unrecognised |
| 501 | 5.0.1 | Syntax error in parameters or arguments |
| 502 | 5.0.2 | Command not implemented |
| 503 | 5.0.3 | Bad sequence of commands |
| 504 | 5.0.4 | Command parameter not implemented |
| 521 | 5.2.1 | <domain> does not accept mail |
| 530 | 5.3.0 | Access denied |
| 550 | 5.5.0 | Requested action not taken: mailbox unavailable |
| 551 | 5.5.1 | User not local; please try <forward-path> |
| 552 | 5.5.2 | Requested mail action aborted: exceeded storage allocation |
| 553 | 5.5.3 | Requested action not taken: mailbox name not allowed |
| 554 | 5.5.4 | Transaction failed |
You can instruct MailerQ to use a "smart host". When you enable this feature, MailerQ does not send the messages directly to the final recipient, but to an alternative mailserver instead: the smarthost.
This feature could for example be useful if you use MailerQ for sending mails from your own server to the internet, but you want to use an external service (like smtpeter.com) to track the clicks and opens for your mailings. Other use cases for smarthosts are when you want to create a chain of MailerQ instances, and have the mails from one instance forwarded to the next instance.
By default, the smarthost feature is disabled. If you want to enable it, you either need to explicitly configure it in the configuration file, or you can enable it on a per-message level. The following config file variables can be used to enable the smarthost for all messages that flow through MailerQ:
smarthost: <hostname> (empty by default)
smarthost-port: <port> (default: 25)
smarthost-username: <username> (empty by default)
smarthost-password: <password> (empty by default)The "smarthost" has to be set to enable the smarthost feature, and should contain the hostname of the server that you want to redirect the email to. The other "smarthost-*" options are optional, and can be used if the target smarthost uses a different port than the default one (default is 25), and if the smarthost requires authentication.
If you do not want to send all messages using a smarthost, but only specific messages or send specific messages to specific smarthosts, you can do so by adding the smarthost variables to the JSON or MIME-header of a specific message. If global smarthost settings are set, these settings will override the settings set in your configuration file.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"smarthost": {
"hostname": "hostname-smarthost",
"port": "25",
"username": "example-user",
"password": "example-password"
}
}You can use MIME headers too to enable the smarthost on a per-message level. The following header variables have the same meaning as the JSON settings mentioned above:
x-mq-smarthost-hostname: <hostname-smarthost>
x-mq-smarthost-port: <25>
x-mq-smarthost-username: <example-user>
x-mq-smarthost-password: <example-password>Only the "hostname" property is mandatory. The port defaults to 25, and not setting the username or password means MailerQ will not try to authenticate to the smarthost.
If you use the "smarthost" option mails are not delivered to the actual recipient but to the smarthost server. As a consequence, MailerQ internally queues all mails as if they were sent to the smarthost domain. If you open the management console, the list of active domains will contain only one item: the smarthost domain indicated by "(smarthost)", which is where all emails are going to be sent to.
If you are debugging, you often want to run MailerQ in (almost) the same configuration as you normally would, with queues for all the domains to which messages are being delivered. The only difference is that you do not want to actually deliver the mails, but send them to a dummy destination: the "smarthost". This is where the smtp-sink options comes in.
smtp-sink-address: <ip address or hostname> (default: 0.0.0.0)
smtp-sink-port: <port> (default: 25)
smtp-sink-username: <username> (empty by default)
smtp-sink-password: <password> (empty by default)If you include the "smtp-sink-address" and "smtp-sink-port" options in the config file, MailerQ runs normally, and all mails are routed through different internal queues with their own send capacities. However, when a TCP connection is created, the connection will be set up to the sink instead of to the actual recipient mail server. Bear in mind that if a hostname is provided, it will only be resolved once, at the start of the application, and the resolved IP will be used until the application is stopped.
The "smtp-sink-username" and "smtp-sink-password" options can be used if your sink requires authentication.
If you are looking for an SMTP server that simply accepts and discards incoming messages, you can use Postfix' smtp-sink.
It is possible to configure both the smtp-sink option and the smarthost at the very same time. If you do this, MailerQ will keep a single queue of messages to the smarthost (and thus not separate queues for hotmail, gmail, yahoo, et cetera) - but when the TCP connection is set up, it does not connect to this smarthost, but to the sink address instead.
This article contains a list of all configuration settings, sorted alphabetically. Each setting links to a documentation article that explains the setting more in-depth. Or do you want to search by topic?
| Setting | Description |
|---|---|
| download-blacklist | IP addresses from which no resources can be downloaded |
| download-cache | Address of the cache (Used in ResponsiveEmail) |
| download-ttl | Max time to keep downloaded resources in the cache (Used in ResponsiveEmail) |
| download-whitelist | Exceptions to the download-blacklist |
| cluster-address | Address for a cluster to share between instances |
| cluster-exchange | Set cluster exchange |
| cluster-timeout | Number of seconds the --list-cluster option waits |
| cluster-verify | Should the server certificate be verified (only used for amqps:// connections) |
| database-address | Relational database for config data and delivery settings |
| database-threads | Number of threads to use for DNS lookups for DKIM keys in the database when data is refreshed |
| database-ttl | Time to live: Database reload interval (Default 600s) |
| database-validate | Check and fix database settings on startup |
| download-log-format | Download log format |
| download-log-file | Download log filename |
| download-proxy | Route outgoing download requests through a proxy |
| dsn-advertise | Toggle announcement about DSN extension and parameters |
| dsn-notify | Events that trigger a notify |
| dsn-ret | Send full mail (FULL) or headers (HDRS) for notifications |
| dns-threads | Number of threads used for DNS lookups |
| dns-getaddrinfo | Whether the MailerQ resolver should rely on getaddrinfo() library calls |
| dns-min-ttl | Minimum TTL to respect when doing domain lookups |
| error-log | Full path to the error log |
| heartbeat-enabled | Toggle user statistics (default: true) |
| license | MailerQ license file |
| lock | Process ID for MailerQ instance to prevent from starting instances more than once |
| max-attempts | Maximum attempts to deliver email |
| max-delivertime | Maximum time spent trying to deliver email |
| plugin-directory | Directory with MailerQ plugins |
| proxy-server | Setup proxy for outgoing HTTP requests |
| rabbitmq-address | Location and authentication to connect to RabbitMQ |
| rabbitmq-consumers | Amount of consumer threads (default: 1) |
| rabbitmq-declare | Enable or disable checking and declaring the queues on startup |
| rabbitmq-dsn | Your RabbitMQ delivery status notification queue |
| rabbitmq-durable | Durable/not durable RabbitMQ queues (default: true) |
| rabbitmq-encoding | Choose encoding for messages (such as gzip) |
| rabbitmq-exchange | RabbitMQ exchange |
| rabbitmq-failure | Your RabbitMQ failure queue |
| rabbitmq-inbox | Your RabbitMQ inbox queue |
| rabbitmq-local | Your RabbitMQ local queue |
| rabbitmq-lazy | Lazy/non lazy RabbitMQ queues (default: non-lazy) |
| rabbitmq-maxpriority | Enable the use of priority queues |
| rabbitmq-qos | Set a maximum number of messages simultaneously consumed per consumer thread |
| rabbitmq-flush-qos | Set a maximum number of messages simultaneously consumed during flush |
| rabbitmq-outbox | Your RabbitMQ outbox queue (must be unique if multiple instances of MailerQ are used) |
| rabbitmq-persistent | Persistent/not persistent RabbitMQ queues (default: false) |
| rabbitmq-publishers | Amount of publisher threads (default: 1) |
| rabbitmq-queues | Names of the RabbitMQ queues to manage email (inbox, failures, retries, etc...) |
| rabbitmq-refused | Your RabbitMQ refused queue |
| rabbitmq-reports | Your RabbitMQ report queue |
| rabbitmq-results | Your RabbitMQ result queue |
| rabbitmq-retry | Your RabbitMQ retry queue |
| rabbitmq-success | Your RabbitMQ success queue |
| rabbitmq-verify | Should the server certificate be verified (only used for amqps:// connections) |
| received-log-format | Received log format |
| received-log-file | Received log filename |
| retry-interval | Interval for retrying to send |
| send-bin-log-file | Binary send log filename |
| send-log-file | Send log filename |
| send-log-format | Send log format |
| server-id | Server ID to prevent multiple instances from assigning same message ID |
| smarthost | Smarthost name (only needed if Smarthost is desired) |
| smarthost-password | Smarthost password |
| smarthost-port | Smarthost port |
| smarthost-username | Smarthost username |
| smtp-auth-results | Toggle adding authentication results to MIME header |
| smtp-certificate | Certificate for secure SMTP connections |
| smtp-check | Comma separated list of checks to perform on incoming messages |
| smtp-ciphers | Supported ciphers for secure SMTP connections |
| smtp-connections | Maximum amount of open TCP connections allowed |
| smtp-default-ips | List of default outgoing IP addresses |
| smtp-extract | Toggle automatic extraction of metadata |
| smtp-ip | SMTP server IP |
| smtp-key | Private key for secure SMTP connections |
| smtp-mappable-ips | List of mappable IP adresses |
| smtp-maxsize | Maximum message size to accept |
| smtp-password | Password for authenticating to SMTP server |
| smtp-port | Ports for incoming MailerQ connections |
| smtp-proxy | IPs to treat as connection from proxy server |
| smtp-range | Range of IP address to allow incoming email for |
| smtp-secure port | Ports for secure incoming MailerQ connections |
| smtp-sink-address | SMTP sink domain name or ip address |
| smtp-sink-password | SMTP sink password |
| smtp-sink-port | SMTP sink port |
| smtp-sink-username | SMTP sink username |
| smtp-threads | Number of threads started for SMTP traffic |
| smtp-unmappable-ips | List of unmappable IP addresses |
| smtp-username | Username for authenticating to SMTP server |
| spamassassin-host | Address for SpamAssassin daemon |
| spamassassin-port | Port for SpamAssassin daemon |
| spool-delay | Delay for processing spool directory files |
| spool-directory | Path to the directory to monitor |
| spool-extract | Toggle scanning for headers in spool directory mail |
| spool-fail-directory | Path to move files to that could not be picked up |
| spool-open-files | Amount of files that may be open simultaneously by the spool directory |
| spool-remove | Toggle removal of files from spool directory |
| spool-threads | Amount of threads to use for the spool directory |
| storage-address | Address for external message storage |
| storage-policy | Message store policy |
| storage-reschedule | Number of seconds to wait before a mail is retried |
| storage-threads | Storage threads |
| storage-timeout | Timeout for fetch operations |
| storage-ttl | Time to live: Storage reload interval |
| user | Change user for MTA |
| validate-envelope | Validate envelope address before sending |
| validate-recipient | Validate recipient address before sending |
| www-cache-control | Enable or disable browser cache |
| www-certificate | Certificate file for secure connection to management console |
| www-ciphers | Supported ciphers for secure management console |
| www-connections | Limit for simultaneous HTTP connections to built-in HTTP server |
| www-ip | Web interface IP (IP/port combination unique for every instance of MailerQ) |
| www-dir | Directory for installing MailerQ management console files |
| www-host | Web interface host (IP/port combination unique for every instance of MailerQ) |
| www-password | Management console password |
| www-port | Port number for the management console |
| www-private-key | Private key for secure connection to management console |
| www-secure-port | Secure (HTTPS) port number for the management console |
| www-url | The URL via which the management console is accessible |
To check your version of MailerQ for example, just type in the cli:
$ mailerq --version| Setting | Description |
|---|---|
| daemon | Deamon process (deafault true) |
| disable-crash-report | Disable crash report |
| disable-crash-post | Disable automatic crash posting |
| extract-recipients | Read in a mime message from standard input and filter out the destination addresses |
| ignore-dot | Instruct MailerQ that dots do not have a special meaning |
| list-ips | List IP addresses |
| loopback | Loopback |
| notify-cluster | Notify cluster |
| paused | Pause |
| purge-database | Rebuilding the database tables |
| repair-database | Repair the database |
| version | Version |
Some mail servers initially seem to accept a message, but send back a bounce email later on in which they inform you that the mail could not be delivered after all. Such asynchronous bounces are sent back to the envelope address of the initial message. There is sadly often no way to find out whether a server is going to send back notifications, and what these messages are going to look like.
However, there is a formal standard for these bounce messages: the Delivery Status Notification (DSN) format. And there is an extension to the SMTP protocol that allows mail servers to exchange with each other whether they respect this standardized format, if the original sender would like to receive such bounce messages, and what kind of information he or she wants to receive. This standard make it much easier to link incoming bounces to the original sent email.
MailerQ supports this standard and SMTP extension: when communicating with a receiving server, DSN parameters can be passed to tell the receiver what kind of bounces should be returned. If MailerQ is configured to receive mail on a SMTP port, it understands these kind of parameters too. MailerQ can also be configured to send out, receive and recognize DSN messages.
Delivery Status Notifications are sent back to the envelope address of your messages. This is the address that is used in the "MAIL FROM" command of the SMTP handshake, and that is stored in the "envelope" property of the input JSON. If you want to receive DSN messages, you therefore first have to make sure that your envelope address is valid, and that it indeed points back to your server.
Besides the envelope address that is used to send back DSN messages, some mail servers (but not all!) have implemented the SMTP DSN extension, and allow extra parameters to be passed to instruct them what kind of DSN messages you would like to receive back. You can for example specify that you want to see the full original mail message in the notification, or just the headers. You can also specify whether you want to receive notifications on failure, on success and/or on a delayed delivery.
{
"envelope": "youraddress@yourdomain.com",
"recipient": "info@example.com",
"mime": "....",
"dsn": {
"notify": "FAILURE",
"ret": "HDRS",
"envid": "unique-identifier",
"orcpt": "info@example.com"
}
}The above JSON sets "youraddress@yourdomain.com" to be the envelope address. If something goes wrong, the notification will be sent to this address. If the receiving mail server also supports the DSN SMTP extension, MailerQ uses the "dsn" property in the input JSON to pass additional DSN parameters to the receiver. You can use the following four DSN settings:
| notify | comma separated events that trigger a notification (FAILURE, DELAY, SUCCESS, NEVER) |
| ret | should the notification hold the full original mail or just the headers (FULL, HDRS) |
| orcpt | the address to be included in the notification as "original-recipient" |
| envid | unique identifier to be included in the notification as "original-envelope-id" |
The "notify" variable specifies what sort of events trigger the Delivery Status Notification. Possible values are "NEVER", "SUCCESS", "FAILURE" and "DELAY". If you set it to "FAILURE", you will only receive bounces on failure. It is also allowed to set this to a comma separated list of values. If you set the "notify" property to "SUCCESS, FAILURE", a bounce message will be sent on successful delivery as well as on failure.
The bounce message holds a full delivery report, as well as the full original message. If you want to save a lot of network bandwidth, you can set the "ret" property to the value "HDRS". By doing so, you instruct MTAs not to include the full MIME message in the status notification, but only the original MIME headers.
The property "orcpt" is an optional property and holds the original recipient address. This original recipient address is included in the delivery report. This property is optional; if you leave it out the actual recipient will be used. This property is only useful if you want the DSN to contain a different "original-recipient" value than the actual recipient.
The last property that you can include in the "dsn" object, is "envid". This is a message specific message identifier, and you can set it to whatever you like. It will be included in the bounce message as "original-envelope-id", and you can use it to match the bounces with a sent message.
Keep in mind that not every mail server supports the DSN extension, and that even if they do support it, they may not always respect your parameters. There is no guarantee that you receive the DSN that you wanted. For example, if you specify that you want to receive the full original mail in the bounce (you set "ret" to "FULL"), it still is possible that you do not receive a bounce at all, or a bounce with just the headers.
In the above example, we've demonstrated the JSON properties for passing DSN parameters. If you inject mails using a non-json format (via SMTP, the spool directory or via MailerQ's command line interface), you can use MIME headers instead. The following MIME headers are recognized, and do the same thing as the JSON settings:
x-mq-dsn-notify: FAILURE
x-mq-dsn-ret: HDRS
x-mq-dsn-envid: your_unique_identifier
x-mq-dsn-orcpt: email@example.comIf one or more of the DSN settings are missing, the default values from the global config file are used. These defaults can be set with the following config file variables:
dsn-notify: FAILURE
dsn-ret: HDRSYou can only set defaults for the "notify" and "ret" settings, because the "envid" and "orcpt" are normally only used on a per-message setting and it is not meaningful to have default values that apply to all emails.
MailerQ has its own ways to report the results: it publishes JSON result objects to the result queues, and it writes results to log files. These reporting methods are powerful, and easy to integrate. Most of the MailerQ users therefore do no let MailerQ send out its own DSN messages back to the envelope address. The only DSN messages that are normally sent back to your envelope address are notifications from MTAs that initially accepted the mail.
However, MailerQ can be configured to send out DSN messages too. If you set up a RabbitMQ DSN queue (using the "rabbitmq-dsn" config file property), MailerQ starts sending out DSN messages. Every time that a message cannot be delivered, an outgoing bounce mail is constructed and published to this DSN queue. If you assign the outbox to this "rabbitmq-dsn" setting, the bounces are even automatically picked up and delivered.
We wrote that a bounce is sent on every failed delivery. This is technically not completely correct. To be precise, a bounce is created everytime that the message specific "dsn.notify" setting matches. Thus, if "notify" is set to "SUCCESS", MailerQ sends out a bounce message on successful delivery, and when it is set to "FAILURE", bounces are sent on failure. The "notify" setting is normally set to "FAILURE" so in normal circumstances bounces are sent on failure.
If you set the "rabbitmq-dsn" setting to an empty string (which is the default), no Delivery Status Notifications are sent by MailerQ. But MailerQ does forward the DSN settings to the receiving MTA, so you could still receive DSNs from MTAs further up the chain.
If you inject mail into MailerQ using SMTP, you can pass DSN parameters with the "MAIL FROM" and "RCPT TO" commands.
MAIL FROM:<envelope@example.com> RET=HDRS ENVID=unique_identifier
RCPT TO:<info@example.com> NOTIFY=FAILURE ORCPT=rfc822;info@example.comThese parameters are directly copied to the "dsn" property in the JSON that is published to the inbox queue. To enable or disable this feature, use the "dsn-advertise" setting in the config file.
dsn-advertise: 1If you set dsn-advertise to 1, MailerQ announces in its SMTP connection handshake that it supports the DSN extension, and converts the DSN parameters that are passed to the "MAIL FROM" and "RCPT TO" commands to JSON properties. If you set it to 0, no DSN parameters can be passed to "MAIL FROM" and "RCPT TO".
MailerQ can also be used to recognize incoming DSN messages. If you submit a DSN message to the SMTP port of MailerQ, and MailerQ detects that both the recipient address is on the list of local email addresses and that the message holds a report message, it publishes this report to the queue for incoming reports. The name for this queue can be set in the config file with the "rabbitmq-reports" variable.
This means that you can not only use MailerQ to send out mass mailings, but that you can also run a MailerQ instance to process the delivery reports that come back. If you use "reports@yourdomain.com" as envelope address, you can run a MailerQ instance to process all mails for the yourdomain.com domain. Via the management console you add the "reports@yourdomain.com" local email address, so that all these incoming messages are accepted and sent to either the "local" message queue for normal email messages, and the "reports" message queue for emails that were recognized as reports.
MailerQ does not only recognize DSN messages, but also other types of reports like DMARC reports and disposition notifications.
Once you have a running MailerQ instance, you want to use it to send out mailings. But how do you get your mails into MailerQ? There are many ways to do this:
After sending the email, MailerQ publishes a JSON-encoded copy to the result queue, and, depending on whether the delivery was successful, into the success or failure queues. Your own application or script can read these results up for further processing.
MailerQ can be accessed by REST API, and delivery flow can be altered. In this way, it allows you to stop the sending of certain emails, intercept emails, or inject emails directly into MailerQ with only a HTTP client.
Authentication to the API can be done using tokens that can be inserted into the database directly, or created in the web interface. To create a token, go to the management console Settings > Rest API. Either copy an already existing token, or generate a new token with the button in the upper right corner.
To authenticate to the API, an Authorization header should be added to each request. For example, Authorization: Bearer <token> would be the correct way to authenticate to the API. Currently, the authentication tokens give full access to the complete API.
The following endpoints are available in the current version of MailerQ.
MailerQ offers very fine-grained control over delivery flow, and allows you to pause almost any combination of pool/mta and domain/ip.
A GET request returns all currently active pauses. It returns a JSON array of pauses, see the example below.
GET /v1/pauses HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"mta": "1.2.3.4",
"ip": "123.2.3.4",
"cluster": false
},
{
"mta": "1.2.3.4",
"domain": "gmail.com",
"cluster": false
},
{
"domain": "hotmail.com",
"cluster: true
},
{
"pool": "shared-pool-1",
"domain": "yahoo.com",
"cluster": false
}
]In this example, traffic from 1.2.3.4 to 123.2.3.4 and gmail.com is paused on the instance, all traffic from shared-pool-1 to yahoo.com is paused, and traffic to hotmail.com is paused for the entire cluster.
A POST request allows you to create a new pause, or to update an existing one. For the request format, see the table below. All fields are optional.
| Field | Required | Type | Description |
|---|---|---|---|
| pool | no | string | Pool that the pause applies to. |
| mta | no | string | MTA IP that the pause applies to. |
| domain | no | string | Domain that the pause applies to. |
| ip | no | string | Remote IP that the pause applies to. |
| tag | no | string | The tag that the pause applies to. |
| cluster | no | bool | Whether or not this pause is for the entire cluster or only this instance. |
Do note that not all fields may be passed at the same time. pool and mta are mutually exclusive, and so are domain and ip. That means that they may not be supplied together, or they will result in a 400 response.
For example, the request below will pause sending to hotmail.com for a single campaign.
POST /v1/pauses HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"domain": "hotmail.com"
"tag": "Campaign1"
}and equivalently, with urlencoded body
POST /v1/pauses HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
domain=hotmail.com&tag=Campaign1A DELETE request allows you to remove a pause. For example, the request below removes the pause for the entire cluster to gmail.com from the list of pauses.
DELETE /v1/pauses HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
cluster=true&domain=gmail.comThe request format is exactly the same as a POST request, but with the DELETE method. Delete also supports both JSON and urlencoded requests.
MailerQ offers an HTTP injection API. Check out the message format for the required structure of injected messages.
POST /v1/inject HTTP/1.0
Authorization: Bearer ...
Content-Type: application/json
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\nTo: info@example.org\r\nSubject: ..."
}Injected messages simply get published to the inbox queue specified in the config file. The injection endpoint does not support urlencoded post, only JSON directly.
Injected messages will get an extra http property, for its properties see incoming messages.
MailerQ offers very fine-grained control over delivery flow, and allows you to intercept it and forcing an error on any combination of pool/mta and domain.
A GET request returns all currently active errors. It returns a JSON array of errors, see the example below.
GET /v1/errors HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"mta": "1.2.3.4",
"ip": "123.2.3.4",
"cluster": false,
"code": 500
},
{
"mta": "1.2.3.4",
"domain": "gmail.com",
"cluster": false,
"code": 421,
"description": "forced deferring"
},
{
"domain": "hotmail.com",
"cluster: true,
"code": 521,
"extended": "5.2.1"
},
{
"pool": "shared-pool-1",
"domain": "yaho0.com",
"cluster": false,
"code": "553",
"description": "invalid mailbox"
}
]In this example, traffic from 1.2.3.4 to 123.2.3.4 is failed with an error code 500, anc traffic to gmail.com is deferred with
an error code of 421 to be tried again later. In the third value, all emails to hotmail.com get an 521 error on the entire cluster.
Lastly, all mail from shared-pool-1 to yaho0.com on this instance are failed because the mailbox is invalid.
A POST request allows you to create a new error, or to update an existing one. For the request format, see the table below. All fields are optional.
| Field | Required | Type | Description |
|---|---|---|---|
| code | yes | int | Numeric error code between 200 and 599 (smtp error codes). |
| extended | false | string | Extended SMTP error code, e.g. 5.7.1. |
| description | false | string | Description to put in the message result. |
| pool | no | string | Pool that the error applies to. |
| mta | no | string | MTA IP that the error applies to. |
| domain | no | string | Domain that the error applies to. |
| tag | no | string | The tag that the error applies to. |
| cluster | no | bool | Whether or not this error is for the entire cluster or only this instance. |
Do note that not all fields may be passed at the same time. pool and mta are mutually exclusive.
That means that they may not be supplied together, or they will result in a 400 response.
For example, the request below will install a forced error with code 421 to hotmail.com for a single campaign.
POST /v1/errors HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"domain": "hotmail.com",
"code": 421,
"tag": "Campaign1"
}and equivalently, with urlencoded body
POST /v1/errors HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
domain=hotmail.com&code=421&tag=Campaign1A DELETE request allows you to remove an error. For example, the request below removes the error for the entire cluster to gmail.com from the list of errors.
DELETE /v1/errors HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
cluster=true&code=421&domain=gmail.comThe request format is exactly the same as a POST request, but with the DELETE method. Delete also supports both JSON and urlencoded requests.
Copernica, the company that develops MailerQ, also operates an online service to create responsive emails based on JSON input: responsiveemail.com. The algorithm to transform JSON into responsive emails has been embedded in MailerQ. If you feed MailerQ JSON (instead of MIME), it will be converted into valid email messages.
{
"envelope": "sender@example.com",
"recipient": "receiver@example.com",
"mime": {
"from": {
"name": "John Doe",
"email": "sender@example.com",
},
"subject": "Example email",
"text": "Text version",
"background": {
"color": "white"
},
"content": {
"blocks": [{
"type": "html",
"content": "<p>Example content</p>"
}, {
"type": "button",
"label": "click here"
}]
}
}
}MailerQ reads JSON objects from the RabbitMQ "outbox" queue. These JSON objects hold properties like "envelope", "recipient" and "mime", where the "mime" property is normally set to a string holding the full message source in MIME format. However, you can also assign a nested JSON object to this "mime" property (see above example). This nested object is processed by MailerQ and transformed into a MIME message.
There are many nested properties that are supported by the algorithm, which are all documented on the ResponsiveEmail.com website. Note that the properties that are described as top-level properties on the responsiveemail.com website, are (of course) nested properties inside the "mime" property of the object that is processed by MailerQ.
The ResponsiveEmail algorithm reads in a JSON object in which the texts, buttons and images are defined, and generates the HTML and MIME content based on that input. This means that you do not have to write the message source (HTML, CSS and MIME) yourself, or worry about the tricks to make your message compatible with all the various email clients.
However, if you do want to take care of the HTML yourself, you can also use the ResponsiveEmail algorithm for just the HTML-to-MIME conversion. In that case you just specify the header properties (from and to address, subject line) and the HTML and text source of your message, and rely on MailerQ to turn this into a valid MIME message:
{
"envelope": "sender@example.com",
"recipient": "receiver@example.com",
"mime": {
"from": {
"name": "John Doe",
"email": "sender@example.com",
},
"to": {
"name": "Mister Receiver",
"email": "receiver@example.com",
},
"cc": {
"name": "Someone Else",
"email": "randomguy@example.com",
},
"subject": "Example email",
"headers": {
"list-unsubscribe": "<mailto:unsubscribe@example.com>"
},
"text": "This is the pure text version of the email",
"content": "<html><head> ..... </body></html>",
"attachments": [ {
"url": "http://www.example.com/document.pdf"
} ]
}
}If you use the full ResponsiveEmail algorithm (where you do not have to write the HTML code yourself), you assign a deeply nested JSON object to the "content" property. Inside this "content" object you specify the images, texts, buttons and other elements that you want to include in your email. MailerQ turns this content into a responsive email message. However, in the above example we assigned a string to the "content" property. By doing this, MailerQ creates a MIME property with exactly the text and HTML that you've set in the JSON.
When emails are generated, MailerQ sometimes has to download resources from the internet to find out the dimensions of images and to fetch attachments. To prevent that the same resources are downloaded over and over again, MailerQ can be configured to use a cache. In the configuration file you can set the address of this cache:
download-cache: directory:///path/to/dir
download-ttl: 3600The "download-cache" property holds the address of the cache. The format of this address is the same as the message-store-options variable, and can refer to a directory (like we did above) or a nosql database if you use an address that starts with "mongodb://" or "couchbase://".
MailerQ inspects the 'cache-control' and 'expires' headers in the HTTP response to decide for how long a resource can be stored in the cache. With the 'download-ttl' config file variable you can set the upper limit for this. If you set it to 3600, downloaded files will stay for at most one hour (3600 seconds) in the cache, even if the HTTP response headers allowed a longer cache time.
For the downloads, a custom user agent can be supplied. By default, the user agent
mailerq uses is 'MailerQ
download-user-agent: My custom user agentMany users run MailerQ on a server with unrestricted access to the local network. MailerQ is then not only capable of fetching resources from the internet, but also directly from servers inside this local network. This makes it, for example, possible to use attachment URL's like "http://192.168.0.22/attachment.pdf". This could be a security risk, especially if the attachment and image URL's in your mails can be entered by third parties.
To prevent such internal downloads, you can use the config file variables "download-blacklist" and "download-whitelist" to limit the IP addresses to which MailerQ can connect.
download-blacklist: 192.168.0.0/16,127.0.0.1
download-whitelist: 192.168.0.30/32:80Downloads from IP address on the blacklist are forbidden, unless the IP is also included in the whitelist. The above configuration prevents that MailerQ can connect to servers on the local network, with an exception for port 80 of IP 192.168.0.30.
It is possible that the server cannot access the internet, except through a proxy. That
is why there is a config option to set up either a SOCKS or an HTTP proxy for downloads.
For the proxy URI format, see proxy settings in other options.
If unset, the global proxy-server variable will be the fallback.
download-proxy: <proxy URI> (empty by default)MailerQ depends on RabbitMQ for message queueing. Before you can even start MailerQ, you first need a running RabbitMQ instance. The www.rabbitmq.com website has all the information you need for setting up RabbitMQ. We do however have some tips, tricks and recommendations for setting up RabbitMQ with MailerQ.
The RabbitMQ version that is installed in the repository of your operating system might be outdated. You really need a version that is up-to-date, because MailerQ uses a couple of RabbitMQ features that were only recently added. We therefore recommend downloading and installing RabbitMQ directly from the www.rabbitmq.com website instead of using the version that comes with your OS.
Click here to download and install RabbitMQ.
The RabbitMQ installation has to be at least version 3.3.1+ for MailerQ to be able to connect to it.
By default, RabbitMQ is installed with a user with login guest and password guest.
These are the login credentials that you have to include in the configuration file
of MailerQ. However, this guest/guest login only works for clients that connect
to RabbitMQ locally (from the same machine). If you run MailerQ and RabbitMQ on
different servers, the guest/guest login does not work. Therefore, if you install
RabbitMQ and MailerQ on different machines, you either need to add a user with a
different name and password, or you should configure RabbitMQ to allow guest/guest
logins from remote hosts as well.
To allow remote guest/guest logins, you can use the loopback_users setting in the
RabbitMQ config file. By including this option in the RabbitMQ config file, you tell
RabbitMQ that it is ok for clients to login with guest/guest, even if the connection
comes from a remote location. If you do include this setting, please make sure that
you also have a firewall running, because you do not want everyone from all over the
internet to connect to your RabbitMQ instance!
Read more about setting up loopback_users.
Just like MailerQ, RabbitMQ comes with a very nice web interface. However, this web interface is not enabled by default, and must be explicitly configured. We recommend doing this, because it is much easier to control RabbitMQ via a web browser, than with command line tools.
Read more about setting up the browser interface..
MailerQ not only uses RabbitMQ to fetch the messages that it is going to send, but also publish back the delivery results (if you have this configured). If you do not process these delivery results in time, the queues in RabbitMQ will get fuller and fuller and you run the risk that your RabbitMQ server runs out of resources (memory or disk space), which might crash the server. This can especially happen in a production environment, where many messages are published and consumed.
So, when you run MailerQ in production, do make sure that you have set up cronjobs or other scripts that periodically or continuously process the messages from the result queues.
All config file settings that start with rabbitmq-* control how MailerQ interacts with RabbitMQ. Amongst these settings are the address of the RabbitMQ server, the queue from which outgoing messages are read, and the queues to which incoming messages are delivered.
MailerQ reads the location and authentication information to connect to RabbitMQ from its config file. This data is stored in the "rabbitmq-address" setting:
rabbitmq-address: amqp://login:passwd@hostname/vhostThe format of the address is obvious: the "login" and "passwd" hold the username and password of the RabbitMQ server, the hostname is the name of the server on which RabbitMQ runs, and the optional vhost is the name of the "virtual host" inside RabbitMQ that you have reserved for all MailerQ related data.
If you leave the "rabbitmq-setting" setting empty, MailerQ uses the "amqp://guest:guest@localhost/" default value. This default only works if you run RabbitMQ and MailerQ on the same server, and when you do not use a special vhost.
If you have a cluster of RabbitMQ nodes, you can use commas to separate the addresses:
rabbitmq-address: amqp://guest:guest@host1/vhost,amqp://guest:guest@host2/vhost,amqp://guest:guest@host3/vhostRunning a cluster allows you to use highly available queues.
If your RabbitMQ server supports secure connections, you can configure MailerQ to connect to an "amqps://" address instead. The communication between MailerQ and RabbitMQ will then be encrypted.
rabbitmq-address: amqps://guest:guest@hostname/vhost
rabbitmq-verify: falseMailerQ checks the server certificate when it connects to a secured RabbitMQ server. If this certificate can not be verified with one of the known openssl certificate authorities, MailerQ refuses to set up a connection. If you use a self-signed certificate, you may want to skip this extra test, and set the config file option "rabbitmq-verify" to "false".
MailerQ uses several queues to manage email messages. All mails that flow through MailerQ are picked up from and published back to specific queues in RabbitMQ. The names of these queues are set in the config file. On startup, MailerQ reads this configuration and tells RabbitMQ to create the listed queues. This means that you do not have to ensure that the queues exists before you start up MailerQ: the queues are automatically created.
rabbitmq-outbox: <Name of your outbox message queue>
rabbitmq-inbox: <Name of your inbox queue>
rabbitmq-reports: <Name of your reports queue>
rabbitmq-local: <Name of your local queue>
rabbitmq-refused: <Name of your refused queue>
rabbitmq-dsn: <Name of your delivery status notification queue>
rabbitmq-results: <Name of your result queue>
rabbitmq-success: <Name of your success queue>
rabbitmq-failure: <Name of your failure queue>
rabbitmq-retry: <Name of your retry queue>The messages published to these queues are always in JSON format, and hold the meta information for each delivery: the envelope, recipient and other message specific settings.
All emails that MailerQ is going to send out are fetched from the outbox queue. The outbox queue feeds MailerQ with messages to be delivered. The name of this queue can be set with the "rabbitmq-outbox" setting in the config file. The default value is "outbox".
The outbox queue is the only queue from which messages are read. If you want to inject outgoing email messages directly into RabbitMQ, you should therefore publish your messages to this outbox queue so that MailerQ automatically picks them up and delivers them.
Watch out if you run multiple MailerQ instances. If the individual instances all send out mail from different IP addresses (because they run on different servers), it is better to use a different outbox queue for every instance. Every MailerQ instance consumes messages from its own queue. You must then also ensure that you publish messages to the right queue to have them being sent from the right server. If you've set up a MailerQ cluster, messages are automatically moved to the correct queue if they end up in a wrong outbox queue.
The queues other than the outbox queue are used by MailerQ to write data to. These are for example queues to which the results are written, and queues that are used for incoming messages. Incoming messages for example come from the MailerQ SMTP port or are the messages that are dropped in the spool directory.
Messages that are received by MailerQ via one of the injection mechanisms are converted into JSON format and published to message queues. The "rabbitmq-inbox" setting specifies the default queue for these incoming messages, but other queues are used as well.
rabbitmq-inbox: inbox
rabbitmq-local: local
rabbitmq-reports: reports
rabbitmq-refused: refusedIf you do not set an explicit "inbox" queue, all incoming messages are automatically published to the "outbox" queue. This means that all incoming messages are automatically sent out again. Leaving this "inbox" queue empty is a very common thing to do, especially if you set up MailerQ as a retransmitter.
The "refused", "reports" and "local" settings are optional too. If you leave them empty, MailerQ simply publishes all incoming message to the inbox queue (or to the outbox queue if no inbox was set). But if you do configure one or more of these queues, MailerQ analyzes each incoming message to see if it should be published to the normal inbox, or to one of the other queues instead.
MailerQ internally keeps a list of "local" email addresses. This list can be edited via the management console. If you have configured a "local" message queue, and a mail comes in for an address that is on that list, the message will not be dropped in a the "inbox" queue, but in the "local" queue instead.
If the local email turns out to be not a normal type of email, but some kind of delivery report (or some other kind of report), the message is not even placed in the "inbox" queue but goes to the "reports" queue instead. To summarize:
The last queue to mention is the "refused" queue. It is used for messages that were not accepted. For example, if you configure MailerQ to listen to one or more SMTP ports, and you require incoming connections to be authenticated, you might receive messages over unauthenticated connections. These messages are rejected and do not end up in the "inbox" queue. However, for debugging and/or security reasons you might still want to find out who is flooding your SMTP server. By assigning a value to "rabbitmq-refused", you instruct MailerQ to send rejected messages to a special queue where you can inspect these refused messages. The default setting for "rabbitmq-refused" is empty, so that refused messages are not collected.
Be careful here: the queue with refused messages does not automatically get emptied, and can fill up fast in case of an attack. MailerQ only adds messages to it, and it is up to you to periodically check the contents of the queue and empty it, or to set a max length and/or max age for messages in the queue (you can use RabbitMQ's web based management console to set such limits).
In the above sections we described the queue from which MailerQ reads its outgoing messages (the outbox queue) and the queues to which all incoming messages are published. However, besides handling incoming and outgoing messages, MailerQ can also construct and send email messages all by itself: delivery status notifications (DSN).
A delivery status notification is an email message that is sent back to the original envelope address when a delivery fails (technically, it is also possible to send such notifications on successful delivery and when a message gets delayed, but in practice it is mostly used for failure notifcations). By default MailerQ does not send such notifications because MailerQ uses result queues and JSON to report back the delivery results. However, if you add a "dsn" property to the JSON of an outgoing message, you can instruct MailerQ to send delivery status notifications too.
Technically, a delivery status notification is a regular email, and MailerQ simply posts a message to the outbox queue when such an email has to be delivered. If you want to use a different queue than the outbox queue, you can use the "rabbitmq-dsn" property.
rabbitmq-dsn: alternative_dsn_queueThe "rabbitmq-dsn" setting is normally used if you want to preprocess delivery reports before they are sent. By setting the "rabbitmq-dsn" value to a custom queue, a custom script can pick up the notifications, preprocess them and put them in the "outbox" queue.
When a message gets delivered or when a delivery fails, the delivery result is published to one or more of the result queues, to allow external monitor scripts to keep an eye on the deliveries.
rabbitmq-results: completed_deliveries
rabbitmq-success: successful_deliveries
rabbitmq-failure: failed_deliveries
rabbitmq-retry: delayed_deliveriesThe "rabbitmq-results", "rabbitmq-success" and "rabbitmq-failure" settings hold the names of the RabbitMQ result queues. MailerQ posts a JSON result object for every successful delivery to the success queue, and for all failed deliveries to the failure queue. This same result object is also sent to the results queue. Every result object is thus published to two queues: failures are posted to the results queue and the failure queue, while successes are published to the results and success queues.
MailerQ only considers a delivery to be a failure when no new delivery attempts get scheduled. When a message cannot immediately be delivered, or when it is greylisted and will be retried, it is published back to the outbox queue, and not to the results and/or failure queues.
If you want to process the intermediate results (like greylisting reports) too, you can use a retry queue by setting the "rabbitmq-retry" config setting. All deliveries that failed, but that are published back to the outbox, are also posted to this retry queue.
If you're not interested in the results, or when you're only interested in specific results (like failures), you can set one or more of the result queues to empty strings. MailerQ will then not publish messages to them.
RabbitMQ supports priority queues. If you use a priority queue for the outbox, messages with a higher priority are consumed by MailerQ before messages with a lower priority, and are therefore also sent out faster. This could be useful if you want to inject messages that should be sent before emails already queued.
To turn a queue into a priority queue, the queue in RabbitMQ must have been declared with the "x-max-priority" property set. If you declare your own queues, you have to do this yourself. If you rely on MailerQ to declare the queues, you can specify the value of this property in the config file:
rabbitmq-maxpriority: 4The above config file option instructs MailerQ to set the "x-max-priority" property on the queues it creates. This setting is used during application startup when the queues are declared, and when new temporary queues are declared. If queues already exist, they must match the "x-max-priority" setting of the existing queues.
The reason MailerQ declares all queues to be priority queues is so that messages that are higher priority can be picked up sooner in every part of the chain, also its reports and its results. The same holds for messages that are injected, as they can also have a priority setting set, allowing for injected messages also to be picked up sooner.
In general, this number should be kept as small as possible, as there is a significant overhead for priority queues. For example, when a priority queue is created with an "x-max-priority" of 4, RabbitMQ internally creates 4 queues. As this happens for every queue MailerQ declares, priority queues with more than two or three priority levels are generally discouraged.
The "rabbitmq-exchange" variable holds the name of the RabbitMQ exchange
that MailerQ uses to publish all messages to. If not explicitly set, MailerQ
uses the default exchange with an empty name. For most setups, this empty default
exchange is good enough, because messages end up in the right queue anyway.
However, if you want to use a custom exchange, the "rabbitmq-exchange" setting allows you to do so. The exchange that you assign to this variable does not have to be created by yourself, because MailerQ automatically creates it on startup.
rabbitmq-exchange: your_exchangeTo understand why you would need a custom exchange, you need a litte more inside knowledge of RabbitMQ. Although we constantly write that MailerQ publishes message to specific queues, this is not exactly how RabbitMQ internally works. RabbitMQ does not allow messages to be published directly to message queues. Instead, it wants messages to be published to an exchange. The exchange then forwards the messages to zero or more queues that are bound to it based on the routing key associated with the message. Therefore, when we write that "MailerQ publishes a message to the inbox queue", we actually mean that "MailerQ publishes a message to the exchange, and uses the name of the inbox queue as the routing key so that the message ends up in the inbox queue." In the end, the result is the same.
This exchange-routingkey-queue architecture can be pretty powerful. It for example allows you to create your own (temporary) queue and bind it to the same exchange as the outbox queue so that a copy of each message that goes to the outbox queue ends up in your private queue too. This allows you to monitor exactly what is passing through RabbitMQ, without interfering with the message flow. Great for monitoring and debugging!
To use the flexibility that RabbitMQ offers, you can use the "rabbitmq-exchange" setting to assign a custom exchange. If you leave the setting empty however, the messages still end up in the right queues.
All queues and the exchange are automatically created when MailerQ starts. If you do not want MailerQ to declare the queues and exchanges, you can use the "rabbitmq-declare" setting in the config file. If you set this to false, no queues or exchanges are declared on startup.
rabbitmq-declare: falseDisabling the declaration of queues and exchanges is helpful if you have already created them yourself and you do not want MailerQ to do this for you. If you have created the queues with slightly different settings than MailerQ does, you even need this setting because otherwise MailerQ reports a queue incompatibility error and refuses to start.
In the MailerQ configuration you can specify whether you want queues to be durable and whether you want messages to be persistent using the following two options:
rabbitmq-durable: true (default: true)
rabbitmq-persistent: false (default: false)
rabbitmq-lazy: false (default: false)MailerQ creates several queues and exchanges in RabbitMQ. When MailerQ starts, it first checks if the queues and exchanges that you have configured in the MailerQ config file exist. If they do not, MailerQ sends instructions to RabbitMQ to create the required exchanges and queues. RabbitMQ allows you to mark your queues and exchanges to be "durable". This means that the exchange or queue will continue to exist even when RabbitMQ is restarted. In theory it is a slightly better to enable durable queues (but you probably won't notice much of a difference because the queues and exchanges are automatically re-created on startup anyway).
The "rabbitmq-persistent" setting toggles whether messages published to RabbitMQ should be cached in main memory or on disk. With the default "false" setting, RabbitMQ keeps messages only in main memory and not on disk. This is much, much faster than storing the messages to disk. It does however bring a higher risk, because if RabbitMQ crashes, the messages will be lost. Turning on persistency will mean your messages are also saved to disk, and can survive a RabbitMQ crash. But at the same time it makes things much slower. We therefore recommend leaving the "rabbitmq-persistent" option off (set to "false"). Note that if you do turn it on, durable queues should also be turned on. Persistent messages are still lost if the queue is not durable, because the queue no longer exists.
The "rabbitmq-lazy" setting toggles whether queues are declared as lazy queues. Lazy queues ensure that messages are always written to disk. This lowers overall performance, but prevents sudden performance drops once RabbitMQ needs to flush to disk, giving predictable performance. See the RabbitMQ documentation for more information on lazy queues.
MailerQ opens a number of different connections to RabbitMQ, and each connection runs in its own thread. There are separate threads for consuming from the inbox queue and threads for publishing messages to the result queues and/or back to the outbox queue.
You can specify in the config file that you want to start up multiple consumer and/or multiple publisher threads. If you notice that the consumer of publisher threads are CPU bound, you can configure MailerQ to start up more threads.
rabbitmq-consumers: 1 (default: 1)
rabbitmq-publishers: 1 (default: 1)By adding the "rabbitmq-consumers" and "rabbitmq-publishers" variables to the config file, you instruct MailerQ to start up more consumer and/or publisher threads.
The data stream between RabbitMQ and MailerQ can be large. It can even be that big that it takes up a significant portion of the capacity of your internal network. To reduce the load on the network, MailerQ supports gzip compression.
MailerQ normally expects messages from RabbitMQ to be JSON encoded. However, if the AMQP envelope in which the message is wrapped has the "content-encoding" property set to "gzip", MailerQ expects the message to be a gzip compressed JSON object instead. It first decompresses the message, then it parses the JSON. This means that you can publish both normal JSON encoded messages to RabbitMQ, as well as gzip compressed data. If you compress your input JSON, you do have to make sure that you also set the "content-encoding" header in the AMQP envelope that is published to RabbitMQ.
MailerQ not only consumes messages from RabbitMQ, it also publishes messages back to it, like the results that are sent to the result queues or the retries that are published back to the outbox queue. By default, MailerQ only sends pure JSON to RabbitMQ, without compressing it, even if you used compression yourself when you published the message to the outbox. If you want MailerQ to compress the data too, you can use a special setting in the config file.
rabbitmq-encoding: gzipThe above setting only affects how MailerQ publishes messages to RabbitMQ. Messages consumed from RabbitMQ can still both be compressed or not, because MailerQ always inspects the "content-encoding" header from the AMQP envelope.
When consuming from the outbox queue, or flushing a queue, MailerQ loads a certain number of messages at the same time. These numbers can be tweaked to lower memory usage.
rabbitmq-qos: 1000 (default: 1000)
rabbitmq-flush-qos: 1000 (default: 1000)This number is interpreted per consumer thread. Therefore, the maximum total number of messages from the outbox that are in memory at the same time is the amount of consumers multiplied by the maximum number of messages. For example, if "rabbitmq-consumers" is set to 5, and "rabbitmq-qos" is set to 200, the maximum total number of unacknowledged messages from the outbox queue is 1000.
The "rabbitmq-flush-qos" can be set to lower the loaded messages when a flush happens. Although MailerQ doesn't load in the messages, their bodies will be sent over TCP and so the TCP buffer may be fully populated with all this data. As this can happen for anything that is unpaused or resumes sending, a lot may be flushed at once.
In the end, MailerQ fetches all messages that it is going to send directly from a RabbitMQ outbox queue. The fastest and most efficient way to inject emails is therefore to bypass SMTP, spool directories or the command line interface, and publish messages directly to RabbitMQ.
If you want to publish messages directly to RabbitMQ, you need a client library to communicate with RabbitMQ. On the RabbitMQ website you can find many plugins and libraries for this.
Besides being faster, publishing messages directly into the outbox queue has the advantage that you can set all sorts of special properties in the JSON objects that control the delivery.
The JSON encoded messages that you add to the outbox message queue should contain a minimum of two properties that hold the recipient's email address, and the full MIME message. All other properties are optional.
{
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\n
To: info@example.org\r\n
Subject:Example subject\r\n\r\n
This is the example message text"
}If you're interested in bounces and out-of-office replies, we recommend to also include an "envelope" property, because that's the address where bounces will be sent to.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\n
To: info@example.org\r\n
Subject:Example subject\r\n\r\n
This is the example message text"
}Note that for ease of reading we added some spaces to the message mimes in the above examples. The above examples are very minimalistic, check the JSON message specification for a full list of all supported properties.
It is possible to compress the data that you send to MailerQ. The JSON message that you send to RabbitMQ can be gzip compressed. If you do this, you also have to add a "content-encoding" header to the AMQP envelope in which the message is wrapped, and set it to the value "gzip". Check the documentation of your AMQP library to find out how to set properties in the AMQP envelope.
If the default behavior of MailerQ is not up to your standards, you can extend MailerQ with your own plugins. MailerQ allows you to develop plugins to add features or functionality to MailerQ. A plugin is a shared object file (like "example.so"), and normally developed in C or C++. The plugins are loaded by MailerQ when the application starts. Every plugin can implement a number of functions that are called by MailerQ to hand control over to the plugin.
In the global configuration file, you can set a "plugin-directory" variable. This variable should hold the name of the directory where MailerQ can find your plugins. The plugins are loaded in alphabetical order. If multiple plugins are loaded, MailerQ will make the calls to the plugins in this same alphabetical order. If the order in which your plugins are called matters, you can prefix your plugin name with a number, like "10-example.so" and "20-anotherexample.so". By doing this, you ensure that the "example" plugin will always be called before the "anotherexample" plugin.
MailerQ makes the calls to your plugin using the C calling convention. As a
result, it is best and easiest to use C or C++ for developing your plugin. When
developing in C++, be sure to extern "C" {} the function which will be called
by MailerQ, to prohibit the compiler from function name mangling.
Plugins are called often -- possibly multiple times for every message. You should be aware of this and write your plugins with care. Blocking IO calls are for example absolutely a no-go as they will massively bring down MailerQ's performance.
To find out how MailerQ hands over control to your plugin, how your plugin can then interact with MailerQ's event loop, and finally give back control to MailerQ, please read our event loop article.
The "mq_version()" function is currently the only function that your plugin is required to implement. All other callbacks are optional. A very basic plugin (one that does nothing) would look like this:
#include <mailerq.h>
/**
* Function called by MailerQ to find out if the plugin is compatible
* with the MailerQ executable
*
* @return int API version number for which the plugin is compiled
*/
MQ_EXPORT int mq_version() {
return MQ_API_VERSION;
}
/**
* Function that is called by MailerQ to initialize the plugin. This
* function is called right after the plugin is loaded.
*
* This is an optional function, you do not have to implement it. MailerQ
* only calls it if it exists.
*/
MQ_EXPORT void mq_initialize() {
// @todo add your own initialization code
}
/**
* Function that is called right before MailerQ stops running.
*
* This is an optional function, you do not have to implement it. MailerQ
* only calls it if it exists.
*/
MQ_EXPORT void mq_cleanup() {
// @todo add your own cleanup code
}The "MQ_EXPORT" directive is a C #define that is turned by the preprocessor in
a statement to ensure that the symbol is correctly exported. If you compile your
extension with gcc, it is for example turned into
__attribute__ ((visibility("default"))).
When compiling your extension, we recommend using hidden visibility for all functions not to be made available externally (all functions that MailerQ won't call). Doing this gives the compiler much more freedom to inline functions (which it cannot do if a function is to be made available externally) and helps keep the DSO small.
For our SMTPeter cloud service we use MailerQ for handling incoming messages via SMTP. For every incoming connection we check the SMTP login credentials, and compare them with the login data that we have on file. Only recognized users are given access.
We use a MailerQ login to handle these incoming login requests. A simplified version of this plugin is available as example to see how such a plugin can be built.
If you have the idea that vital functionality is missing in the plugin API, or if you need other callbacks you can always get in contact with us. We can accomodate your needs, extend the API and implement your request!
To integrate MailerQ into your PHP environment, you can either write PHP scripts that inject emails directly into RabbitMQ (using the AMQP protocol), send mails to MailerQ over SMTP, or change the php.ini config file so that the builtin PHP "mail()" function automatically injects mail into MailerQ.
The PHP mail() function internally uses a command line utility (by default "sendmail") to send out the mail. Every time that you make a call to this mail() function, the PHP engine starts up this command line utility and sends out the message. By modifying the "php.ini" file you can change the command line utility into "mailerq" so that all mails sent from PHP are automatically delivered to MailerQ.
To use MailerQ for sending emails in PHP using mail() function, set the "sendmail_path" property in the php.ini config file to:
sendmail_path = mailerq --envelope my-sender-address@my-domain.com --extract-recipients --ignore-dot
Despite the simplicity of the above solution, it is more efficient to inject mails directly into RabbitMQ. In the following example we will demonstrate you how to access RabbitMQ from PHP using the AMQP protocol. This example assumes that MailerQ is running and configured and that the PECL AMQP package extension is installed.
Read more about the AMQP protocol
The example consists of three files; settings.php, send.php and result.php.
If you want to use these scripts, save the three files inside the same folder.
The settings.php file defines the settings that are needed to connect to
RabbitMQ and it also contains the settings that are being used for sending
an email. If you want to use the PHP example to test your MailerQ configuration,
you will have to adjust the values inside the settings.php file to match your
configuration of MailerQ.
<?php
/**
* This file holds the settings that the PHP test script uses to connect
* to RabbitMQ and to construct the message that will be placed on the
* outbox message queue.
*/
$address = 'amqp://guest:guest@localhost'; // same as the rabbitmq-address option in the config file
$outbox = 'outbox_test'; // same as the rabbitmq-outbox option in the config file
$resultbox = 'results_test'; // same as the rabbitmq-results option in the config file
// email message settings
$recipientDomain = 'example.org'; // domain where the test message should be delivered
$recipientEmail = 'info@example.org'; // email where the test message should be delivered
$fromAddress = 'me@my-domain.com'; // address where the email was sent from
?>
This PHP script connects to RabbitMQ and places a JSON encoded message on
the outbox message queue. To connect to RabbitMQ and to create the JSON encoded
message, the script uses the values from the settings.php file. If you want
to use this script to test your MailerQ configuration, you will have to run
the send.php script before running the result.php script.
<?php
/**
* send.php
* Script that connects to RabbitMQ, constructs a json encoded message
* and puts that message on the outbox message queue.
*/
// include the settings
require_once('settings.php');
// try to set up a connection to the RabbitMQ server / try to log in
try
{
// construct the connection to the RabbitMQ server
// could add vhost, port as well
$connection = new AMQPConnection(array(
'host' => $hostname,
'login' => $username,
'password' => $password
));
// connect to the RabbitMQ server
$connection->connect();
}
catch (AMQPException $exception)
{
echo "Could not establish a connection to the RabbitMQ server.\n";
}
// try to create a channel for sending instructions
try
{
// create the channel
$channel = new AMQPChannel($connection);
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (creating channel).\n";
}
// try to declare the exchange, which will function as an outbox were
// rabbitmq will find the messages that need to be sent
try
{
// declare the exchange
$exchange = new AMQPExchange($channel);
$exchange->setName('exchange1');
// this type sends to all queues connected to this exchange
$exchange->setType('fanout');
$exchange->declareExchange();
}
catch (AMQPExchangeException $exception)
{
echo "Channel is not connected to a broker (declaring exchange).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (declaring exchange).\n";
}
// create the queue mailerq will get its messages from,
// which will get those from the exchange
try
{
// create the queue
$queue = new AMQPQueue($channel);
$queue->setName($outbox);
// queue survives rabbitmq restart
$queue->setFlags(AMQP_DURABLE);
$queue->declareQueue();
// route from exchange1 to this queue;
// the routing key is optional and what it does depends on the
// exchange type. fanout ignores it
$queue->bind('exchange1','key2');
}
catch (AMQPQueueException $exception)
{
echo "Channel is not connected to a broker (creating queue).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost. (creating queue)/\n";
}
// JSON encoded message to put on the queue
$jsonMessage = json_encode(array(
'envelope' => $fromAddress,
'recipient' => $recipientEmail,
'mime' => "From: " . $fromAddress . "\r\n"
. "To: " . $recipientEmail . "\r\n"
. "Subject: Example subject\r\n\r\n"
. "This is the example message text"
));
// try to publish the message on the queue. fanout sends to all, key1 is used as filter
try
{
$message = $exchange->publish($jsonMessage, 'key2');
}
catch (AMQPExchangeException $exception)
{
echo "Channel is not connected to a broker (publishing message on queue).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (publishing message on queue).\n";
}
catch (AMQPChannelException $exception)
{
echo "The channel is not open (publishing message on queue).\n";
}
// done, close the connection to RabbitMQ
$connection->disconnect();
?>
This PHP script connects to the RabbitMQ server, and gets the message that was
placed on the outbox message queue by the send.php script back from the result
message queue. The result message from the result queue is shown to the user.
The result.php can only output any relevant information when it is executed after
there is something to show in the results queue, for example by executing the send.php script.
<?php
/**
* result.php
* Script that connects to RabbitMQ, and takes all messages from
* the result message queue. This could be useful when you want to
* parse messages from a custom queue and later re-post them, for example
* to the outbox queue.
*/
// include the settings
require_once('settings.php');
// try to set up a connection to the RabbitMQ server
try
{
// construct the connection to the RabbitMQ server
$connection = new AMQPConnection(array(
'host' => $hostname,
'login' => $username,
'password' => $password,
'vhost' => $vhost
));
// connect to the RabbitMQ server
$connection->connect();
}
catch (AMQPException $exception)
{
echo "Could not establish a connection to the RabbitMQ server.\n";
}
// try to create the channel
try
{
// open the channel
$channel = new AMQPChannel($connection);
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (creating channel).\n";
}
// try to create the queue
try
{
// create the queue and bind the exchange
$queue = new AMQPQueue($channel);
$queue->setName($resultbox);
$queue->setFlags(AMQP_DURABLE);
// again with the optional routing key
$queue->bind('exchange1', 'key2');
$queue->declareQueue();
}
catch (AMQPQueueException $exception)
{
echo "Channel is not connected to a broker (creating queue).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost. (creating queue)/\n";
}
// Get all messages from the queue.
while ($envelope = $queue->get()) {
echo "Received:\n";
echo $envelope->getBody() . "\n";
echo "----------------------------------------------\n\n";
}
// done, close the connection to RabbitMQ
$connection->disconnect();
?>
MailerQ can personalize email messages on the fly. If you add an extra "data" property to the input JSON, MailerQ treats the subject line and the text and HTML versions of your mime messages as templates, and replaces all variables in them with the values from the JSON "data" property. You can use these values in the "from" and "to" address too.
{
"recipient": "john@example.com",
"mime": "....",
"data": {
"ourname": "The MailerQ test team",
"name": "John Doe",
"age": 33,
"job": "programmer",
"children": [
"Peter", "Angela", "Brandon"
]
}
}If you use the above JSON data for your mail, you can use these variables inside the "from" and "to" address, the subject line, and inside the text and HTML versions of your email.
From: {$ourname} <info@example.org>
To: {$name} <{$recipient}>
Subject: Hello {$name}!
Content-Type: text/plain
Hi {$name},
Your age is {$age}, and your job is {$job}.
Cheers,
{$ourname}If you had used the above MIME as input, MailerQ would replace the variables in the "from", and "to" address, the subject line and the text and HTML versions. For ease of use the "envelope" and "recipient" are already extracted from the mail for you. In case you don't want to personalize at all because your body has to contain {$name} for example, you can disable personalizing by omitting the data property entirely.
Read our programming page if you want to know more about how to use the personalization capabilities.
This page gives a syntax overview for writing a template that can be used to personalize your mail. Note that although the syntax is quite easy, you can build complicated concepts, which brings the danger of bugs (heck, even simple stuff can have bugs).
Let's just start with the basics of printing a user-supplied variable. A variable
looks like {$name}, {$city} or {$country}: it starts with a {$, followed by
the variable name and is closed with a }. Formally, a variable in personalization:
_);{$NAME} is different from {$name}.The next table gives all variable notations:
| Syntax | Meaning |
|---|---|
{$foo} |
Display a simple variable (no array or object). |
{$foo[4]} |
Display the 5th element of a zero-indexed array. |
{$foo.bar} |
Display the "bar" key value of an object. |
With these notations you may make combinations. Examples of combinations are:
| Syntax | Meaning |
|---|---|
{$foo.bar.baz} |
Display the value behind the key "baz" inside the array "bar" which is a part of $foo. |
{$foo[4].baz} |
Display the value behind the key "baz" inside the 5th element of $foo. |
{$foo.bar.baz[4]} |
Display the 5th element of baz, which is in bar which is in $foo. |
If a variable happens to be an object where the index are keys, you can still access elements in the array with an index number (starting from 0).
While you have to use underscores (_) in your templates when using
variable names, you must supply these in the JSON with dashes (-).
These are converted automatically. Example: When using {$total_price}
in your template, you have to supply this in JSON with "data": { "total-price": 10 }.
Dashes in the template will be interpreted as the subtract operator. Variables with underscores in the JSON won't be accessible in the templates.
You can alter the content of your variables by applying modifiers to them. E.g. you can capitalize the content, calculate the length of a string, or calculating a hash sum. A list of all modifiers and an explanation about their usage can be found here.
Variables containing a numerical value can be used to do some simple math. Just like with normal variables, all math should be done within the {} brackets. So you can for example do the following:
{$var + 10}Besides the surrounding of curly braces, all standard math rules apply.
The standard math operators (+, -, *, /) and the modulo operator (%)
are available. Note that if a value does not exist or does not contain a
numeric value, it will behave as the value zero.
One of the key concepts of any programming language are conditional statements.
A conditional block always starts with the {if} keyword (always with the
curly braces) followed by the statement that is tested. A conditional
block always ends with the if closing tag {/if}. A conditional block is only
executed if the statement in the {if} part is true.
In the next example, the text 'Hello John' is only displayed when the value
of the variable $name is equal to 'john'.
{if $name == 'john'}Hello John{/if}But what if there's also a Sarah in your mailing list. You wouldn't want
to display nothing to her, would you? That's where the {elseif} becomes
the protagonist.
{if $name == 'john'}Hello John{elseif $name == 'sarah'}Hello Sarah{/if}Now, if we want to say something to anyone except to John and Sarah, we will use the {else} keyword. The code after the {else} keyword will be executed if none of the preceeding statements returned true.
{if $name == 'john'}
Hello John
{elseif $name == 'sarah'}
Hello Sarah
{else}
Hello anybody else
{/if}This is of course a very bad example of how you should write your personal salutation because your conditional block would need to be as long as the list of all names in the world.
A much better example would obviously be:
{if $name == ''}
Dear subscriber,
{else}
Dear {$name},
{/if}The code snippet above means: if the value of name is empty, show 'Dear subscriber', otherwise, show Dear John (or Sarah, or what is in the {$name} variable).
In the preceeding example, the operator == was used. This operator means 'is equal to'.
It will evaluate to true if the left hand side of the == is exactly equal
to the right hand side. Operator == is just one of the many operators that
you can use to test the statements.
Here's the complete list:
| Syntax example | Meaning |
|---|---|
| $a == $b | $a and $b are equal |
| $a != $b | $a and $b are not equal |
| $a > $b | $a is greater than $b |
| $a >= $b | $a is greater than or equal to $b |
| $a < $b | $a is less than $b |
| $a <= $b | $a is less than or equal to $b |
| $a =~ "regex" | $a matches with supplied regular expression |
| $a AND $b | $a and $b are true |
| $a OR $b | $a or $b (or both) are true |
| not $a | Negation, will invert the boolean value of $a |
With the first six operators ("==" to "<=") you compare the values of variables $a and $b. If the comparison holds, a true will be returned. If the comparison does not hold, you will get a false. The regular expression operator is special and powerful, and allows you to check if the variable matches a certain pattern.
With the next two operators in the table above (AND and OR) you can combine
statements. If you are writing conditional blocks, you will sometimes find
yourself ending up with blocks that become too long or complex. To shorten
long or more complex conditional blocks, you can put multiple statements in
one single {if} block using the AND and/or OR operators. The AND and/or OR
operators combine the value of $a and $b into a true or a false.
E.g. you can use AND like:
{if $a >= $b AND $b <= $c}true{else}false{/if}In the above snippet first $a >= $b will be tested. Only if this is true,
$b <= $c will be tested. If this happens to be true as well, true will
be printed, else, false will be printed.
Subsequently there is the not operator. This operator will invert the value
that is given. So true becomes false and false becomes true.
Finally it is good to know that a variable itself will be converted to a
boolean when used in an if statement. If the variable contains an empty string
or the value 0, it will be evaluated as false. If it is not an empty string
and it isn't zero, it will be evaluated as true.
If you have a collection of data (an array), and want to see if a specific something is inside that collection, you'll have to loop through all the items in this collection.
Let's use a real world example and display all members of a soccer team,
stored in an array called $soccerTeam.
To get the members of the team, the foreach statement is used. Its syntax is fairly straightforward:
{foreach $player in $soccerTeam}
{$player.name}
{/foreach}This will loop over the items (team members) in $soccerTeam and assign
each player to the variable $player in each iteration. Inside this foreach
block you can do whatever you want with the outputted information. You can
for example generate an HTML list with soccer players from that team.
It becomes a little bit more technical now...
Of course we also support looping over arrays that have non-standard keys.
{foreach $list as $key => $value}
Key {$key} contains {$value}.
{/foreach}As you can see here, you specify both a $key and a $value which will be used to store these values on each iteration.
And finally, sometimes you want certain code to execute if there is no data at all, this is done using the {foreachelse} statement.
{foreach $item in $list}
{$item.name}
{foreachelse}
No items in list.
{/foreach}This foreachelse statement is only executed in case of no data. It is completely ignored otherwise.
It is possible to assign values on runtime. You can for example use this to calculate the total price of a set of purchased items. Or to remember a certain item inside a foreach statement. Assigning variables is done as follows:
{assign $item to $topitem}After this statement the variable $topitem is available and it will contain what $item contained when the email was being compiled and sent to the user.
This will allow you to do things like the following.
{foreach $item in $list}
{assign $total + $item.price to $total}
{if $item.price > $topitem.price}
{assign $item to $topitem}
{/if}
{/foreach}Which will eventually have the most expensive item in the $topitem variable
and the total price in $total.
As you can see above the syntax heavily depends on the curly braces {}.
Therefore, these braces cannot appear in the text without leading to a
valid syntax. If you do need these braces in you text, you can use: {ldelim}
to get a { and {rdelim} to get a }. If you have a large text with
these braces you can enclose the text in {literal} and {/literal}.
This text will then not be processed.
The code snippets above are formatted in a way to make them readable: we used newlines, indentation, etc. When you are generating text for the html part of your mail this is advisable since it is easier to capture mistakes. However, if you are generating content for the text part of your mail, you should be aware that the formatting of your template code affects the format of your text.
E.g. above we gave a snippet that prints out names of players in a soccer team
If the soccerTeam array looks like: ["Ronaldo", "Messi", "Ibrahimovic"]
The output that we get in the text section will be:
Ronaldo
Messi
Ibrahimovic
This is probably not what you had in mind while typing the foreach loop. But this is what you get since there is a new line before and after the variable and there are some spaces in front of it used as indentation. To get a list of the names without the extra whitespace you should write:
{foreach $player in $soccerTeam}{$player}
{/foreach}This is less readable but gives what you want
Ronaldo
Messi
IbrahimovicThe variables used for personalization can be altered with modifiers. A modifier
is called on the variable by appending it with a | before the closing curly
brace. E.g. if you want to apply the tolower modifier on
variable {$name} you use: {$name|tolower}.
An example of how the modifiers can be used for a personalized mail is:
Hello {$name|escape},
Your name {$name|escape} is {$name|strlen} characters long.
Bye!Besides calling one modifier on a variable it is possible to chain modifiers.
E.g. you have name data. However, the strings containing names, are sometimes
capitalized and sometimes not. You want to use these names and you want to
be sure that a name starts with a capital and the rest of the name is in
lower case. Doing this is easy. You use {$name|tolower|ucfirst}. The first
modifier tolower, will make sure that the name is all in lower case. Then
modifier ucfirst is applied, which will capitalize the first character.
Note that if you call a modifier that does not exist, no effect takes place.
The following table lists all supported modifiers:
| Modifier | description |
|---|---|
| base64_encode | base64 encoder |
| base64_decode | base64 decoder |
| cat:"string" | concatenates a string to variable |
| count | count number of elements in variable |
| count_characters | count number of characters in a string |
| count_paragraphs | count number of paragraphs in a text (by counting newlines) |
| count_words | count number of words in a text |
| default:default value | use default value if variable is not set |
| empty | check whether a variable is empty |
| escape:"string" | escape html characters (or other chars) inside a string |
| indent:num = 1:char = " " | put num whitespaces in front of every line |
| md5 | perform md5 hashing |
| nl2br | replace newlines with html br tags |
| range:start = 0:end | truncate list to get the items between positions start and end |
| regex_replace:regex:replace_text | replace substrings using regular expression |
| replace:"string1":"string2" | replace occurrences of string1 with string2 |
| sha1 | perform sha1 hashing |
| sha256 | perform sha256 hashing |
| sha512 | sha512 hashing |
| spacify:separator = " " | place a separator between every input character |
| strlen | count the characters in a string |
| strstr:"substring":before = false | return the string starting from the first occurrence of substring if before = false. otherwise return the string until the first occurrence. |
| substr:start position:length | return the substring from start position onward, optionally truncated after length characters |
| tolower | convert all characters to lower case |
| toupper | convert all characters to upper case |
| trim | trim the white space and endline characters off both sides of the input |
| truncate:length = 80:etc = "...":break_words = false | truncate inputs that are longer than length and append etc at the end. break_words = true allows truncating parts of words |
| ucfirst | replace first character with an upper case character |
| urlencode | encode input for use in an url |
| urldecode | decode input for use in an url |
With this modifier you encode the data into base64. Note that this modifier does not have an effect on arrays. Usage:
The base64 encoding of {$name} is {$name|base64_encode}.With this modifier you can decode base64 encoded information. Usage:
The decoded information is {$base64encoded|base64_decode}With this modefier you can concatenate a string to your variable. If the variable is an arrary, the string will be used. Usage:
{$name|cat:"string"}With this modifier you can count the number of elements in an array. If the variable is not an array a 0 will be returned. Usage:
{$names|count}With this modifier you can count the number of characters in your text. If this modifier is called on an variable that contains an array, 0 is returned. Usage:
{$name|count_characters}With this modifier you can count the number of paragraphs in your text. If this modifier is applied on an array a 0 is returned. Usage:
The following text has {$text|count_paragraphs} paragraph.
Text:
{$text}With this modifier you can count the number of words in your text. If this modifier is applied on an array a 0 is returned. Usage:
"{$text}" has {$text|count_words} words.With this modifier you can set a default value that will be used if the value does not exist. Usage:
This will always show {$name|default:"something"}With this modifier you can check if the variable is set or not. It will return true if the variable is set or false if it isn't. Usage:
{if $name|empty}
Dear customer,
{else}
Dear {$name},
{/if}With this modifier you can escape, or encode, the variable. The form is: escape:"value". The possible values are: "html", "url", and "base64". If no value is specified "html" is used. On an array this modifier is ingored. Usage:
{$text|escape:"html"}
is equal to:
{$text|escape}With this modifier you can add indentation to your text. You can specify the amount of indentation and the character(s) that is used for indentation. The syntax is indent:num:char, where "num" is the number of characters and "char" the character. The default of these is 1 and space respectively. On an array this modifier is ignored. Usage
{$text|indent:4:" "}With this modifier you get the MD5 checksum of your text. If you use it on an array, the MD checksum is calculated over the entire array excluding the keys. Usage:
{$text|md5}This modifier replaces your newlines with the equivalent HTML br tags. This enables you to write plain text that will be truncated in html mode. On an array this value is ignored. Usage:
{$text|nl2br}With this modifier you can take a range form your input if your input is an array. The form is range:start:end, where start is the start position and end is the exclusive end position. If the variable is not an array, the value will be ignored. Usage:
{$array|range:2:5}With this modifier you can replace parts of your text, base on a regular expression, with other text. If the variable is an array this value will be ignored. Usage
This will replace each number in the variable string with the string " a number "
{$text|regex_replace:"\d":" a number "}With modifier replace you can replace a part of your text with some other text. The syntax is: replace:"string1":"string2", where all occurrences of "string1" will be replaced by "string2". When the variable is an array, this modifier does not have an effect. Usage:
{$text|replace:"hi":"hello"}With this modifier you get the SHA1 hash of your text. If the variable is an array the SHA1 hash will be calculated over the entire array, excluding the keys. Usage:
{$text|sha1}With this modifier you get the SHA256 hash of your text. If the variable is an array the SHA256 hash will be calculated over the entire array, excluding the keys. Usage:
{$text|sha256}With this modifier you get the SHA512 hash of your text. If the variable is an array the SHA512 hash will be calculated over the entire array, excluding the keys. Usage:
{$text|sha512}With this modifier you can add a character, or characters, between each character in your variable. The syntax is spacify:separator, where the default separator is space. If the variable is an array the modifier will be ignored. Usage:
{$text|spacify:"."}With this modifier the length of the text in a variable can be obtained. If the variable is an array a 0 will be returned. Usage:
{$text|strlen}With this modifier you can search for a string in you variable and get the first occurrence plus the rest of your variable, or you can get everything in the variable till the searched string. The syntax is: strstr:string:before, where string is the searched string, and before is a boolean, default = false. If the boolean is true, you get what is in the variable before the first occurrence of string. If it is false you get what is in the variable from the first occurrence of string onwards. If the variable is an array the modifier is ignored. Usage:
If the variable holds "Hello world!", this will print Hello, {$variable|strstr:" ":true}
and this will print world!, {$variable|strstr:"w":false}, just like this
{$variable|strstr:"w"}With this modifier you can obtain a substring from you variable. The syntax is substr:start:length, where start is the starting position in the string (zero indexed) and length is the length you want to obtain. The default for length is the the length of the string minus the starting position. Usage:
If the variable is "0123456789" this will print 2 to 9
{$variable|substr:2}
and this will print 456
{$variable|substr:4:3}With this modifier you can change all character in your text to lowercase characters. Usage:
{$text|tolower}With this modifier you can change all characters in your text to uppercase characters. Usage:
The next part looks like it is shouted {$text|toupper}With this modifier you can trim trailing white space and new line characters from your text. The characters that are trimmed are: spaces, tabs, newlines, carriage returns, vertical tabs, and end of strings. Usage:
{$text}With this modifier it is possible to truncate your text to a certain length. If the text is truncated it is possible to add some characters to indicate that the text is truncated. These extra characters are counted for the truncation length as well. You can also specify whether it is allowed to break words or not. The syntax is: truncate:length:etc:break_words, where length has a default of 80, etc (the replacement), has a default of ... and break_words has as default false.
Usage:
{$text|truncate:50:"....":true}With this modifier you replace the first character of your text with an uppercase character. Usage:
{$name|ucfirst}With this modifier you can url encode your text. Usage:
{$text|urlencode}With this modifier you can url decode your text. Usage:
{$text|urldecode}MailerQ can be configured using the "/etc/mailerq/config.txt" config file. Most of the variables and settings in this config file are discussed in separate articles. However, the following settings are not mentioned anywhere else:
max-delivertime: 86400
max-attempts: 0
retry-interval: 600,600,1800,1800,3600,3600,7200
server-id: 1
license: /path/to/license.txt
user: username
lock: /path/to/lock.txtMost of the settings speak for themselves. A small overview.
When a mail gets greylisted, or when there is an other type of soft failure, MailerQ reschedules the mail for later delivery. Inside the JSON data of each message you can add the optional properties "maxdelivertime" and "maxattempts" that instruct MailerQ how long it should retry to deliver the message, and when it should give up trying.
If you do not include these properties, MailerQ falls back to the "max-delivertime" and "max-attempts" config file settings. The "max-delivertime" variable holds the number of seconds MailerQ should keep on trying after the message was consumed for the first time. The "max-attempts" setting holds the total number of attempts before giving up.
You should normally set the "max-attempts" value to 0, meaning unlimited. MailerQ will then only give up after the number of seconds set in the "max-delivertime" variable, no matter how many attempts are made.
If a mail is published back to the outbox queue to be tried again, MailerQ uses the "retries" property in the JSON to find out how long it should wait for the next attempt. Because this setting is JSON based, it is possible for each message to have its own retry interval. This could be useful if you send both transactional as well as commercial mass mailings. For transactional emails you normally want small intervals because these mails should be delivered as soon as possible.
If the "retries" property is not set in the input JSON, MailerQ falls back to the setting from the config file. The config file variable "retry-interval" is used for that. This setting should hold a comma separated list of the delays between the attempts. This should be set to a number of seconds.
The default value is "600,600,1800,1800,3600,3600,7200". This means that the second and third attempt is sent 10 minutes after the previous attempt. The next two attemps use an interval of half an hour, then an interval of one hour for the next two attempts, and all subsequent attempts are sent after three hour intervals. You can change variable to anything you want.
By default, any injection method causes MailerQ to strictly check whether or not an email address is actually fully RFC compliant, and an error will occur if this is not the case. On SMTP injection, the message will simply be rejected, but on AMQP injection, the message will not be tried.
This can be loosened by the following properties.
validate-envelope: loose (default: strict)
validate-recipient: loose (default: strict)Setting these to loose causes the address to be interpreted as loosely as possible,
only requiring the @ sign in the middle and a valid domain on the right. The local-part
may be anything, and will not be restricted by the RFC any more. Note that although
attempts will be made, the other server might reject the message anyway, since it might
be in violation with the standard.
Each message that is accepted by MailerQ gets a unique message ID. To prevent that two running MailerQ instances both assign exactly the same ID to a message, you can assign a "server-id" variable. This identifier guarantees that message ID's will never conflict.
server-id: 1To work properly, MailerQ needs a license file. The license file can be downloaded from the MailerQ website. You can store the file anywhere on the file system. The path to the license file can be configured by setting
license: <Path to your license> (empty by default)If you have configured MailerQ to use ports lower than 1024 (like port 25 for SMTP and/or port 80 for the management console), the MTA must be started as user "root". Once the ports have been opened, MailerQ changes its identity to the user set in the config file.
user: <user name> (empty by default)The username to change identity to after the SMTP and HTTP ports have been opened.
To set a proxy for outgoing HTTP requests, the "proxy-server" variable can be used.
proxy-server: protocol://hostname:port (empty by default)Currently, two proxy types are supported, HTTP and SOCKS. For example, to route all HTTP
requests through an http proxy, the required uri would be http://example.org:8888, for a
proxy running on the host example.org on port 8888. For the SOCKS proxy, the URI would be
socks://example.org:8888 for a SOCKS proxy running on port 8888.
For the HTTP proxy, the proxy itself should support the CONNECT request. The HTTP proxy will be used as a tunnel, so this should be the case for both secure and insecure requests. The DNS gets resolved by MailerQ, so the final outgoing request will be similar to the request below.
CONNECT 1.2.3.4:80 HTTP/1.1To prevent that MailerQ starts more than once, MailerQ stores its process ID (pid) in a lockfile. The name and location of the lockfile can be set in the configuration file.
lock: <filename> (default: /tmp/mailerq.pid)MailerQ sends a message to Copernica hourly. This message contains some usage statistics
for that hour, e.g. the number of errors that occured. This information is used
to improve MailerQ. Users with a paid license can disable this message by adding
the heartbeat property to the config file:
heartbeat: false (default: true)You can run multiple MailerQ instances on a single server. But it does require some extra work.
If you run multiple instances on the same machine, these instances will still all load their configuration from the central /etc/mailerq/config.txt config file. This is no problem for most of the settings, but some specific settings need to different for each instance. For example, the different instances all need a seperate lock file, and have to use different port numbers to listen to.
If you want to run multiple instances at the same time, you still need to create a central /etc/mailerq/config.txt file with MailerQ's configuration. This file contain the settings that apply to all instances. But besides this central config file, you also need additional config files (for example /etc/mailerq/instance-1.txt, /etc/mailerq/instance-2.txt, et cetera) for the individual instances. In these config file you put the instance-specific settings.
If you start a MailerQ instance, you have to use the --config-file command
line option to specify the configuration file to use:
mailerq --config-file /etc/mailerq/instance-2.txtWith the above command you start a MailerQ process that loads both the normal /etc/mailerq/config.txt file, as well as the additional /etc/mailerq/instance-2.txt file.
When MailerQ starts, the settings are loaded in the following order:
1. Hardcoded default settings
2. Default config file (/etc/mailerq/config.txt)
3. Config file set in --config-file argument (optional)
4. Command line arguments (optional)To simplify the configuration files, you can use the default config file for all the shared settings and instance specific files with just the unique options.
There are a couple of settings that almost always must be unique if you run more than once instance:
The outbox queue must be unique per instance to manage which messages are sent by which MailerQ instance. Port number are limited resources and it is normally not possible to have multiple processes that all listen to the same port number. These settings have to be unique too:
rabbitmq-outbox: outbox1
smtp-ip: 10.0.0.1
www-ip: 10.0.0.1
www-host: mta001.example.comTo avoid corruption (you don't want that different processes write to the same log file at the same time) it is advised to use separate log files or log directories for the instances. If you choose to use different directories, make sure to create then and change ownership to the MailerQ user.
This is an example if you use different directories:
error-log: /var/log/mailerq/instance1/errors.log
send-log-directory: /var/log/mailerq/instance1/
download-log-directory: /var/log/mailerq/instance1/
received-log-directory: /var/log/mailerq/instance1/This is an example if you use different prefixes:
error-log: /var/log/mailerq/instance1-errors.log
send-log-prefix: instance1-attemtps.log
download-log-prefix: instance1-downloads.log
received-log-prefix: instance1-received.logTo prevent a MailerQ instance from starting more than once, MailerQ stores its process ID (pid) in a lockfile. This must be unique for every instance:
lock: /tmp/mailerq-instance1.pidEach message that is accepted by MailerQ gets a unique message ID. To prevent that multiple instances assign exactly the same ID to a message, you can assign a "server-id" variable. This identifier guarantees that message ID's never conflict.
server-id: 1We recommend to enable the clustering when running multiple MailerQ instances. Because when changes are applied in one web interface, they are also immediately applied to all instances in a cluster. To enable clustering set the following options in the default configuration file:
cluster-address: amqp://user:password@rabbitmq.example.com/
cluster-exchange: clusterIf you like to use init scripts to start multiple MailerQ instances, you can copy the default init script:
$ sudo cp /etc/init.d/mailerq /etc/init.d/mailerq-instance1and alter the following settings inside the copied file (/etc/init.d/mailerq-instance1):
NAME="mailerq-instance1"
DESC="MailerQ daemon instance1"
PIDFILE="/tmp/mailerq-instance1.pid"
DAEMON="/usr/bin/mailerq --config-file /etc/mailerq/instance1.txt"Now you can start your instances from the terminal like:
$ sudo service mailerq-instance1 startor
$ sudo systemctl start mailerq-instance1To keep instances running you can use a process supervisor like Monit. With our previous example, a simple Monit script for MailerQ instance looks like:
check process mailerq with pidfile /tmp/mailerq-instance1.pid
group mail
start program = "/etc/init.d/mailerq-instance1 start"
stop program = "/etc/init.d/mailerq-instance1 stop"
if failed host 10.0.0.1 port 25 protocol smtp then restartAlternatively you can use cron to restart your MailerQ instances if they are not running. This can be set in the crontab for either the root or MailerQ user.
* * * * * kill -0 $(cat /tmp/mailerq-instance1.pid) || /usr/bin/mailerq --config-file /etc/mailerq/instance1.txt &
* * * * * kill -0 $(cat /tmp/mailerq-instance2.pid) || /usr/bin/mailerq --config-file /etc/mailerq/instance2.txt &An example of two configuration files for different instances:
/etc/mailerq/instance1.txt:
rabbitmq-outbox: outbox1
rabbitmq-inbox: outbox1
smtp-ip: 10.0.0.1
www-ip: 10.0.0.1
www-host: mta001.example.com
error-log: /var/log/mailerq/instance1-errors.log
send-log-prefix: instance1-attempts.log
download-log-prefix: instance1-downloads.log
received-log-prefix: instance1-received.log
lock: /tmp/mailerq-instance1.pid
server-id: 1/etc/mailerq/instance2.txt:
rabbitmq-outbox: outbox2
rabbitmq-inbox: outbox2
smtp-ip: 10.0.0.2
www-ip: 10.0.0.2
www-host: mta002.example.com
error-log: /var/log/mailerq/instance2-errors.log
send-log-prefix: instance2-attempts.log
download-log-prefix: instance2-downloads.log
received-log-prefix: instance2-received.log
lock: /tmp/mailerq-instance2.pid
server-id: 2When you write your own MailerQ plugin, you must implement the mq_version() function. This function is called by MailerQ to find out if your plugin is compatible with the MailerQ binary.
Normally, this function can be very simple and just has to return the MQ_API_VERSION value. This is an integer value that increments with every new version of MailerQ.
#include <mailerq.h>
/**
* Function called by MailerQ to find out if the plugin is compatible
* with the MailerQ executable
*
* @return int API version number for which the plugin is compiled
*/
MQ_EXPORT int mq_version() {
return MQ_API_VERSION;
}
If the returned version number is not valid, MailerQ will report an error message and quit. If the function number is valid, MailerQ will continue to run, and will call your mq_initialize() function.
This function stops a timer that was previously created. Note that this function should only be called to stop a timer that has not yet fired yet, because a timer will automatically be cleaned up and calling MQ_stopTimer after this is not legal.
/**
* Stop a timer before it expires
*
* @param timer the timer to stop
*/
void MQ_stopTimer(MQ_Timer *timer);
For a detailed example, see MQ_startTimer
This function starts a timer. When the given time runs out, the callback will be invoked, unless the timer is stopped first with MQ_stopTimer.
/**
* Start a timer
*
* @param context the context to start the timer in
* @param timeout the number of seconds to wait before invoking the callback
* @param callback the callback to invoke when the timer runs out
* @param data customer user data to provide to the callback
*/
MQ_Timer *MQ_startTimer(MQ_Context *context, float timeout, void(*callback)(void *data), void *data);
Using timer functionality, we can amend the example for the MQ_ioWatch function so that it times out when no input is entered for at least five seconds.
#include <mailerq.h>
#include <string.h>
#include <stdio.h>
/**
* Structure that references both timers
*/
struct Timers
{
/**
* The IO watcher
*/
MQ_IOWatcher *io;
/**
* The timer
*/
MQ_Timer *timer;
};
/**
* Callback function to receive standard input
*
* @param watcher the watcher that is monitoring the file descriptor
* @param fd the file descriptor that received events
* @param data custom data pointer
*/
void io_callback(MQ_IOWatcher *watcher, int fd, int events, void *data)
{
// buffer for data to process
char data[1024];
ssize_t length;
// retrieve the timers
struct Timers *timers = (struct Timers*)data;
// we are working with STDIN, which is never writable
// (because that's what we have STDOUT for), so we don't
// have to check the events variable, otherwise we could
// check for readability or writability like this:
//
// events & MQ_READ
// events & MQ_WRITE
// normally, when we check for readability we should not
// assume that we are receiving complete lines, however
// STDIN is line-buffered, so the callback is only invoked
// whenever the user presses return after input
// read data from input
length = read(fd, data, sizeof(data));
// if there are only two characters we received an empty line
// since the line includes the CRLF produced by the terminal
if (length == 2)
{
// stop watching the file descriptor
MQ_ioUnwatch(watcher);
// deactive the timer
MQ_stopTimer(timers->timer);
// clean up the timers
free(timers);
}
else
{
// reset the timer to five seconds
MQ_resetTimer(timers->timer, 5.0);
}
}
/**
* Callback function for an expiring timer
*
* @param data custom user data
*/
void timer_callback(void *data)
{
// get the timers
struct Timers *timers = (struct Timers*)data;
// we did not receive any input for 5 seconds
// so we stop listening to input
MQ_ioUnwatch(timers->io);
// and free the timers
free(timers);
}
/**
* Monitor standard input for data
*
* @param context the context to monitor in
*/
void monitor(MQ_Context *context)
{
// create the structure holding the timers
Timers *timers = malloc(sizeof(struct Timers));
// set the context
timers->context = context;
// monitor stdin for input
timers.io = MQ_ioWatch(context, STDIN, MQ_READ, callback, timers);
// and stop monitoring after five seconds
timers.timer = MQ_startTimer(context, 5.0, timer_callback, timers);
}
If your plugin contains this function, it is called right after a message has been sent over an SMTP connection. The function is called after the message was sent, but before the message was accepted.
This function can be useful if you want to write a plugin that monitors which messages were actually sent over the line.
This method is designed to run asynchronously. You should not execute any blocking calls from within this function, if you do you will disturb other active connections resulting in timeouts.
#include <mailerq.h>
/**
* Load a MIME message
*
* @param context Access to the context
* @param connection Access to the SMTP connection
* @param message Access to the JSON message
* @return boolean Do we accept control from here?
*/
MQ_EXPORT bool mq_smtp_out_sent(MQ_Context *context, MQ_Connection *connection, MQ_Message *message) {
// if we return true here, MailerQ will stop processing this message
// until either the MQ_complete, MQ_continue or MQ_retry function is
// called. if none of the functions is called, MailerQ will wait for the
// remote server to close the connection and mark it as an error.
return true;
}
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
If your plugin contains this function, it will be called as soon as the SMTP connection enters the data state, giving you the chance to manipulate the message body before it is sent.
Normally, the full MIME messages that are going to be sent by MailerQ are stored in the JSON objects that are loaded from RabbitMQ. When a SMTP connection has been set up, the MIME is extracted from the JSON and sent over the SMTP connection.
But you may write a plugin that does this in a different way. If you want to write a plugin to generate the MIME in a different fashion, you can do so by implementing the mq_smtp_out_data() function. This function is called by MailerQ after the SMTP connection has been set up, to load the actual MIME message.
This method is designed to run asynchronously. You should not execute any blocking calls from within this function, if you do you will disturb other active connections resulting in timeouts.
#include <mailerq.h>
/**
* Load a MIME message
*
* @param context Access to the context
* @param connection Access to the SMTP connection
* @param message Access to the JSON message
* @return boolean Do we accept control from here?
*/
MQ_EXPORT bool mq_smtp_out_data(MQ_Context *context, MQ_Connection *connection, MQ_Message *message) {
// in most cases, plugins implementing this function will alter the
// message contents before it is sent. for this, you can use the
// MQ_setMime() function. other message properties can of
// course also be updated, but at this stage in the sending process
// it most likely won't matter (we have already connected to the
// SMTP server and have given it the email address to which we want
// to send a message, so this cannot be changed).
// if we return true here, MailerQ will stop processing this message
// until either the MQ_complete, MQ_continue or MQ_retry function is
// called. if none of the functions is called, MailerQ will wait for the
// remote server to close the connection and mark it as an error.
return true;
}
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
When an incoming SMTP connection is turned into a secure SMTP connection, MailerQ will call the mq_smtp_in_secure() function in the plugins. This function is called after the connection is secured, so you can be sure that the passed in connection object is guaranteed to be secure.
Your function should return true if you want your plugin to take over control from MailerQ, or false if control should stay with MailerQ.
bool mq_smtp_in_secure(MQ_Context *context, MQ_Connection *connection);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
Function that can be added to a plugin that is called every time a "RCPT TO" instruction is received by MailerQ.
bool mq_smtp_in_rcptto(MQ_Context *context, MQ_Connection *connection, MQ_Address *address);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
MailerQ calls this plugin function for every message that is received on the SMTP port. If you want to write a plugin that processes or modifies such messages, you can do so by implementing this function.
Your plugin may modify the JSON data in the MQ_Message. After all the plugins are ready with processing (and optionally modifying) the message, MailerQ will publish the JSON data to the RabbitMQ inbox queue.
Note that the SMTP protocol allows one to use multiple "RCPT TO" instructions to send a single MIME message to multiple addressees. If this is being used, the mq_smtp_in_message() function will also be called multiple times, once for every addressee.
bool mq_smtp_in_message(MQ_Context *context, MQ_Connection *connection, MQ_Message *message);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
Function that can be added to a plugin that is called every time a "MAIL FROM" instruction is received by MailerQ.
bool mq_smtp_in_mailfrom(MQ_Context *context, MQ_Connection *connection, MQ_Address *envelope);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
Function that can be implemented by a plugin and that is called by MailerQ every time a MIME message is received over a SMTP connection
bool mq_smtp_in_data(MQ_Context *context, MQ_Connection *connection, const char *mime, size_t size);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
When MailerQ receives an incoming SMTP connection, it will accept that connection, and iterate over all loaded plugins, and call the mq_smtp_in_connect() function in each of these plugins.
You can implement this function in your plugin if you want to run certain code right after an incoming TCP connection comes in on the SMTP port of MailerQ. Your function should return true if you want your plugin to take over control from MailerQ, or false if control should stay with MailerQ.
bool mq_smtp_in_connect(MQ_Context *context, MQ_Connection *connection);
The function takes two parameters:
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
Function that can be implemented by a plugin, and that is called by MailerQ when an incoming SMTP connection is closed. This is the perfect moment for a plugin to run cleanup code, for example to deallocate memory that is assigned to the connection with the MQ_setConnectionData() function.
bool mq_smtp_in_close(MQ_Context *context, MQ_Connection *connection);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
After a user has sent its login credentials over a SMTP connection, MailerQ will notify all the plugins about this, and call the mq_smtp_in_authenticate() function in each of the plugins (only if the function exists, of course).
Most plugins implement this function to add their own login administration system to MailerQ. If authentication succeeds, you can call MQ_setAuthenticated() to mark the connection as being an authenticated connection.
This function should return false if MailerQ can stay in control, or true if the plugin takes over control (for example because it starts a non-blocking database lookup). If the plugin takes over control, it should of course give back control to MailerQ when the credentials lookup has been completed.
bool mq_smtp_in_authenticate(MQ_Context *context, MQ_Connection *connection, const char *login, size_t loginsize, const char *passwd, size_t passwdsize);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
If you want to change the recipient address that is associated with a MQ_Message, you can use this function. This function allows you to store a new recipient address in a message.
/**
* Update the recipient address
*
* @param message the message to set the recipient address in
* @param recipient actual recipient address
* @param size length of the recipient string
*/
void MQ_setRecipient(MQ_Message *message, const char *recipient, size_t size);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for setting the "recipient" property in the JSON. Changing the recipient address by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the recipient with this function will not modify the JSON data. If your plugin calls this function before the "RCPT TO" instruction is sent, the newly set recipient address is going to be used in the SMTP protocol, but when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still have the original recipient address in the JSON. If you also want to modify the recipient address in the JSON data, you should use the MQ_json() function as well.
If you want to change the MIME data that is associated with a MQ_Message, you can use this function. This function sets the MIME data.
/**
* Update the MIME data
*
* @param message the message to set the envelope address in
* @param mime actual mime string
* @param size length of the mime string
*/
void MQ_setMime(MQ_Message *message, const char *mime, size_t size);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for setting the "mime" property in the JSON. Changing the MIME by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the MIME data with this function will not modify the JSON data. If your plugin calls this function before the actual data is sent over the SMTP connection, the newly set data is going to be used. However, when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still use the original JSON object. If you also want to set "mime" property in the JSON, you should use the MQ_json() function as well.
This function is often used in plugins that implement the mq_smtp_out_data() function.
Incoming messages are published to either the local message queue, or to the regular inbox queue. In a normal setup, this is normally used to decide whether the email is going to be delivered to a local address, or that it is going to be relayed.
If you write a plugin that intercepts incoming messages, you can update the "local" property by calling this method. If you set it to true, the message is going to be processed as a local email and will be published to the local message queue.
/**
* Update the local setting
*
* @param message the message to change
* @param local new setting
*/
void MQ_setLocal(MQ_Message *message, bool local);This function is only meaningful for incoming messages. You can call it on outgoing message too, but that will not change the behavior of MailerQ. This function is therefore only used in plugins that implement the mq_smtp_in_message() function.
If you want to change the envelope address that is associated with a MQ_Message, you can use this function. This function allows you to store a new envelope address in a message.
/**
* Update the envelope address
*
* @param message the message to set the envelope address in
* @param envelope actual envelope address
* @param size length of the envelope string
*/
void MQ_setEnvelope(MQ_Message *message, const char *envelope, size_t size);
This function has a different behavior if you call it on a message _that is being received_ and on a message _that is being sent_. If you call it on a message that is being received, for example inside your [mq_smtp_in_message()](mq_smtp_in_message) function, it is simply an alias for setting the "envelope" property in the JSON. Changing the envelope address by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the envelope with this function will not modify the JSON data. If your plugin calls this function before the "MAIL FROM" instruction is sent, the newly set envelope address is going to be used in the SMTP protocol, but when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still have the original envelope address in the JSON. If you also want to modify the envelope address in the JSON data, you should use the [MQ_json()](mq_json) function as well.This function can be called to store data which can then be retrieved later using the MQ_contextData() function. The data is segmented per plugin, so data stored will not overwrite data stored by other plugins, or in different contexts.
/**
* Store data in the context
*
* @param context the context to store the data in
* @param value the data to store
*/
void MQ_setContextData(MQ_Context *context, void *value);Function to associate plugin data with a SMTP connection. This data void-pointer is completely ignored by MailerQ, but can be useful for plugins that want to preserve data between calls to the plugin callback functions.
The data pointer is segmented per plugin, so this function will not not overwrite data stored by other plugins.
If the data is allocated by yourself, you will also have to make sure that the data gets deallocated, for example by also implementing the mq_smtp_in_close() function, in which you deallocate the data after the connection is closed.
/**
* Store userdata in the connection
*
* @param connection the connection in which to store the data
* @param void* the data to store
*/
void MQ_setConnectionData(MQ_Connection *connection, void *data);Set whether this connection should be considered authenticated. Note that if a previous plugin had already set this connection to be authenticated and you set it to be unauthenticated, the login will be rejected.
/**
* Set whether the connection should be considered authenticated
*
* @param connection the connection that authentication should be set for
* @param bool should the credentials be considered valid
*/
void MQ_setAuthenticated(MQ_Connection *connection, bool authenticated);Send raw data directly to the SMTP connection.
MailerQ takes care of its own output buffering, so you can be sure that all data is always delivered, unless the connection is suddenly closed by the peer.
/**
* Send data over a TCP connection
* @param connection the connection in which the data was stored
* @param buffer data to size
* @param size size of the buffer
* @return size_t number of bytes sent
*/
size_t MQ_send(MQ_Connection *connection, const void *buffer, size_t size);
Returns whether the connection is secure or not. A connection is secure if it has passed the "STARTTLS" state, and all traffic over it is encrypted.
/**
* Get whether the connection is secure or not
*
* @param connection the connection that we want to know the status of
* @return bool
*/
bool MQ_secure(MQ_Connection *connection);
MailerQ has a non-blocking nature, and uses callbacks to pass control to a plugin. It is then up to the plugin to interact with the event loop, and pass control back to MailerQ when it is ready with its task. Passing back control can be done in three different ways:
The MQ_retry() implements the third behavior, and is probably the most peculiar way of handing back control to MailerQ. If you call it, MailerQ will start all over, and will call all plugins again.
This is a very odd way of handing back control, and you normally should not use it, because you run the risk of ending up in an infinite loop. It could however be useful if your plugin makes a change to a received message, and you want to run all plugins again so that they can all respond to the modified message.
/**
* Finish plugin processing
*
* @param connection the connection that may move on to the next plugin
*/
void MQ_retry(MQ_Connection *connection);See also MQ_continue() and MQ_complete() for the other (and probably more useful) ways of handing back control.
This function is used to reset the timeout for a given timer without having to destroy and recreate the timer. This function may not be called on a timer that has already expired.
/**
* Reset a timer
*
* @param timer the timer to reset
* @param timeout the new timeout for the timer
*/
void MQ_resetTimer(MQ_Timer *timer, float timeout);
For a detailed example, see MQ_startTimer
If you want to retrieve the recipient from a MQ_Message, you can use this function for it. It returns a NULL terminated string, or simply NULL if no recipient has been set.
/**
* Retrieve the recipient
*
* @param message the message to retrieve the recipient from
* @return the recipient
*/
const char *MQ_recipient(MQ_Message *message);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for retrieving the "recipient" property from the JSON.
For outgoing messages however, the recipient set in the JSON and the recipient that is returned by this function may be different. This function returns the recipient that is actually being used in the SMTP protocol for the "RCPT TO" instruction, and could be different than the one that was originally loaded from RabbitMQ (for example, when a plugin modified the recipient in the mean time).
For more info, see the documentation about MQ_setRecipient.
If you want to retrieve the MIME data from a MQ_Message, you can use this function. It returns a NULL terminated string, or simply NULL if no mime has been set.
/**
* Retrieve the MIME data
*
* @param message the message to retrieve the mime from
* @return the MIME data
*/
const char *MQ_mime(MQ_Message *message);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for retrieving the "mime" property from the JSON.
For outgoing messages however, the "mime" property set in the JSON and the data that is returned by this function may be different. In fact, very often the original JSON does not even hold any MIME data, and it is exactly the task of the plugin to generate the MIME based on other properties in the MIME. Exactly for that reason, this function normally returns NULL if you call it from a plugin that intercepts outgoing messages - unless some other plugin has already set the data using the MQ_setMime() function.
For more info, see the documentation about MQ_setMime.
This structure is provided for plugin functions that work with message contents. It can be used to both retrieve and update message properties. If you use it for updating messages, you should be aware of the phase in the SMTP protocol that your plugin is intercepting: it is for example pointless to change the envelope address if the "MAIL FROM" instruction has already been sent.
All members in this MQ_Message structure are private, to interact with this structure, accessor functions have been created.
A MQ_Message pointer is passed to your plugin when a message is received on the incoming SMTP port, and when it is about to be sent over an outgoing SMTP connection. If your plugin implements one of the following functions, it can get access to a message, retrieve information from it, or make modifications to it.
| Function name | Description |
|---|---|
| mq_smtp_in_message() | Called by MailerQ when a message is received on an incoming SMTP connection |
| mq_smtp_out_data() | Called by MailerQ right after it has sent the "DATA" command and is about the send MIME data |
The MQ_Message structure is essentially a wrapper around the original JSON data that was loaded from RabbitMQ (if your plugin interacts with outgoing messages), or a wrapper around the JSON data that is going to be published to RabbitmQ (for incoming messages). Your plugin can get access to the JSON data, and make modifications to it.
| Function name | Description |
|---|---|
| MQ_json() | Get access to the JSON data of a message |
Besides the JSON, you can also access derived properties. You do not have to inspect the JSON directly, but can use one or more of the following methods to get access to the message properties:
| Function name | Description |
|---|---|
| MQ_envelope() | Get access to the envelope address |
| MQ_setEnvelope() | Set the envelope address |
| MQ_recipient() | Get access to the recipient address |
| MQ_setRecipient() | Set the recipient address |
| MQ_mime() | Get access to the MIME data |
| MQ_setMime() | Get the MIME data |
Available from MailerQ 5.6.5.
/**
* Log priorities
*/
#define MQ_CRIT 1
#define MQ_WARN 2
#define MQ_INFO 3
/**
* Log something in the MailerQ application-log. This will be prepended
* by the priority and the time in the final log entry.
* @param priority
* @param message
*/
void MQ_log(int priority, const char *message);This function is safe to be called from any thread simultaneously, and will log to the open MailerQ application-log output streams as set in the settings.
MQ_log(MQ_CRIT, "log-1");
MQ_log(MQ_WARN, "log-2");
MQ_log(MQ_INFO, "log-3");
MQ_log(-1, "log-invalid")If the plugin contains the snippet above, upon execution the log outputs as specified to the application log outputs. If an incorrect priority is passed, as above, (invalid priority) will be prepended to the log line and it will be logged under info priority.
[Critical] [2019-09-23 12:46:56] [plugin] log-1
[Warning] [2019-09-23 12:46:56] [plugin] log-2
[Info] [2019-09-23 12:46:56] [plugin] log-3
[Info] [2019-09-23 12:46:56] [plugin] (invalid priority) log-invalidIf you want to check whether a received incoming message was marked as a local message for local delivery, you can call this function. It returns boolean.
/**
* Retrieve the local setting
*
* @param message the message to retrieve the setting from
* @return the local setting
*/
bool MQ_local(MQ_Message *message);For more info, see the documentation about MQ_setLocal.
If you want to get access to the JSON data that was loaded from RabbitMQ (for plugins that interact with messages that are going to be sent) and/or with the JSON data that is going to be published to RabbitMQ (for plugins that want to modify incoming messages), you can call the MQ_json() function.
This function returns a "struct fjson_object". This is a pointer to a data from the >a href="https://github.com/rsyslog/libfastjson">libfastjson library. If you want to work with this data pointer, you therefore also need access to this JSON handling library.
/**
* Retrieve JSON data from a message
*
* @param message the message to retrieve the JSON from
* @return json pointer to the JSON object
*/
struct fjson_object *MQ_json(MQ_Message *message);
For more information on how to work with the JSON structure, please check the documentation of the libjson library:
This function will start monitoring a file descriptor for the requested events. When one (or more) of these events occur, the given callback will be invoked.
/**
* Monitor a file descriptor for activity
*
* @param context the context to register the file descriptor in
* @param fd the file descriptor to monitor
* @param events the events to monitor, binary or'ed MQ_READ and/or MQ_WRITE
* @param callback callback to invoke when one or more of the events occur
* @param data custom data to supply to the callback
*/
MQ_IOWatcher *MQ_ioWatch(MQ_Context *context, int fd, int events, void(*callback)(MQ_IOWatcher *watcher, int fd, int events, void *data), void *data);As an example, we will read from the standard input until an empty line is found, at which point we will stop listening to the input. This example makes no sense for a MailerQ plugin, but will illustrate the point.
#include <mailerq.h>
#include <string.h>
#include <stdio.h>
/**
* Callback function to receive standard input
*
* @param watcher the watcher that is monitoring the file descriptor
* @param fd the file descriptor that received events
* @param data custom data pointer
*/
void callback(MQ_IOWatcher *watcher, int fd, int events, void *data)
{
// buffer for data to process
char data[1024];
ssize_t length;
// we are working with STDIN, which is never writable
// (because that's what we have STDOUT for), so we don't
// have to check the events variable, otherwise we could
// check for readability or writability like this:
//
// events & MQ_READ
// events & MQ_WRITE
// normally, when we check for readability we should not
// assume that we are receiving complete lines, however
// STDIN is line-buffered, so the callback is only invoked
// whenever the user presses return after input
// read data from input
length = read(fd, data, sizeof(data));
// if there are only two characters we received an empty line
// since the line includes the CRLF produced by the terminal
if (length == 2)
{
// stop watching the file descriptor
MQ_ioUnwatch(watcher);
}
else
{
// process the data here
}
}
/**
* Monitor standard input for data
*
* @param context the context to monitor in
*/
void monitor(MQ_Context *context)
{
// monitor stdin for input
MQ_ioWatch(context, STDIN, MQ_READ, callback, NULL);
}This function will stop monitoring a file descriptor.
/**
* Stop monitoring a file descriptor
*
* @param watcher the watcher to deactivate
*/
void MQ_ioUnwatch(MQ_IOWatcher *watcher);For a more detailed example, see the MQ_ioWatch documentation
The mq_initialize() function is an optional function that you could implement in a plugin. When MailerQ starts and has loaded all the plugins, it will check every plugin to find out if this mq_initialize() function is available, and if it is, it will call it.
You can use this function to run your own initialization code.
#include <mailerq.h>
/**
* Initialize the plugin
*/
MQ_EXPORT void mq_initialize() {
// @todo add your initialization code
}The counterpart of the mq_initialize() function is the mq_cleanup() function.
Get access to the socket filedescriptor that represents the underlying TCP connection of the SMTP connection.
Watch out! The socket is completely managed by MailerQ, and it is in general not a good idea to use the filedescriptor for sending or receiving data, or to close it. Your plugin should only use the filedescriptor to retrieve information about the connection, for example to retrieve the local or remote IP addresses of the connection using regular C functions getsockname() or getpeername().
/**
* Retrieve file descriptor that is used for the SMTP connection
* @param connection the connection in which the data was stored
* @return int
*/
int MQ_fileDescriptor(MQ_Connection *connection);
Internally, MailerQ works with libev for registering timers and watching file descriptors and while some simplified wrapper functions are available for the MQ_Context, you might find it easier to work directly with libev instead.
/**
* Retrieve the underlying libev event loop from
* the given context
*
* @param context the context to retrieve the loop from
* @return struct libev
*/
struct ev_loop *MQ_ev(MQ_Context *context);
The loop instance that is returned will already be running and since it is likely to be handling many other smtp and database connections it is highly discouraged to stop this loop, since it will most likely break other open connections. You should only use it for adding timers and filedescriptors to it.
If you want to retrieve the envelope from a MQ_Message, you can use this function for it. It returns a pointer to a MQ_Address structure, or simply NULL if no envelope was set.
/**
* Retrieve the envelope address
* @param message the message to retrieve the envelope address from
* @return the envelope
*/
MQ_Envelope *MQ_envelope(MQ_Message *message);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for retrieving the "envelope" property from the JSON.
For outgoing messages however, the envelope set in the JSON and the envelope that is returned by this function may be different. This function returns the envelope that is actually being used in the SMTP protocol for the "MAIL FROM" instruction, and could be different than the one that was originally loaded from RabbitMQ (for example, when a plugin modified the envelope in the mean time).
For more info, see the documentation about MQ_setEnvelope.
If you want to retrieve the full email address stored inside the MQ_Address structure you can use this function.
The pointer that is returned should not be freed by the user. MailerQ will automatically free it when it is done processing the message.
/**
* Retrieve the email address given an envelope structure
*
* Be aware: This function may return a nullptr in case no
* envelope address exists. This is most commonly
* seen in DSN messages.
*
* @param envelope the envelope address
* @return const char *
*/
const char *MQ_email(MQ_Address *address);MailerQ has a non-blocking nature, and uses callbacks to pass control to a plugin. It is then up to the plugin to interact with the event loop, and pass control back to MailerQ when it is ready with its task. Passing back control can be done in three different ways:
The MQ_continue() function can be used for the first of the above options to hand control back to MailerQ. If you call MQ_continue(), MailerQ will call the next plugin in line, or continues with its original algorithm if no other plugins are available.
/**
* Finish plugin processing
*
* @param connection the connection that may move on to the next plugin
*/
void MQ_continue(MQ_Connection *connection);See also MQ_complete() and MQ_retry() for the other ways of handing back control.
This function can be called to retrieve user data set earlier with MQ_setContextData.
Every plugin may associate data with a plugin. This data pointer is completely ignored by MailerQ, but allows plugins to store data between different callbacks for the same context. The data pointer is unique per plugin, so you do not have to worry that this function returns pointer to data that is set by a different plugin.
If no data was set earlier, this function returns NULL.
/**
* Retrieve previously stored data.
*
* @param context the context to retrieve the data from
* @return void*
*/
void *MQ_contextData(MQ_Context *context);The mq_context_initialize() is an optional function that you can add to your plugin. It is called by MailerQ during startup for each worker thread that is created. This is useful if, for example, you want to set up a database connection per thread, or want to do other initialization.
This function can be implemented by the plugin and is called by MailerQ when a worker thread starts. This function cannot take control by returning boolean true, but since this function is executed only during startup, it is acceptable to block here. MailerQ will wait until all contexts are initialized before continuing.
#include <mailerq.h>
/**
* This function is called by MailerQ for every worker thread
* that is created
*/
MQ_EXPORT void mq_context_initialize([MQ_Context](mq_context) *context) {
// set up a connection to MySQL
MYSQL *connection = mysql_init(NULL);
// TODO: connect here and do other useful stuff
// store the connection so it can be used in later callbacks
MQ_setContextData(context, connection);
}If you allocate memory in this function, you probably also have to add a mq_context_cleanup to your plugin.
This function is called by MailerQ right before a worker thread is stopped and destroyed. You can for example use it to deallocate memory that you had earlier allocated.
This function can be implemented by the plugin and is called by MailerQ when a worker thread is about to stop. It is not possible to take control by returning boolean true here, but since this function is called during cleanup it is acceptable to block here.
#include <mailerq.h>
/**
* This function is called by MailerQ for every worker thread
* that comes to an end
*/
MQ_EXPORT void mq_context_cleanup(MQ_Context *context) {
// get the mysql connection resource that we stored in the context
MYSQL *connection = (MYSQL *)MQ_ContextData(context);
// close the connection
mysql_close(connection);
// TODO: add other useful cleanup stuff
}This structure is provided for every asynchronous function a plugin can implement. It can be used to get access to the event loop as well as indicating to MailerQ when the plugin is done.
MailerQ is a multi-threading application, with multiple working threads that are all sending and receiving messages at the same time. For every single worker thread, a separate context object is created. If you want to be notified when a new context is created, or when a context is destroyed, you can define the following functions in your plugin:
| Implementable function | Description |
|---|---|
| mq_context_initialize() | Called by MailerQ when a worker thread is created |
| mq_context_cleanup() | Called by MailerQ when a worker thread is destroyed |
Both of the listed functions are optional. It is not necessary for your plugin to export them, but you can for example use these functions to initialize plugin specific data, and to deallocate that data when the worker thread exits.
Your plugin may store a pointer to plugin specific data in a context. This is a pointer that is completely ignored by MailerQ, but that might be useful for your plugin. If you allocate data for this, it is also your responsibility to deallocate the data when the context is destroyed.
| Callable function | Description |
|---|---|
| MQ_contextData() | Retrieve the plugin managed data from the context |
| MQ_setContextData() | Assign plugin managed data to the context |
The plugin data is not shared between plugins. Every plugin can therefore associate its own data with a context, and you do not have to worry about other plugins that overwrite your data.
The main feature that a MQ_Context offers, is access to the event loop. There are a number of functions available that you can use for registering file descriptors and timeouts with the event loop. Internally, MailerQ uses a libev event loop, and you can also get access to this loop pointer.
| Callable function | Description |
|---|---|
| MQ_ev() | Access to the underlying libev instance |
| MQ_ioWatch() | Watch a filedescriptor for activity |
| MQ_ioUnwatch() | Stop watching a filedescriptor for activity |
| MQ_startTimer() | Start a timer |
| MQ_stopTimer() | Stop a timer |
| MQ_resetTimer() | Reset a timer to a new timeout |
Retrieve the plugin managed data pointer stored for this connection. If you've made a call to MQ_setConnectionData to associate data with a SMTP connection, you can use this function to get back that data pointer.
The data pointer is completely ignored by MailerQ. This API function is only provided to give plugins the possibility to associate data with a connection.
The data pointer is segmented per plugin, so this function will not return data stored by other plugins. If no data was stored earlier, NULL will be returned.
/**
* Retrieve userdata stored using MQ_setConnectionData
* @param connection the connection in which the data was stored
* @return void*
*/
void *MQ_connectionData(MQ_Connection *connection);
Be aware that if you store data in the connection that you had allocated yourself, it is also your responsibility to deallocate that data before the connection is closed.
The MQ_Connection structure is passed to all callbacks dealing with SMTP connections. It allows your plugin to find out information about the connection, and to send data over the connection.
There is a whole list of functions that your plugin can export that deal with connections.
| mq_smtp_in_connect() | Called after an incoming TCP connection is established. This is the first time that the MQ_Connection is passed to the plugin, and a good place for per-connection initialization code. |
| mq_smtp_in_secure() | Called when the STARTTLS instruction is received and the connection is turned into a secure connection |
| mq_smtp_in_authenticate() | Called when the SMTP client sends the login+password credentials. Allows you to run your own authentication checks. |
| mq_smtp_in_mailfrom() | Called when the "MAIL FROM" command is received |
| mq_smtp_in_rcptto() | Called when the "RCPT TO" command is received |
| mq_smtp_in_data() | Called when the "DATA" command is received |
| mq_smtp_in_message() | Called when a message is received. When multiple "RCPT TO" addresses were given, this function is called once for each address. |
| mq_smtp_in_close() | Called after an incoming SMTP connection is closed. This is the last time that the MQ_Connection is passed to the plugin. You can use this function for cleanup code. |
The above functions are only called for MQ_Connection pointers that represent incoming connections.
All members in the MQ_Connection structure are private. To interact with the structure the following accessor functions have been created.
| Callable function | Description |
|---|---|
| MQ_fileDescriptor() | Retrieve the filedescriptor linked to a connection |
| MQ_send() | Send data over a SMTP connection |
| MQ_authenticated() | Returns whether this is an authenticated connection |
| MQ_setAuthenticated() | Mark a connection as an authenticated connection |
| MQ_secure() | Returns whether this is a secure TLS connection |
| MQ_connectionData() | Retrieve the data that is associated with the SMTP connection |
| MQ_setConnectionData() | Associated data with the SMTP connection |
MailerQ has a non-blocking nature, and uses callbacks to pass control to a plugin. It is then up to the plugin to interact with the event loop, and pass control back to MailerQ when it is ready with its task. Passing back control can be done in three different ways:
The MQ_complete() implements the second behavior. If you call it, MailerQ will skip all subsequent plugins and continues with its original algorithm.
/**
* Finish plugin processing
*
* @param connection the connection that may move on to the next plugin
*/
void MQ_continue(MQ_Connection *connection);See also MQ_continue() and MQ_retry() for the other ways of handing back control.
The mq_cleanup() function is an optional function that you could implement in a plugin. When MailerQ is about to shut down , it will check every plugin to find out if this mq_cleanup() function is available, and if it is, it will call it.
You can use this function to run your own cleanup code.
#include <mailerq.h>
/**
* Clean up stuff
*/
MQ_EXPORT void mq_cleanup() {
// @todo add your cleanup code
}Returns whether the connection is considered authenticated. A connection is authenticated if the user has sent a valid username and password, or when a plugin has explicitly marked it as authenticated using the MQ_setAuthenticated() function.
/**
* Get whether the connection is considered authenticated
* @param connection the connection that we want to know the status of
* @return bool
*/
bool MQ_authenticated(MQ_Connection *connection);Envelope and recipient addresses are wrapped in MQ_Address structures. You can get a reference to this structure by calling the MQ_envelope() or MQ_recipient() function on a MQ_Message structure, or by implementing the mq_smtp_in_mailfrom() or mq_smtp_in_rcptto() functions.
Once you have access to a MQ_Address struct, you can call the following functions on it to get access to the underlying address, and to other address properties.
| Callable function | Description |
|---|---|
| MQ_email() | Access to the underlying email address |
| MQ_local() | Check whether a recipient address is "local" |
| MQ_setLocal() | Mark a recipient as "local" |
MailerQ is configured via one central configuration file: "/etc/mailerq/config.txt". It holds many options. The most important ones are the address and login credentials of your RabbitMQ message broker and the address of your database. You probably need to change these settings. All other config file settings have decent defaults that work directly (although you might want to take a look at them later).
The "rabbitmq-address" setting in the config file holds the address of the RabbitMQ server that you want to use for message queueing. It has the format "amqp://user:password@hostname/vhost". The default value ("amqp://guest:guest@localhost/") works if your RabbitMQ server is running on the same machine as MailerQ, and when you've not altered the default guest/guest credentials.
rabbitmq-address: amqp://user:password@hostname/vhostIf you have a cluster of RabbitMQ nodes, the hostname can be separated by semicolons (e.g. amqp://user:password@host1;host2;host3/vhost). Setting up a RabbitMQ cluster means you're less likely to lose messages when one RabbitMQ server fails.
MailerQ stores all runtime settings in a relational database. This can be a MySQL, MariaDB, PostgreSQL or SQLite3 database. The SQLite3 database is by far the easiest one to set up, because it does not require a database server and all you have to do is just enter the path to a file on your server where MailerQ can store its runtime settings. This file does not even have to exist as it will be automatically created by MailerQ when it is missing.
database: sqlite:///path/to/database/file.sqlWhen MailerQ first connects to a database (or first opens the database file), it automatically creates all database tables and initializes everything.
To be able to connect, you do have to ensure that the appropriate database client libraries are installed on your system. If you use the sqlite database engine, make sure you have libsqlite3 on your system.
If you inject mails in SMTP format into MailerQ, you can add special headers to your message that are read and recognized by MailerQ, and that control the delivery of your message. Headers that start with "x-mq-*" are automatically recognized by MailerQ, stripped from the email message and converted into JSON properties before the message is sent to one of the message queues.
Almost all "x-mq-*" properties are converted into JSON properties, even custom properties that do not have a special meaning (like "x-mq-meaningless"). However, the following properties have a special meaning and have influence on the handling of your message.
| x-mq-envelope | explicit envelope address |
| x-mq-ip | IP address to send mail from |
| x-mq-pool | pool identifier to send the message from |
| x-mq-tag | adds a tag to the message |
| x-mq-delayed | time when message should be sent |
| x-mq-maxdelivertime | max time for delivery |
| x-mq-maxattempts | max number of delivery attempts |
| x-mq-retries | interval in seconds between the attempts |
| x-mq-inlinecss | instruct MailerQ to inlinize the CSS code |
| x-mq-results-queue | alternative queue for the results |
| x-mq-failure-queue | alternative queue for the failures |
| x-mq-success-queue | alternative queue for the failures |
| x-mq-retry-queue | alternative queue for the retries |
| x-mq-smarthost-hostname | hostname of smarthost to send email to |
| x-mq-smarthost-port | portnumber of the smarthost |
| x-mq-smarthost-username | login for the smarthost |
| x-mq-smarthost-password | password to access the smarthost |
| x-mq-dsn-notify | the events for which to receive a DSN |
| x-mq-dsn-ret | whether to send the full body or just the headers in DSN |
| x-mq-dsn-orcpt | the original recipient for the DSN |
| x-mq-dsn-envid | unique identifier for the DSN message |
| x-mq-keepmime | do not remove the full mime object after delivery |
The following message would be an example MIME message that holds some extra meta properties:
from: info@example.com
to: peter@smtpeter.com
subject: example message
x-mq-ip: 10.11.12.13
x-mq-maxdelivertime: 2016-10-23 23:00:00
x-mq-inlinecss: 1
x-mq-custom-identifier: my-id
this is the body of the mailWhen the message above is injected into MailerQ, it will be transformed into a JSON object that is sent to the inbox queue. The JSON object will have the following properties:
{
"envelope": "info@example.com",
"recipient": "peter@smtpeter.com",
"mime": "....",
"ips": [ "10.11.12.13" ],
"maxdelivertime": "2016-10-23 23:00:00",
"inlinecss": true,
"custom-identifier": "my-id"
}To ease readability, we have left out the mime data. The mime data would be identical to the injected mime, without all the x-mq-* headers.
The envelope address from an injected mail is extracted from the MIME message. MailerQ checks (in this order) the following properties to extract the envelope address: "x-mq-envelope", "return-path", "from" and "sender". The first valid email address to be found in one of these headers will be used as envelope address.
The envelope address is the address that is used in the "MAIL FROM" communication.
If your server has multiple IPs to send out mail from, you can use all of them to send out mail from. To control the IP address that is going to be used, you can use the "x-mq-ip" property.
To allow multiple IP address for sending out the mail, you can add a list of available IP addresses:
x-mq-ip: 231.34.13.156
x-mq-ip: 231.34.13.158
MailerQ will pick one of the IPs to send out the mail. Be aware that you can of course only use addresses that are actually bound to the host that MailerQ runs on. Other IP addresses will result in failed deliveries.
You may be injecting different types of messages into MailerQ, for example messages belonging to your various customers or campaigns. To help you get a better overview, MailerQ allows you to tag your messages with one or more labels of your choosing. These tags will show up in the management console, allowing you to monitor and control all deliveries belonging to a tag. To add a tag in a MIME email you can add one or more "x-mq-tag" properties to the header:
x-mq-tag: Customer Name
x-mq-tag: Example CampaignMailerQ will track messages belonging to a tag and allow you to monitor them in the management console. See the tags page for more information.
When a message cannot be delivered immediately because of unresponsive receivers, greylisting or throttling, MailerQ publishes back the email to the outbox queue for later delivery. This can result in emails that are sent much later than the time that you first added them to the message queue.
If you do not want an email to be delivered after a certain time, all you need to do is add the "x-mq-maxdelivertime" header to the message. The "x-mq-delayed" property does exactly the opposite: it delays the email so that it is not delivered before a specific time.
x-mq-delayed: 2016-02-10 12:00:00
x-mq-maxdelivertime: 2016-03-10 00:00:00Timestamps in MailerQ are always in UTC. MailerQ is not very tolerant in parsing timestamps, so better make sure that you use the right formatting (YYYY-MM-DD HH:MM:SS).
If you do not specify an explicit max delivery time, MailerQ will attempt to deliver the mail within 24 hours (default) after the mail was first picked up from the outbox. You can change the default in the configuration file.
Just like a maximum delivery time, you can control the maximum number of attempts MailerQ should make sending an email. If a first attempt fails because a remote server is unreachable or does not immediately accept the message, MailerQ will make a new attempt a little later.
x-mq-maxattempts: 3
If a mail is retried, MailerQ waits a while before the next attempt is sent. You can set this dely in the headers of the message.
x-mq-retries: 300,600,3600The above setting means that the second attempt is ent 300 seconds (5 minutes) after the first attempt failed. If the second attempt failed too, MailerQ waits another 600 seconds (10 minutes) for the next attempt. After that attempt, all further attempts will be sent with a one hour (3600 seconds) delay.
When you send out HTML emails, you face the problem that not all email clients
support stylesheets that are set in the header. Some email clients (especially
web based clients) strip out the CSS code from the HTML header. This often messes
up the layout and look of your messages. To overcome this, it is better not to set
CSS settings in the header in the first place, but use style="..." attributes
in the HTML code instead.
MailerQ can do this automatically. If you set the x-mq-inlinecss header to true,
MailerQ parses the HTML email, and converts the CSS code from the HTML header
into inline style="..." attributes in the HTML body.
x-mq-inlinecss: 1
After MailerQ has processed an email, it publishes the result of the delivery to one or more result queues. The default queues to use for this are configured in the global configuration file, but you can set other queue names in properties to tell MailerQ to use other result queues.
x-mq-results-queue: name-of-results-queue
x-mq-failure-queue: name-of-failure-queue
x-mq-success-queue: name-of-success-queue
x-mq-retry-queue: name-of-retry-queue
Note that when only one if these "x-mq-*-queue" properties is used, all the queues mentioned in the global configuration file are ignored. This is even so if you have not even specified all possible queues in the MIME header.
Instead of having MailerQ sending messages directly to the final recipient, it can be sent to an alternative mailserver instead: the smarthost. The default credentials can be set in the config file, or they can be provided the following properties:
x-mq-smarthost-hostname
x-mq-smarthost-port
x-mq-smarthost-username
x-mq-smarthost-passwordRead more about using a smarthost
Some mail servers initially seem to accept a message, but send back a bounce email later. Delivery Status Notification (DSN) messages help you keep track of what happened to your mails. The following options specify which Delivery Status Notifications you want to receive:
x-mq-dsn-notify
x-mq-dsn-orcpt
x-mq-dsn-ret
x-mq-dsn-envidRead more about Delivery Status Notifications
When a message is completely processed - either because it was successfully delivered, or the delivery failed - MailerQ publishes it to the results queue, where you can pick it up for further processing.
By default, MailerQ throws away the mime data to make room in the JSON object and
in the message store. If you do not want the message data to be removed, you can
tell so by adding the keepmime option:
x-mq-keepmime: 1
To reduce the load on RabbitMQ, MailerQ can use an external message store. In this case, only email metadata (the recipient, the envelope address, et cetera) has to be stored in the JSON object in RabbitMQ, while the full MIME data can be stored in the message store.
storage-address: mongodb://hostname/database/collection
storage-threads: 1
storage-policy: all
storage-ttl: 3600This message store is completely optional: if you set the "storage-address" variable to an empty string, MailerQ works just as well (even faster because no extra communication with the storage server is necessary), but the load on RabbitMQ and the network will be much higher.
A number of different storage systems are supported: Couchbase, MongoDB or plain files stored on the local file system. In theory it is also possible to use MySQL, SQLite or PostgreSQL, but it is better not to use a relational database, because such systems are not optimized for document storage.
The address of the message store can be set with the "storage-address" config file variable. The following addresses are supported:
storage-address: couchbase://password@hostname/bucketname
storage-address: mongodb://hostname/database/collection
storage-address: sqlite:///path/to/database/file
storage-address: mysql://user:password@hostname/databasename
storage-address: postgresql://user:password@hostname/databasename
storage-address: dir:///path/to/directoryIf you have a cluster of Couchbase or MongoDB servers, you can split the
hostnames with semicolons (;). For MongoDB you can also specify
the name of the replica set in the address if you have more than one server.
storage-address: couchbase://password@host1;host2;host3/bucketname
storage-address: mongodb://host1;host2;host3/replicaset/database/collectionMailerQ relies on external libraries to be present on your system to communicate with the external storage engine. The Couchbase C client library has to be installed if you want to use the Couchbase storage engine, and the Mongo C Driver is needed to connect to MongoDB.
The address string to connect to MongoDB is directly passed to the MongoDB driver. All the options that are supported by this driver can be used in the address string.
Click here for the MongoDB documentation
Sometimes MongoDB cannot succesfully retrieve data, even though it is available. In this case, it is useful to repeat the fetch operation a couple of times. To enable this feature, you can add a special option to the address:
storage-address: mongodb://hostname/database/collection?readAttempts=3The default number of attempts is 1. If you want to repeat failed lookups a couple of times, you can pass in a higher value.
To prevent that many small read operations are fired at MongoDB, MailerQ normally groups fetch operations into a single "multi-get" operation that fetches many documents with just a single query. This reduces the number of queries that are sent to MongoDB, but if one of the requested documents is hard to find, it also slows down the lookup of all the other documents in the same query. To get a balance between this, you can limit the number of fetch operations that can be grouped together:
storage-address: mongodb://hostname/database/collection?maxQuerySize=10MongoDB has a limitation of around 16 MB per document (there is some overhead due to the usage of their internal BSON representation). If MailerQ has to store a bigger document, the message is split up into smaller parts that are all individually stored into MongoDB. As a consequence, you can find three types of documents in the database.
Most mails are less than 16mb big. So you will mostly see regular messages in the database with the following properties:
If the message is bigger than 16mb, MailerQ splits up the message and stores a master document with the following properties:
Each individual part of a big message has the following properties:
If you use the "directory://" storage backend, MailerQ stores all the messages in separate files on the file system. To prevent that the number of files in a directory becomes too big, MailerQ creates a nested directory structure. By default, this directory structure is four files deep. If you want to use a different depth, you can specify this via a "depth" parameter:
storage-address: directory:///path/to/directory?depth=3Strangely enough, most storage drivers only offer a synchronous and blocking API. This means that storage operations can only be started after previous operations were completed. To prevent that the entire MailerQ process gets blocked while a storage operation is in progress, MailerQ maintains multiple connections to the storage servers, and starts separate threads in which the storage operations are being executed.
The number of threads (and the number of storage connections) can be set using the "storage-threads" variable. If you set this to a higher value, the throughput of storage operations gets better too.
The "storage-policy" config file setting tells MailerQ what type of messages should be stored in the message store. Valid values are "all", "out", "in" and "none". The "none" setting is meaningful if you only want MailerQ to retrieve mime data from external storage, without ever starting storage operations.
Before MailerQ publishes a message to RabbitMQ (for example, before it sends a received message to the inbox queue, or before it send a delayed message back to the outbox queue) it checks the storage policy to see whether the mime data should be sent to RabbitMQ too, or whether the mime data should be stored in a different storage system.
If you want all messages to be stored in the message store, use the "all" policy. If this policy is enabled, MailerQ checks each JSON object before it gets published to RabbitMQ. If the JSON still contains mime data, MailerQ removes this data from the JSON and stores it in the message store instead. The JSON data will be updated with a "key" property that refers to the data in the message store.
The "in" and "out" policies are more complex. The "out" policy instructs MailerQ to use the message store for outgoing messages only. If a message is greylisted or delayed and is published back to the outbox, MailerQ first strips the mime data from the JSON, and stores that in the message store. Incoming messages (like the ones that come in on the SMTP port, or the messages dropped in the spool directory) are not checked and the full mime data is published to RabbitMQ.
The "out" policy is often used, because most emails get delivered at the very first attempt, and it is therefore often a waste of resources to store incoming messages first in a NoSQL environment: the messages will probably be retransmitted a fraction of a second later. By using the "out" storage policy, initial injected emails are completely sent to RabbitMQ. Only if the initial delivery fails and the message is sent back to the outbox for later delivery, the full MIME data is stripped from the JSON and stored in the separate storage.
The "out" policy especially makes sense in setups where the majority of all deliveries succeed at the first attempt, and rescheduled attempts are likely to be pumped around between MailerQ and RabbitMQ for a number of times.
Only "all", "out" and "none" are meaningful policies. For completeness however, we also support the "in" property which does exactly the opposite as the "out" policy: all incoming messages are stored in the message store, and only metadata is published to RabbitMQ. However, when a mail is delayed and has to be published back to the outbox, the mime data is kept inside RabbitMQ.
When you store messages, you probably don't want to keep them forever in your message store. To overcome this, every message has a time-to-live value, and expired message are automatically removed from storage. The "storage-ttl" config option specifies the default time-to-live that is used for message that are stored in the document store.
Note that the time-to-live is added to the mail max delivery time. If you try to send out an email using MailerQ, and that email has to be delivered within 24 hours, and your "storage-ttl" is set to 3600 seconds (one hour), the mime data will be stored in NoSQL for at most 25 hours.
MailerQ uses a timeout for storage operations. If the storage server does not respond within this time, the message is published back to RabbitMQ and will be retried later.
storage-timeout: 20
storage-reschedule: 120With the above settings you tell MailerQ to timeout fetch-operation after 20 seconds, and publish the mail back to RabbitMQ for 120 seconds.
You may choose to compress the messages you store to reduce the load on your storage server and limit the amount of data you need to send. This feature can be turned on by specifying "compression=gzip" in the storage URL, e.g.:
storage-address: mongodb://hostname/database/collection?compression=gzipOur NoSQL library will then take care of compressing all data behind the scenes. Note that this does mean that the contents of your database are no longer human readable.
The compression feature is currently enabled only for
Support for the couchbase and SQLite backends is still in an experimental state.
MailerQ relies on external RabbitMQ message queues for message queueing. All incoming and outgoing emails are read from and published to RabbitMQ queues, and all results are published to such queues as well.
The messages in the queues all are JSON formatted. This means that if you develop scripts or programs that inject emails directly into a message queue, your messages must be JSON encoded as well, and must use properties that are understood by MailerQ. Scripts or programs that process results from the result queues, should be able to read in and understand the JSON formatted result objects.
If you inject mails in MIME format (for example via the SMTP interface, the spool folder or the command line interface) you can of course not make use of the JSON properties to control the delivery. The properties that you would otherwise set in the JSON can then be set using special MIME headers that can be added to the email message.
MailerQ comes equipped with a full MTA management console. The management console allows you to monitor the performance of your email delivery in real-time. If necessary, the management console can be used to change settings to maximize deliverability on the fly. If you have multiple instances of MailerQ running in a cluster, you can easily switch between them.
The management console can be enabled in MailerQ's configuration file. The following variables should be used:
www-port: 8485 (default: 8485)
www-ip: 1.2.3.4 (default: 0.0.0.0, meaning all available IP's)
www-password: admin (empty by default)
www-dir: /usr/share/mailerq/www (default: /usr/share/mailerq/current/www)
www-connections: 10The www-port variable holds the port number for the management console;
8485 is the default. If you use port 80 (which is the default port for HTTP
traffic) you can access the management console with using a browser via
address http://hostname.of.your.server. If you assign a different port number
(like 8485), you must include the port number in the URL: http://hostname.of.your.server:8080.
In its default setting of 0.0.0.0, the management console is accessible via
all IP addresses that are assigned to the server on which MailerQ runs. If you
only want to make it accessible via one specific IP, you can set the www-ip
variable. Of course, the IP address that you assign must be bound to the server.
The management console is protected with a password to prevent anyone from
accessing it. This password can be set with the www-password
variable. Besides setting a password, we also recommend to put the
management console behind a firewall so that you will not have to worry
about people breaking into it.
All HTML, CSS and Javascripts that are necessary for the management
console are automatically installed into the /usr/share/mailerq/current/www
directory. If you want to run the console from out of a different
location, you can change this directory with the www-dir variable.
To limit the number of resources that can be used by the built-in HTTP server, you can use the "www-connections" variable to limit the number of simultaneous HTTP connections that can be handled. This number includes active web sockets.
If is a good idea to secure your management console, as it will also be used to manage private DKIM keys; by definition, these should be kept private and thus not transferered over interceptable non-secure HTTP connections.
The following configuration file variables are relevant for enabling HTTPS support:
www-secure-port: 443 (empty by default)
www-certificate: /path/to/certificate.crt (empty by default)
www-privatekey: /path/to/privatekey.key (empty by default)
www-ciphers: !aNULL:!eNULL:!LOW:!SSLv2:!EXPORT:!EXPORT56:FIPS:MEDIUM:HIGH:@STRENGTH (empty by default)If you enable both HTTP and HTTPS, users who access the non-secure interface
will automatically be forwarded to the secure interface. The www-secure-port
holds the port number for the HTTPS connections (443 is the default for
this, so that you won't have to include the port number in URLS). The
certifate and key files, and the supported ciphers can be set using
the www-certificate, www-privatekey and www-ciphers variables.
Once enabled, the encrypted management console can be accessed using
the address https://hostname.of.your.server if you use default port 443,
or https://hostname.of.your.server:port for any other port.
If you have a cluster with multiple MailerQ instances, the web interface of these interfaces contains links to the other instances. MailerQ does its best to find out the URL for each of the other interfaces (by combining the host names and port numbers), but you can use the following optional config file variables to help a hand:
www-host: your.hostname (default: auto-detected)
www-url: https://your.hostname:port (default: auto-detected)The management console allows administrators to monitor live SMTP traffic. All incoming or outgoing connections can be intercepted, and the entire SMTP handshake (EHLO, MAIL FROM, RCPT TO, DATA) is real time visible via the management console. You can thus see he raw MIME message data that is being sent or received.
In fact, the management console even has an option to not only display the raw MIME data, but to extract the HTML source code too, and render this in your browser. With this tool you can exactly see what type of messages your users are sending, and you can take action if you see messages that look like spam, phishing or other types of abuse.
However, if you use the console to look at rendered emails, your browser automatically downloads images and other resources from the mail as well. This could trigger actions (like statistics updates) on the servers where these files are hosted. If you do not want this and you have control over these servers, you can take precautionary measures, for example by ignoring downloads that come from your IP address, or by ignoring downloads from clients with a specific user agent setting.
At Copernica we use this approach. Our devops have browsers with a special user agent setting (with Firefox you can change this using the "about:config" url), so that clicks and opens from us do not pollute mailing stats. In MailerQ's config file we have set the "render-useragent" value too, so that MailerQ refuses to share emails with browsers with a different user agent setting.
render-useragent: Copernica DevOpsThe "render-useragent" setting ensures that only browsers with the specified user agent can render emails. If you have also updated your tracking servers to ignore clicks and opens from browsers with this user agent setting, you can make sure that you can safely look at rendered emails on the management console, without triggering any actions on your tracking servers.
The resources for the web interface (like the html, css, and javascript files) are sometimes cached by your browser. This is normally ok (because you probably do not update the files anyway), but if you plan to install a new MailerQ release once every while, you can set the cache-control header in the config file:
www-cache-control: must-revalidateThis config file option was mainly added to MailerQ to help the MailerQ web developers: they make changes to the interface all the time, and they wanted to prevent that the testers were reviewing an outdated interface. But in production environments, this setting might be useful too.
Check out the documentation on mozilla.org for a list of supported options for this header.
All delivery attempts like received emails, downloads and errors can be logged by MailerQ in different log files. The log file settings can be specified in the "/etc/mailerq/config.txt" config file.
In previous version of MailerQ (< 5.6), log rotation was handled in the application itself. This is not very flexible, and has therefore been removed. Instead, log rotation is now delegated to logrotate. The packaged config file for logrotate looks as follows. For specific settings, or different filenames, see the man page.
#
# Logrotate config file for MailerQ
#
# This file contains settings to automatically rotate the logfiles
# using logrotate. To see all possible settings, use `man 8 logrotate`.
#
# Settings for all the user formatted logfiles. If you renamed logfiles, or the files
# are in a non-standard location, alter the glob below.
/var/log/mailerq/*.log {
# General settings, may be altered.
rotate 20
daily
compress
missingok
notifempty
maxsize 100M
# Required setting for normal logfiles, because MailerQ monitors the filesystem
# and needs time to close the writers to a logfile
delaycompress
}
# Settings for all the binary logfiles. If you renamed logfiles, or the files
# are in a non-standard location, alter the glob below.
/var/log/mailerq/*.bin {
# General settings, may be altered.
rotate 20
daily
missingok
notifempty
maxsize 100M
# Required setting for binary logfiles, because MailerQ reads out the logfiles for the
# web interface and processing purposes.
nocompress
}Note that logrotate is not required. In fact, any application can simply remove the logfiles, and MailerQ opens it almost immediately. Naturally, if an application removes a logfile, the application should wait until MailerQ has finished writing and opened a new logfile.
Information about MailerQ is logged to the application log. This includes failures like databases that are suddenly offline or RabbitMQ connections that are lost. Currently the log levels Info, Warning and Critical are supported. The error log has been deprecated in favor of the application log. The application log can be set using the option application-log, and supports logging to stdout, stderr, syslog and files. If this setting is kept empty no logs are stored.
application-log: syslog,/var/log/mailerq/mailerq.logFiles must, of course, be writable for MailerQ.
You can use the Smart-TPL templating language to define your own log lines. With this template language, you can use some simple logic to log specific errors. You can specify this in the configuration file.
Example configuration:
send-log-format: [{$date}] {$result.state}:{$result.result} {$attempt} <{$email.envelope}> <{$email.recipient}> [{$result.from}] [{$result.to}] {$email.tags} {$result.code} {$result.status} {$result.description}Example output:
[2018-08-29 07:20:16] message:accepted 1 <info@company.com> <info@outlook.com> [100.20.20.20] [100.30.30.30] [ "a", "b" ] 250 2.0.0 Ok (edited)You can only use the Smart-TPL templating language for the send-log-format, received-log-format, download-log-format and the www-log-format settings.
You can only create your own log lines with the available variables and properties. Access to variables and properties is available via this syntax:
Access to variable variable = {$variable}
Access to property variable.property = {$variable.property}
With the help of logical constructors, you can log specific errors.
{if $variable}Print:{$variable} {else}show something else{/if}For more information about this templating language, you can read the Smart-TPL documentation.
MailerQ creates log files for all delivery attempts. The directory in which these log files are stored can be set here, as well as the name of the log file.
The management console web interface uses binary log files to display real-time information of all delivery attempts.
The configuration file holds the following options:
# Log of send attempts
send-bin-log-file: /var/log/mailerq/send.bin
send-log-file: /var/log/mailerq/send.log
send-log-format: [{$date}] {$result.state}:{$result.result} {$attempt} <{if $email.envelope}{$email.envelope}{/if}> <{$email.recipient}> [{$result.from}] [{$result.to}] {$email.tags} {$result.code} {$result.status} {$result.description}You can specify your own log lines via the send-log-format setting. These are the available variables.
$email Email JSON object, as placed in queue.
$result Last result object, of this connection
$attempt int, attempt number of this mail
$date formatted DateTime (e.g. 01-01-1970 11:53)The available properties are specified in the JSON for outgoing messages.
The "send-log-file" is the location for the send log. The default is "send.log". Its directory must be writable for MailerQ.
Incoming messages - messages sent to the SMTP port of MailerQ - are logged to the received log file. These settings work exactly like the send log settings:
# Log of incoming messages
received-bin-log-file: /var/log/mailerq/received.log
received-log-file: /var/log/mailerq/received.log
received-log-format: [{$date}]{if $json.message_id} {$json.message_id} {/if}<{$json.envelope}> <{$json.recipient}> [{$json.connection.remote_ip}] [{$json.connection.local_ip}]You can specify your own log lines via the received-log-format setting. These are the available variables.
$json The received / constructed JSON
$report (boolean) whether or not this is a delivery report
$date The datetime
$time unix timestampThe available properties are specified in the JSON for incoming messages
The log file only holds messages that are received over the SMTP port. Messages dropped in the spool directory or that are injected using the command line interface are not logged.
MailerQ can create MIME message itself. To do this, MailerQ sometimes has to download resources from the internet (like images that should be embedded, or attachments that should be included in the mail). All these download operations are written to the download log.
The settings for the download log are essentially the same as the settings for the send-log:
# Log of downloads
download-bin-log-file: /var/log/mailerq/download.bin
download-log-file: /var/log/mailerq/download.log
download-log-format: {$date}, {$time}, {$url}You can specify your own log lines via the download-log-format setting. These are the available variables.
$url The url
$time The unix timestamp
$date The formatted DateTime (e.g. 01-01-1970 11:53 oid)You can monitor who accessed the management console. The settings for the management console log are essentially the same as the settings for the send-log:
# Log of management console
www-bin-log-file: /var/log/mailerq/download.bin
www-log-file: /var/log/mailerq/
www-log-format: {$date}, {$ip}, {$request}You can specify your own log lines via the www-log-format setting. These are the available variables.
$date The formatted DateTime (e.g. 01-01-1970 11:53 oid)
$ip The IP that connected
$request A string with the request (e.g. "GET/settings")From MailerQ 5.6 and onward, user formatted logs can be combined. Simply specify the same filename as a target for multiple logs. For example, to combine the user formatted send/receive logs, the config settings below can be used.
send-log-file: /var/log/mailerq/combined.log
received-log-file: /var/log/mailerq/combined.logThis only works for a single instance. If multiple instances point to the same file, no guarantees about data integrity will be made. Same holds for previous versions on MailerQ (< 5.6), there was no logic to guarantee data integrity even on the same instance. This works fine for smaller loglines, but it depends on the platform and the size of the logged information when this corruption will occur.
MailerQ writes all events to RabbitMQ message queues. The recommended way of handling events is therefore to write scripts or applications that process the data from these message queues. This is much more powerful than periodically processing log files:
For all of the above reasons, the first MailerQ versions did not even have logging capabilities. We did not want to slow down our high-performance MTA by having it write data to disk - it was up to other scripts and applications to read the results from the message queues, react to them and write data to appropriate log files. With this architecture, hiccups in disk IO could not slow down email deliveries.
However, although this worked (and works!) perfectly for us, we found out that many users still want to have log files to monitor what is going on. So we've added the logging feature, and we use threads to prevent that IO hiccups can be a problem. But keep in mind that once you find yourself writing cron jobs to process log files, you would probably be better off writing a script that processes data from the message queues.
To run MailerQ, you need a valid license file. You can get free trial license files and commercial license files from the MailerQ website.
If you already have a license, but want to install a newer version of MailerQ, you need a new license file. This new license file can also be retrieved for free via the above hyperlink.
It is best to store your license file as "etc/mailerq/license.txt". If you store it in a different location, you have to update your config file.
If you have questions about your license, feel free to send an email to info@mailerq.com.
MailerQ picks up messages from the outbox queue, tries to send them out and then publishes the results back to result queues. Other programs or scripts can read out these result queues and further process the failures and succesful deliveries. The messages on the result queues are all JSON encoded.
MailerQ uses four different queues to which results are published. Based on the type of result (success, failure or retry) a message is published to one or more of these queues:
The "results" queue is used by MailerQ to publish all the results: both the successful results as well as the failures. The "success" and "failures" queues are used for just the successes and just the failures. These queues receive a subset of the messages published to the "results" queue. The "retry" queue is used for transient results for deliveries that have not yet failed or succeeded, and that are going to be retried.
Depending on your needs, you can enable or disable these different queues (for example, if the only thing you're interested in are the failures, you do not configure the results, success and retry queues, and only set the failures queue). The result queues can be configured in the RabbitmQ section in the config file.
A typical delivery scenario is this: you first publish your JSON message to the outbox queue. MailerQ picks up this message and tries to send it. If this send attempt fails (because the receiving server rejected it with a soft error), MailerQ modifies the initial JSON to include info about this first soft error, and publishes it back to the outbox queue to retry the delivery later. The message is also sent to the retry queue to notify external programs that the message will be retried.
After a while, MailerQ picks up the message from the outbox queue again for a second attempt. This time the delivery succeeds and the message is modified once more to also contain info about this correct delivery, and the message is published to both the success queue and the generic results queue.
The messages that MailerQ publishes to the results queues are JSON encoded, and hold the same properties as the original message that was read from the outbox queue, including all the custom properties that you added (thus: if you added some custom identifier to the initial JSON object, it will also be present in the JSON object that ends up in the result queue.
The "mime" property is often stripped from the initial JSON before it is published to a result queue, so that these queues do not take up too much space from RabbitMQ. However, if you want to keep the mime data in the JSON, you can add a "keepmime" property. For more info about the supported properties, check the JSON specification for outgoing messages.
Besides stripping the MIME data, MailerQ also adds a "results" property to the JSON. This "results" property holds an array with information about each delivery attempt that was taken. A result message will thus look something like this:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"results": [
{
"state": "intro",
"result": "error",
"time": "2016-02-04 13:45:15",
"mta": "mta1.example.com",
"from": "1.2.3.4",
"to": "10.11.12.13",
"messages": 1,
"code": 451,
"status": "5.0.0",
"description": "Please come back later",
},
{
"state": "message",
"result": "accepted",
"time": "2016-02-04 14:01:25",
"mta": "mta2.example.com",
"from": "1.2.3.4",
"to": "10.11.12.14",
"messages": 4,
"code": 250,
"status": "2.0.0",
"dsn": true,
"description": "Requested mail action okay, completed",
}
]
}The "results" property holds an array of JSON objects. Each object contains the result of one delivery attempt. Every result object can have the following properties:
| state | state in the delivery where the error occured |
| result | type or result (accepted,timeout,lost,error,invalid) |
| time | time of the result |
| mta | name of the receiving mta (reported when the connection was set up) |
| from | ip from which the mail was sent |
| to | ip to which the mail was sent |
| authentication | if authentication/login was necessary for access, the authentication protocol used |
| cipher | the encryption protocol used (if the message was sent over a secure connection) |
| certificate | information about the ssl certificate that was issued by the receiving server |
| messages | number of messages sent over the same connection |
| code | SMTP result code |
| status | extended SMTP status code |
| description | answer received from receiving server |
| dsn | did the remote server implement the dsn protocol? |
Note that some properties are optional. For example, the properties "to", "from", "mta" and "messages" are only used if an actual TCP connection was set up and are not present if the mail failed because of a DNS lookup failure. The "code", "status" and "description" properties are only used when the remote server returned an error code and/or human readable description to explain why the message was or was not accepted.
The "mta" property holds the name of the MTA that accepted the message. To get this mta name, MailerQ reads the answer from the initial welcome message that the receiving server sent right after the connection was set up.
The "dsn" property is only used for succesful deliveries and tells you whether the incoming server supports the DSN extension. If this is set to true, and you had also included a DSN setting in your input JSON to request a success notification, you can expect a notification from the server that accepted the message.
For logging purposes, MailerQ also extracts some data from the MIME when publishing the emails back. This way, these properties can be logged using the flexible log format.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"results": [
{
"state": "message",
"result": "accepted",
"time": "2016-02-04 14:01:25",
"mta": "mta2.example.com",
"from": "1.2.3.4",
"to": "10.11.12.14",
"messages": 4,
"code": 250,
"status": "2.0.0",
"dsn": true,
"description": "Requested mail action okay, completed",
}
],
"extracted": {
"from": "my-sender-address@my-domain.com",
"to": ["info@example.org", "other@example.org"],
"subject": "Test Subject",
"headers": {
"x-example-property": "some-mime-property"
}
}
}As is shown in the example, some properties are exposed in a parsed way (from, to, subject), as well as unparsed exposure of all the MIME-headers that were present in the (personalized) email.
MailerQ logs the state in which the error occurs (this is the "state" property), and the type of result in that state. An email that is processed by MailerQ goes through the following states:
| process | the input is checked and assigned to the from and to ip address between which it will be sent |
| storage | the mime data is loaded from external nosql storage |
| responsive | the json message is converted into a responsive email |
| personalize | the message is personalized if personalization data is available |
| dns | the hostname to which the mail should be sent is looked up in DNS |
| bind | a local IP address is chosen from which the mail is going to be sent |
| connect | a tcp connection is set up |
| intro | the TCP connection has been set up, waiting for initial welcome message from server |
| ehlo | the "EHLO" message has been sent |
| helo | the "HELO" message has been sent |
| starttls | the "STARTTLS" message has been sent |
| authplain | the "AUTH PLAIN" message has been sent |
| authlogin | the "AUTH LOGIN" message has been sent |
| authusername | the username for authentication has been sent |
| authpassword | the password for authentication has been sent |
| authcram | the "AUTH CRAM-MD5" message has been sent |
| authresponse | the response to the cram-md5 challenge has been sent |
| mailfrom | the "MAIL FROM" message has been sent |
| rcptto | the "RCPT TO" message has been sent |
| data | the "DATA" message has been sent |
| message | the full mime data followed by a dot has been sent |
| unknown | the connection is in an unknown state, for example when a pipeline failed |
During all of the above states errors might occur. If this happens, the state of the delivery and the type of error is set added to the result JSON. A successful delivery has gone through all the states, and is published to the result queue with the property "state" set to "message" and "result" set to "accepted".
During all the above states errors can occur. The type of error is logged in the "result" property. It always has one of the following values:
| accepted | message was accepted (only reported in combination with state "message") |
| timeout | no answer was received in time |
| lost | connection was lost while waiting for an answer |
| invalid | an answer was received, but it could not be recognized as valid answer |
| error | an answer was received, but the answer contained a fatal error message |
The SMTP protocol requires communication between two mail servers: a sending server and a receiving server. MailerQ is the sending server, and sends instructions to the receiving server (like "MAIL FROM", "RCPT TO", et cetera). After each of these instructions it waits for an answer from the remote server.
If the answer is not received in time, the "timeout" result is used. It is also possible that the receiver did sent back an answer, but that the answer could not be recognized as a valid SMTP response message. In such a case we use the "invalid" result type. When the TCP connection was lost while waiting for an answer the "lost" type is result, and the "error" type is used in case the receiver server sent back a valid SMTP answer, but the answer was that the message was rejected.
If MailerQ's instruction was accepted, the delivery moves to the next state, and no result is logged in the JSON object.
Not every result state is linked to a state in the SMTP protocol. The following combinations can also occur, and should be interpreted in a special way:
| state | result | |
| process | timeout | incoming message was expired (maxdelivertime or maxattempts exceeded) |
| process | error | the input json contains a mail that could not be assigned to an ip address (for example because no sufficient secure ip exists) |
| process | invalid | the input json contains a mail that could not be sent (for example due to license constraints, or because the recipient is missing or not parsable) |
| storage | lost | smtp connection was lost while loading data |
| storage | invalid | smtp response was received while loading data |
| storage | error | no data was found |
| responsive | error | responsiveemail.com algorithm could not turn json into mime |
| dns | error | dns lookup failure (domain does not exist / no ip addresses available for accepting mail) |
| bind | error | no local ip addresses available for sending mail |
MailerQ does its best to fill in an appropriate error code in the result object too. Be aware that the error code and status code sometimes come from the remote server and is blindly copied by MailerQ to the result JSON, while in other situations it is generated by MailerQ.
If the receiving server supported STARTTL, and MailerQ managed to secure the connection, the result object also holds an "cipher" and "certificate" key. The "cipher" holds the settings of the encryption algorithm that was used, while the "certificate" holds the settings of the SSL certificate that was issued by the receiving server. Both the "cipher" as the "certificate" are stored as JSON objects, holding all sorts of sub values.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "john.doe@gmail.com",
"results": [
{
"state": "message",
"result": "accepted",
"time": "2016-07-20 14:01:25",
"mta": "mx.gmail.com",
"from": "1.2.3.4",
"to": "10.11.12.14",
"messages": 1,
"code": 250,
"status": "2.0.0",
"dsn": false,
"description": "Requested mail action okay, completed",
"cipher": {
"name": "ECDHE-RSA-AES128-GCM-SHA256",
"bits": 128,
"algbits": 128,
"version": "TLSv1\/SSLv3",
"id": 49199
},
"certificate": {
"notbefore": "2016-07-13 14:40:53",
"notafter": "2016-10-05 14:17:00",
"version": 3,
"verified": true,
"result": "ok",
"keyusage": 1,
"self-signed": false,
"subject": {
"common": "mx.google.com",
"country": "US",
"locality": "Mountain View",
"state": "California",
"organization": "Google Inc"
},
"issuer": {
"common": "Google Internet Authority G2",
"country": "US",
"organization": "Google Inc"
},
"altnames": [
"gmr-smtp-in.l.google.com",
"gmr-mx.google.com",
"gmail-smtp-in.l.google.com",
"aspmx5.googlemail.com",
"aspmx4.googlemail.com",
"aspmx3.googlemail.com",
"aspmx2.googlemail.com",
"aspmx.l.google.com",
"alt4.gmr-smtp-in.l.google.com",
"alt4.gmail-smtp-in.l.google.com",
"alt4.aspmx.l.google.com",
"alt3.gmr-smtp-in.l.google.com",
"alt3.gmail-smtp-in.l.google.com",
"alt3.aspmx.l.google.com",
"alt2.gmr-smtp-in.l.google.com",
"alt2.gmail-smtp-in.l.google.com",
"alt2.aspmx.l.google.com",
"alt1.gmr-smtp-in.l.google.com",
"alt1.gmail-smtp-in.l.google.com",
"alt1.aspmx.l.google.com",
"mx.google.com"
]
},
}
]
}With the response pattern system, properties can be added to the result json. This is useful for your own classification and processing of messages, but there are also some special properties that can be used to override error handling logic.
| fatal | should be the message be retried, or was this result already a permanent failure |
| individual | override whether or not an error applies to a single message, or rather to any message sent to the domain |
| strange | is this is a strange type of result |
| greylisted | should this result be sent from the same ip later |
| fallback | should this result trigger a fallback server |
These properties are not set by MailerQ itself, but can be set by setting these properties in a response pattern. If they are not present, MailerQ tries to figure out these properties by reasoning about the response it got from the server.
MailerQ expects all messages that are loaded from the RabbitMQ outbox queue to be JSON encoded. This means that if you want to inject email directly into RabbitMQ, you must use JSON. In this JSON you should set the envelope ("MAIL FROM") address, the recipient ("RCPT TO") address and the full mime data to be sent.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "..."
}Strictly speaking, the "envelope" property is not even required, because the SMTP protocol also permits deliveries without an envelope address ("MAIL FROM:<>"). However, without an envelope you cannot receive bounces, delivery status notifications and/or out-of-office replies.
To make the examples a little easier to read, we often remove the mime data from the example JSON, and replace it with "...".
Besides the three properties demonstrated above, you can add all other kind of other properties to the JSON object to control the delivery of the mail. The following properties are recognized by MailerQ:
| envelope | envelope ("MAIL FROM") address |
| recipient | recipient ("RCPT TO") address |
| mime | full mime data to be sent |
| key | key to the mime data in external storage |
| keepmime | do not remove mime data after delivery |
| data | personalization data |
| ips | ip addresses to send the message from |
| pool | pool identifier to send the message from |
| nextattempt | properties for when the next attempt should be |
| maxdelivertime | time until which a delivery should be retried |
| maxattempts | max number of delivery attempts |
| retries | the delays between the delivery attempts |
| force | force delivery, even when errors occur or conversion is impossible |
| inlinecss | turn style blocks in html mails into inline style attributes |
| dkim | private keys to sign the mail |
| dsn | settings for delivery status notifications |
| queues | alternative rabbitmq queues for results |
| smarthost | smarthost settings |
| tags | the tags to add to the message |
| headers | add or change the mime headers |
| priority | the priority of the message |
The most simple JSON to send to RabbitMQ contains just an "envelope", "recipient" and "mime" property as we demonstrated above. The "mime" property should be a string value, holding a valid MIME object. This mime object holds the entire email, including all the headers and possible text and html versions, attachments, and so on:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\nTo: info@example.org\r\nSubject: ..."
}However, the "mime" property can also be a nested JSON object, holding all individual properties of the mime, which will then be turned into a valid mime string by MailerQ:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": {
"from": "my-sender-address@my-domain.com",
"to": "info@example.org",
"subject": "This is the subject line",
"text": "text version of the mail"
}
}The number of properties that are supported inside the nested "mime" property is pretty huge. It uses the very same algorithm as the responsiveemail.com web service to convert JSON objects into valid MIME data. To give you an idea of a possible valid JSON input, consider this:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": {
"from": "my-sender-address@my-domain.com",
"to": "info@example.org",
"subject": "Example subject",
"textVersion": "This is the example message text",
"headers": {
"x-my-special-header": "special-value"
},
"content": {
"blocks": [ {
"type": "image",
"src": "http://www.example.com/logo.png"
}, {
"type": "button",
"label": "Click the button!",
"link": {
"url": "http://www.mailerq.com"
}
} ]
},
"attachments": [ {
"url": "http://www.example.com/a-special-file.pdf",
"name": "brochure.pdf"
} ]
}
}The number of options is huge. You can construct responsive emails using images, HTML text, social media, et cetera. MailerQ downloads all the resources and converts the JSON code into a valid MIME strings.
For more information about the supported JSON properties nested under the "mime" property, check the responsive email documentation.
MIME bodies can become large, especially if your emails contain attachments or embedded content. To prevent that such big messages have to be processed by RabbitMQ, MailerQ can be configured to use an external message store (for example: MongoDB). You can then publish much smaller JSON objects to RabbitMQ, and keep the full MIME messages in the external message store.
MailerQ waits with loading the message from message store until the SMTP connection has been set up, so no time and resources are wasted on fetching information that is not needed. The mime data is only loaded when it is really needed.
With an external message store, you can use the "key" property instead of the "mime" property. This property holds the key under which the mime data is stored in your external storage system.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"key": "message-store-key-where-data-can-be-found"
}Even when you have a message store you can of course still use the "mime" property and store the full mime in the JSON. This is often even the recommended way of doing things, because most mails get delivered at the very first attempt, and storing-and-fetching the mail to and from external storage is then only a waste of time and resources. The message store is in such a setup only used for messages that were delayed and that are published back to RabbitMQ. MailerQ takes care of removing the mime from the JSON, and stores it in the message store when delayed messages are published back to RabbitMQ.
You may be injecting different types of messages into MailerQ, for example messages belonging to your various customers or campaigns. To help you get a better overview, MailerQ allows you to tag your messages with one or more labels of your choosing. These tags will show up in the management console, allowing you to monitor and control all deliveries belonging to a tag. To specify the tags in the JSON you can add a "tags" property with an array containing all the labels:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"tags": ["Customer Name", "Example Campaign"]
...
}This is just an example, you are free to add any number of tags to your messages and interpret them as you wish. There are virtually no restrictions on the format of a tag; all characters including punctuation and international characters are accepted in the tag specification as long as the message is valid JSON.
When a message has been completely processed - either because it was successfully delivered, or the delivery failed - MailerQ publishes the delivery result to the results queue, where you can pick it up for further processing.
By default, MailerQ throws away the mime data to make room in the JSON object and in the message store. The result queues only hold the meta data for each sent message, but not the full mime data. If you do not want the message data to be removed, you can indicate this by adding the "keepmime" option:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"keepmime": true
}It is possible to include personalization data in the JSON. If you do this, MailerQ will treat the subject, html and text version of your email as templates, and replaces variables in it with the values loaded from the JSON.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: ...\r\nTo: ...\r\nSubject: Message from {$firstname}\r\n...",
"data": {
"firstname": "Emiel",
"lastname": "Bruijntjes",
"email": "emiel.bruijntjes@copernica.com"
}
}Personalization variables can currently only be used for the subject and the text and HTML versions of the mail. However, if you assign a nested object to the "mime" property using the responsive email specification, you can use personalization data in almost all fields.
For more information about using personalization, check our personalization documentation.
MailerQ can send out mail from all IP addresses configured on your server and will always try to make as many connections as it takes to send out your mailing. This can be useful to increase the delivery rate, because receivers often restrict deliveries per IP address.
It is however wise to limit the range of local IP addresses that a mailing can use, as you may not want to expose all your IPs to every campaign or group of mailings you send. The reputation of your IPs is important for getting your mailings delivered, so you may not want to risk that a single dubious campaign greylists all your IP addresses. (Especially if these messages are provided by third parties) Setting a predetermined range in the JSON can help prevent these sorts of problems.
To limit the range of IP addresses that a specific mailing can use, you may specify an array of "ips" as a property in the JSON body:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"ips": ["231.34.13.156", "231.34.13.158"]
}MailerQ will then only try to send the message from these IPs. Note that you can of course only use addresses that are actually bound to the host that MailerQ runs on. If the array provided is empty or if it only contains IP addresses that are not bound to your server, this will result in a failed delivery.
If no property "ips" is set in the JSON or if it is set to something other than a JSON array, MailerQ will fall back on a list of default IPs, which can be defined in the config with the property "smtp-defaultips":
smtp-defaultips: 1.2.3.4; 1.2.3.5; 1.2.3.6If this property is missing or left empty, MailerQ will instead opt to use ALL available IP addresses on the host server. However, we do recommend recommend that you set this property if you have many IPs configured on your server.
Outside of being able to set single IP addresses, a pool can also be specified. First, setup an IP pool in the Management Console and assign some IPs to the pool. Then, to set the pool, you may simply set the identifier in the json:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"pool": "example-pool"
}MailerQ will then only use the IPs that are currently assigned to that pool. This way, if an IP performs poorly, it can easily be removed from the pool and MailerQ will no longer use it for the messages with the pool assigned, instead of having to rewrite every specific message to remove the IP where applicable.
Note that the pools are mutually exclusive with the IPs, that is, if the ips property is set in the JSON, the pool
property will be ignored, as a specific set of IPs is more specific than a pool setting.
Messages loaded by MailerQ are delivered right away. If you want to delay a delivery, you can add a "nextattempt" property to the JSON with the desired time of delivery for the next attempt.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"nextattempt": {
"time": "2019-01-10 00:00:00"
}
}In normal circumstances you want messages to be delivered right away, so you do not need this "nextattempt" property. However, under the hood, MailerQ uses this "nextattempt" property when a mail cannot be delivered because of greylisting. The message is then published back to the outbox queue with a "nextattempt" property set to a couple of minutes after the initial attempt.
This property is also set when response patterns cause a mail to be retried again after a specified interval.
When a message cannot be delivered immediately because of unresponsive receivers, greylisting or throttling, MailerQ publishes back the email to the outbox queue for later delivery. This can result in emails that are sent much later than the time that you first added them to the message queue.
If you do not want an email to be delivered after a certain time, all you need to do is add the "maxdelivertime" property. MailerQ will give up on delivering the mail after this time, and send a failure to the results queue.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"maxdelivertime": "2016-02-10 00:00:00",
"maxattempts": 3,
"retries": [ 600, 600, 1800, 3600 ]
}The "maxattempts" is an alternative way to control after how many attempts MailerQ should give up trying. The "maxattempts" setting limits the number of attempts MailerQ makes before it gives up. You normally should not use this setting because "maxdelivertime" is much more precise.
Timestamps in MailerQ are always in UTC. MailerQ is not very tolerant in parsing timestamps, so make sure that you use the right formatting (YYYY-MM-DD HH:MM:SS). If you do not specify an explicit max delivery time or max attempts setting in the JSON, and a default value has also not been set in the config file, MailerQ will attempt to deliver the mail within 24 hours after the mail was first picked up from the outbox.
The "retries" property can be added to specify exactly how long MailerQ should wait until the next delivery of the mail. This should be set to a JSON array holding the number of seconds between the attempts. In the above example, the first and second retry are started 10 minutes after the previous attempt. If the third attempt (which is the second retry) fails too, MailerQ will wait half an hour (1800 seconds). If that attempt fails too, all subsequent attempts will be sent with one hour intervals (3600 seconds).
When you send out HTML emails, you face the problem that not all email clients support stylesheets that are set in the header. Some email clients (especially web based clients) strip out the CSS code from the HTML header. This often messes up the layout and look of your messages. To overcome this, it is better not to set CSS settings in the header in the first place, but use "style=..." attributes in the HTML code.
MailerQ can do this automatically. If you set "inlinecss" property to true, MailerQ parses the HTML email, and converts the CSS code from the HTML header into inline "style=..." attributes in the HTML body.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"inlinecss": true
} You can include private DKIM keys in the JSON to let MailerQ sign the mail.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"expire" : "2017-01-01 00:00:00"
}
} It is also possible to include multiple keys in the JSON. The "dkim" property supports arrays:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": [ {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n....."
}, {
"domain": "example.com",
"selector": "y",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"expire" : "2017-01-01 00:00:00"
} ]
} Besides the private keys that you include in the JSON, MailerQ also keeps a set of private keys in its local database (and that can be edited using the management console). If there are multiple matching keys, they are all used for signing the mail.
By default, only regular headers are used for the signature: the "from" and "to" address, the "subject" and so on. If you want to include your own custom headers in the signature too, you can pass in an extra "headers" option:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"headers": [ "x-custom-header", "x-other-header" ]
}
} The headers that you included in the "headers" option are also signed. This is especially useful if you want to include the "feedback-id" header in a signature. This "feedback-id" header is required to be signed for Google/Gmail feedback loops.
If you want to receive reports from remote servers whenever a DKIM signature fails to verify you can add a flag to the DKIM object to indicate this:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"report": true
}
}Note that you will have to update your DNS records with the options defined in RFC 6651 for the remote server to know where it should send their reports.
A DKIM key can also be used for creation of ARC signatures. This is enabled by setting the "protocols" option. This option should be an array of the protocols you want to use (DKIM and/or ARC):
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": [ {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"protocols" : ["dkim", "arc"]
}, {
"domain": "example.com",
"selector": "y",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"protocols" : ["arc"]
} ]
} The first key is used for both DKIM and ARC signing, whereas the second key is only used for ARC signing.
A DKIM can be signed using different forms of canonicalization. Default is relaxed/simple.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": [ {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"canonicalization" : "simple/simple"
}, {
"domain": "example.com",
"selector": "y",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"canonicalization": "relaxed/relaxed"
} ]
} In the example, the first key is signed with simple/simple canonicalization, and the second
key is signed with relaxed/relaxed canonicalization.
The "dsn" property can be added to control whether MailerQ should send back an email to the envelope address in case of a failed delivery.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dsn": {
"notify": "FAILURE",
"ret": "HDRS",
"orcpt": "info@example.org",
"envid": "my-identifier",
}
}The above JSON contains a DSN setting that says that a delivery status notification should be sent back to the original envelope address in case of a failure (this is what the "notify" setting says). The notification should include the headers of the original mail (ret=HDRS). The notification should specify that the original recipient was "info@example.com", and the unique envelope identifier was "my-identifier".
For more information about delivery notifications, see the Delivery Status Notification documentation.
After MailerQ has processed an email, it publishes the mail to one or more result queues. The default queues to use for this are configured in the global configuration file, but you can set other queue names to tell MailerQ to use other result queues.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"queues": {
"results": "name-of-results-queue",
"failure": "name-of-failure-queue",
"success": "name-of-success-queue",
"retry": "name-of-retry-queue",
"dsn": "name-of-dsn-queue"
}
}All properties inside the "queues" object are optional. If you leave an option out, MailerQ will use the default queue from the config file. If you include a queue but set it to a null value, the queue will not be used. Thus, if you for example want to process only the errors for a certain e-mail, you can only set the "failure" queue, and set all other queues to null. When the delivery succeeds, MailerQ will silently discard the mail, without adding it to any result queue:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"queues": {
"results": null,
"failure": "name-of-failure-queue",
"success": null
}
}There is an important difference between not mentioning a queue in the "queues" object, and setting it to null. In the above example, the "dsn" and "retry" queues are missing from the "queues" object. This means that for these two queues the setting from the config file will be used (set with the "rabbitmq-dsn" and "rabbitmq-retry" options). For the "results" and "success" queues a null parameter was used, so that the settings from the config file are ignored, and no messages are ever published to the results or success queues.
If you do not want to send the message right to the recipient right away, but to an alternative SMTP server on the internet, you can add a "smarthost" option.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"smarthost": {
"hostname": "mail.smtpeter.com",
"port": 25,
"username": "my-username",
"password": "my-password"
}
}The above message will not be sent to "example.org", but to "mail.smtpeter.com" instead.
MailerQ can be used to alter the MIME headers. To do so, you can add a "headers" option.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: ex@ex.com\r\nTo: info@ex.com\r\nDummy: xyz\r\nSubject: ...",
"headers": {
"remove" : ["dummy", "bcc"],
"prepend" : {
"cc" : "cc@ex.com",
"x-example" : ["example 1", "example 2"]
},
"append" : ...,
"replace" : {
"from" : "xe@xe.com"
}
"update" : ...,
}
}The "remove" property can be a string or an array of the header fields that have to be removed from the mime. Using the "prepend" option one can prepend headers to the mime. Multiple headers with the same key can be prepended by supplying an array of values. The "append" (or "add") option holds the headers that have to appended (same syntax as prepend). The headers described in "replace" will replace the first occurrence and remove the other headers with the same key. If no existing header is found, it will simply be appended. The "update" option works similar, but it non-existing headers will not be appended.
To give the message a priority, the priority field should be used. This setting will only have
an effect if the rabbitmq-maxpriority is set in the config file, since the queues will then be
constructed as priority queues.
To have better control over your message queue, you can add additional properties in the JSON. These properties are ignored by MailerQ, but they will end up in the result queue, and allow you to link result data with the original mail.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"custom-property-name": "debug data",
"mime": "..."
}When an email is received by MailerQ, the mail is turned into a JSON object and published to the inbox queue. This happens to messages that are received on the SMTP port, but also to messages that come in via the spool directory or that were injected via the command line interface: every message is turned into a JSON object and published to the inbox queue.
The JSON messages that are published by MailerQ follow exactly the JSON specification for outgoing messages. It is therefore possible to let MailerQ publish these incoming messages directly to the outbox queue to deliver them to the final recipient.
MailerQ also adds some extra properties to the JSON of incoming messages that are not needed for the delivery of the email, but that are useful in case you write scripts or programs that process incoming messages. The following properties can be found on incoming messages:
| hostname | server name that received the message |
| received | time when the mail was received |
| message-id | unique message id generated for the mail (only used when received via smtp) |
| connection | connection info (only used when received via smtp) |
| spool | spool info (only used when received from spool dir) |
| cli | command info (only used when received from cli) |
| checks | array with the results of the SPF, DKIM and DMARC checks |
The above properties are never used by MailerQ when the email is sent, but they can be useful to recognize incoming messages.
When a connection comes in over an SMTP connection, MailerQ reports a unique internal identifier to the sender that can be used to track the email.
250-Ok, your message has been queued using the following identifiers:
250 gdsfu232 for info@example.comEvery recipient gets a unique identifier. This unique identifier is stored in the "message-id" property.
{
"message-id": "gdsfu232",
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "..."
}The "received" property holds the time when the email was received, while "hostname" holds the name of the server that received the message. These two properties are included for all injection mechanisms: SMTP, CLI, and spool.
If the mail came in via an SMTP connection, the generated JSON object holds properties about the TCP connection.
{
"message-id": "gdsfu232",
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"connection": {
"local-ip": "1.2.3.4",
"local-port": 25,
"remote-ip": "5.6.7.8",
"remote-port": 25324,
"secure": true,
"user": "mailerq"
}
}If MailerQ runs behind a HAProxy server, the connection data is extracted from the PROXY header that is sent by the HAProxy server.
If the TCP connection is secure and some sort of SMTP authentication mechanism was used, the "user" property holds the username of the user who submitted the message.
For mails injected to the spool dir, the JSON holds properties that identify the file from which the message was loaded.
{
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"spool": {
"directory": "/path/to/directory",
"file": "filename.mime",
"user": "owner",
"size": 23299
}
}The properties speak for themselves: the spool directory path, name, owner and size of the injected file.
Message injected via MailerQ as command line utility hold information about the started program:
{
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"cli": {
"command": "/usr/bin/mailerq",
"arguments": [ "--extract-recipients", "--ignore-dot" ],
"user": "john"
}
}In the above example, the email was injected using MailerQ as command line utility. The program was started by user "john", and two options were passed: "--extract-recipients" and "--ignore-dot".
Message injected via MailerQ REST API hold information about who connected
and the name of the authentication token in the http property.
{
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"http": {
"authentication": "rest token name",
"remote-ip": "1.2.3.4",
"remote-port": 12345
}
}If you enable checks for incoming messages, MailerQ runs SPF, DKIM and DMARC checks on each incoming message. The result of these checks are added to the JSON.
{
"message-id": "gdsfu232",
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"connection": {
"local-ip": "1.2.3.4",
"local-port": 25,
"remote-ip": "5.6.7.8",
"remote-port": 25324,
"secure": true,
"user": "mailerq"
},
"checks": [ {
"spf": "pass",
"smtp.mailfrom": "whatever@example.com"
}, {
"dkim": "pass",
"domain": "example.com",
"selector": "dkim",
"header.i": "fsjfksjfslkdf",
"header.b", "sodfidjsfdsjfsfjs"
}, {
"dmarc": "pass",
"header.from": "whatever@example.com"
} ]
}The properties for each test depend on the type of test, and hold the input parameters that were used to execute the test.
The checks are only executed for messages that are injected via SMTP, and only if checks are enabled in the config file using the "smtp-check" variable.
This example assumes that MailerQ is running and configured. It uses the Java client library provided by RabbitMQ.
The Java example consists of two files; Send.java and Result.java. If you want
to use these scripts, save the two files inside the same directory.
SendThis class connects to RabbitMQ and places a JSON encoded message on the outbox
message queue. To connect to RabbitMQ and to create the JSON encoded message,
the program uses the private variables defined inside the class. If you want
to use the Java example to test your MailerQ configuration, you will have to run
the Send class before running the Result class.
Send.java// import the rabbitmq libraries that are needed to connect to RabbitMQ
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
// import the rabbitmq lbrary that is needed to create the RabbitMQ channel
import com.rabbitmq.client.Channel;
/**
* Simple Java class that creates a connection to a RabbitMQ server and
* places a message on the queue, so MailerQ can take this message from
* the queue and send it out as an email.
*/
public class Send {
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static String hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static String username = "guest";
// same as the rabbitmq-password option in the config file
private static String password = "guest";
// same as the rabbitmq-vhost option in the config file
private static String vhost = "/";
// same as the rabbitmq-outbox option in the config file
private static String outbox = "outbox_test";
/**
* Settings for sending out the test email
*/
// domain where the test message should be delivered
private static String recipientDomain = "example.org";
// email where the test message should be delivered
private static String recipientEmail = "info@example.org";
// address where the email was sent from
private static String fromAddress = "me@my-domain.com";
/**
* Main method
*/
public static void main(String[] argv) throws Exception {
// create a new ConnectionFactory
ConnectionFactory factory = new ConnectionFactory();
// set the host to RabbitMQ
factory.setHost(hostname);
// set the user name to connect to RabbitMQ
factory.setUsername(username);
// set the password to connect to RabbitMQ
factory.setPassword(password);
// set the virtual host to RabbitMQ
factory.setVirtualHost(vhost);
// create the new connection
Connection connection = factory.newConnection();
// create the channel
Channel channel = connection.createChannel();
// declare the queue
channel.queueDeclare(outbox, true, false, false, null);
// create the message
String message = "{ "\"envelope\": \"" + fromAddress + "\", "
+ "\"recipient\": \"" + recipientEmail + "\", "
+ "\"mime\": \"" + "From: " + fromAddress + "\r\nTo: "
+ recipientEmail
+ "\r\nSubject: Example subject\r\n\r\n"
+ "This is the example message text\" }";
// publish the message on the queue
channel.basicPublish("exchange1", "key1", null, message.getBytes());
// some output
System.out.println("Sent: '" + message + "'");
// close the channel
channel.close();
// close the connection
connection.close();
}
}This class connects to the RabbitMQ server and gets the messages that were
placed on the outbox message queue by the Send class back from the result
message queue. The result messages from the result queue are shown to the user.
The Result class can only output any relevant information when it is executed
after the Send class was run. The Result program will keep running, waiting for
new messages until it is terminated (use Ctrl-C). You could run the Send class
from another terminal, to test the effect.
Result.java// import IOException
import java.io.IOException;
// import the rabbitmq libraries that are needed to connect to RabbitMQ
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
// import the rabbitmq lbrary that is needed to create the RabbitMQ channel
import com.rabbitmq.client.Channel;
// import the rabbitmq library that can be used to buffer the message that
// was pushed by the RabbitMQ server
import com.rabbitmq.client.QueueingConsumer;
public class Result
{
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static String hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static String username = "guest";
// same as the rabbitmq-password option in the config file
private static String password = "guest";
// same as the rabbitmq-vhost option in the config file
private static String vhost = "/";
// same as the rabbitmq-results option in the config file
private static String resultbox = "results_test";
public static void main(String[] argv)
throws java.io.IOException, java.lang.InterruptedException
{
// create a new ConnectionFactory
ConnectionFactory factory = new ConnectionFactory();
// set the host to RabbitMQ
factory.setHost(hostname);
// set the user name to connect to RabbitMQ
factory.setUsername(username);
// set the password to connect to RabbitMQ
factory.setPassword(password);
// set the virtual host to RabbitMQ
factory.setVirtualHost(vhost);
// create the new connection
Connection connection = factory.newConnection();
// create the channel
Channel channel = connection.createChannel();
// declare the queue
channel.queueDeclare(resultbox, true, false, false, null);
// message telling the user that the script is waiting for messages
// that will be placed on the result queue by MailerQ
System.out.println("Waiting for messages to be placed on the result queue. To exit press CTRL+C");
// create the QueueingConsumer
QueueingConsumer consumer = new QueueingConsumer(channel);
// start the consumer
channel.basicConsume(resultbox, true, consumer);
// keep waiting for messages
while (true)
{
// get a delivery from the result queue
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
// get the message as a string
String message = new String(delivery.getBody());
// print a message from the queue on the screen
System.out.println("Received:");
System.out.println(message);
System.out.println("------------------------------------------");
}
}
}You can compile both of these classes with just the RabbitMQ client on the classpath.
$ javac -cp rabbitmq-client.jar Send.java Result.java
To run the the Sender class
$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Send
To run the the Result class
$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Result
MailerQ runs on Linux. To get MailerQ up and running on your own computer or on a server you need to take a couple of simple steps:
To make this very simple, we have made a quick start to download and install MailerQ.
MailerQ comes with a web based management console that you can use to monitor exactly what is happening, and to adjust all your delivery settings.
Below you will find a short list of our most frequently asked questions about our Mail Transfer Agent. For more information please see our documentation. If you cannot find what you are looking for, please contact support.
Before you download and install MailerQ, make sure you have a working RabbitMQ message broker (version 3.3.1+) . There are three ways to send email with MailerQ. However, posting messages directly to the AMQP message queue is the fastest way. Below you will find the 'technical' questions that are asked most frequent.
Yes, MailerQ supports IPv6. However, do note that not all receiving mail servers have support for IPv6. It is therefore recommended to use IPv4.
No, statistics, such as opens, clicks and more are handled outside of MailerQ. MailerQ focuses completely on the delivery process of your e-mails. All delivery related results are registered by MailerQ and returned to the RabbitMQ results queue. Here your own application can read and process these results.
There are many reasons why an email cannot be delivered (initially). MailerQ will automatically place these e-mails in the retry queue and send them again after a set amount of time. Each attempt will also add a new "results" property to the JSON so you will know exactly what happened during the delivery of the email.
After downloading and installing MailerQ, you will need a license to be able to use the software. This license holds a special 'key', the end date and the IP addresses you have specified as the ones you want to use.
You can find MailerQ licenses on the create License page. On this you can choose to create a Free License (which allows you to use one IP address, send 1000 messages a minute and expires after one month) or a Commercial License (unlimited messages, unlimited IP addresses, renewed montly or annually and professional support). Make sure you add your VAT number when purchasing a commercial license. After reading the terms and conditions press the 'create' button, pay for your license if applicable, and you can download your license.
On the license page of your personal profile you can renew your current license. Renewing a license will create the exact same license, with the same IP addresses, but with a different end date. Make sure you use the 'create new license' option when you want to add new IP addreses.
In the license page of your personal profile you can download any of your current and old licenses.
No. Once you have purchased your license we will not charge you anything extra for the services offered within that license.
After your plugin is loaded, MailerQ starts making calls to it. To do this, MailerQ simply tries to locate specific functions in the plugin shared object file, and if it finds one or more of these functions, it will make calls to them every time a certain event occurs. There is one function that is required, mq_version(), but all others are optional. You only have to implement the optional functions if you want your plugin to override the default MailerQ behavior.
| Function name | Description |
|---|---|
| mq_version() | Returns API version number to ensure that MailerQ and plugin are compatible |
| mq_initialize() | Called right after the plugin is activated, allows plugin to run initialization code |
| mq_cleanup() | Called right before MailerQ closes, allows plugin to run cleanup code |
Before we continue, a single paragraph about the function naming convention of the MailerQ API. There are two type of functions: The first type of functions are the functions that your plugin can implement, and that are called by MailerQ when certain events occur -- these are functions like the ones listed in the table above. For these type of functions we use all lowercase characters. The other type of functions are functions that are offered by the MailerQ API, and that can be called by your plugin. These functions all have the uppercase "MQ_" prefix, and use CamelCase notation. The declarations of this second group of functions can be found in the "mailerq.h" header file.
MailerQ is a multi-threading application, with multiple worker threads that are all busy sending and receiving emails. If you write a plugin, it is important to realize this, because you cannot always safely access the same global data from your plugin functions. Locking might be necessary if you access global data from these multiple threads, and it could be useful to use per-thread data.
For every worker thread, MailerQ creates a MQ_Context structure that is passed to most of the plugin callback functions. For every worker thread that is created, and for every worker thread that is stopped, MailerQ calls the mq_context_initialize() and mq_context_cleanup() functions (but of course only if these functions exist in your plugin). You can implement these functions to run your own per-thread initialization or cleanup code.
| Function name | Description |
|---|---|
| mq_context_initialize() | Called after a worker thread is started, allows plugin to run thread specific initialization code |
| mq_context_cleanup() | Called right before a worker thread is stopped, allows plugin to run thread specific cleanup code |
If you want your plugin to interact with incoming SMTP traffic, you can implement one or more of the functions from the following table. When MailerQ loads your plugin, it tries to locate any of the following functions, and when it can find one or more of these functions, it will make calls to them every time the relevant event occurs. All functions are optional, you only have to write the ones that you need.
| Function name | Description |
|---|---|
| mq_smtp_in_connect() | Called after an incoming TCP connection is established |
| mq_smtp_in_secure() | Called after an incoming TCP connection is turned into a secure connection |
| mq_smtp_in_authenticate() | Called after someone sends login credentials over an incoming SMTP connection |
| mq_smtp_in_mailfrom() | Called after the MAIL FROM instruction was received |
| mq_smtp_in_rcptto() | Called after the RCPT TO instruction was received |
| mq_smtp_in_data() | Called after MIME data was received over the incoming connection |
| mq_smtp_in_message() | Called after the entire message was received and is turned into a JSON message |
| mq_smtp_in_close() | Called after an incoming SMTP connection is closed |
If you want your plugin to interact with outgoing SMTP traffic, you can add one or more of the functions from the following table to your plugin. MailerQ will call them every time a message is being sent.
| Function name | Description |
|---|---|
| mq_smtp_out_data() | Called by MailerQ when the connection enters data state |
| mq_smtp_out_sent() | Called by MailerQ after all mime data has been sent |
MailerQ is compatible with a lot of languages. For a language to be compatible with the MTA, all that is needed is support for JSON and the language being able to communicate with RabbitMQ. Some of the languages where this is possible are: C#, Erlang, Java, .NET, Perl, PHP, Python, Phyton-Puka and Ruby.
For the examples below, it is asumed that MailerQ will be able to communicate with the RabbitMQ and MySQL servers.
Everything in MailerQ has a non-blocking nature. Things like network communications, database queries and timeouts have been implemented in a non-blocking fashion. This means that MailerQ does not block while it is waiting for an answer from a remote server or from a database, but that it continues with other tasks for as long as no answer is available.
In order to get most out of MailerQ and your plugin, all your plugin functions should be implemented in the same way, and should also respect this non-blocking nature. For example, if your plugin needs to download something, you should not make a blocking network call, because this will block the entire MailerQ process (or to be more precise: an entire MailerQ worker thread). You should be using the supplied MailerQ event loop instead.
Another thing to carefully consider is how many times your implemented function will run during the application lifetime, especially with regard to speed of your plugin. When you create the plugin, keep in mind that your implemented function will be called for every message and MailerQ may bottleneck on a slow plugin.
Every time MailerQ is processing or delivering messages, one or more of your plugin callback functions can be called. When a certain event occurs, MailerQ checks the loaded plugins in alphabetical order to see if a certain callback function is implemented in the plugin shared object file. If the function is indeed implemented, it will be called by MailerQ to hand over control to the plugin. For example, if three plugins "A", "B", and "C" are loaded and MailerQ is at the "DATA" state during a SMTP handshake, MailerQ will check all these three plugins to see if they have implemented the mq_smtp_out_data() function. For every plugin that has implemented the function, it will call that function. Plugins that have not implemented the function are skipped.
Plugin functions should return a boolean value. When a plugin returns "false", it means that the plugin does not take over control from MailerQ, and MailerQ can proceed with calling the next plugin, or if there are no other plugins, proceed with delivering or receiving the message. However, if your plugin returns "true", MailerQ will stop processing the message, and will continue doing other tasks. Your plugin now is in control, and will stay so until you hand the control back to MailerQ.
MailerQ passes a number of parameters to a plugin function. One of these parameters is a pointer to a MQ_Context. This MQ_Context pointer gives you access to the MailerQ event loop, and allows you register file descriptors and set timers. If your plugin wants to do a blocking network call, it can do this in a non-blocking fashion instead, and register the socket and a callback function. Your callback will be called as soon as the filedescriptor becomes active.
When your plugin is ready with its task - it has for example downloaded the appropriate resources, or has modified or filtered the message that is going to be delivered - it must give control back to MailerQ. It can do this in three possible ways, with one of the following methods:
| Function name | Description |
|---|---|
| MQ_continue() | Tell MailerQ to call the next plugin |
| MQ_complete() | Tell MailerQ to skip all further plugins |
| MQ_retry() | Tell MailerQ to restart calling all plugins |
After your plugin has done its work, it can tell MailerQ that it is ready and that the next plugin can be called (this is what normal plugins do), or it can instruct MailerQ to skip all the other plugins and continue with the normal work, or that MailerQ should start all over again with calling the plugins.
Be aware that you really must call one of these functions to give back the control to MailerQ. If your plugin fails to return control, the message will not be picked up again by MailerQ for further processing, but will stay in memory. This will eventually lead to MailerQ running out of resources.
With the first of the listed functions, MQ_continue(), your plugin instructs MailerQ to advance in the list of plugins, and to call the next plugin. Naturally, all next plugins have the same options as this plugin, and the subsequent plugins will work with the message as adapted by the current plugin. If no other plugins exist, MailerQ will move on to the next step in the process.
When the MQ_complete() function is called, MailerQ advances to the next step in the process. This means that any plugins in the list after the plugin will be skipped and MailerQ moves on without calling these plugins.
Lastly, your plugin can call MQ_retry() to instruct MailerQ to restart its plugin choosing process in the current step, effectively restarting the step from the first plugin. Beware though, since MailerQ will restart from the beginning of the list, all plugins prior to the current one will run again, possibly causing the altered message to be processed again.
Warning: You should take extra care when returning control of the message to MailerQ with the retry function, because it is possible to inadvertently create a loop. For most plugins, MQ_continue() is the best function to give back control to MailerQ.
MailerQ calls a plugin callback function. This function should return true if it wants to take over control from MailerQ (for example to do a non-blocking asynchronous operation). When it is ready and wants to hand over control to the next plugin or back to MailerQ, it should call MQ_continue(). In special circumstances you can call MQ_complete() or MQ_retry() instead.
The easiest way to get your hands on the software is to add the Copernica repository to the source list of your package manager. You can then use "apt-get" or "yum" to install MailerQ, and to get updates. If you do not want to use your package manager, you can download MailerQ manually. Check out our downloads page for a list of available files that can be manually installed.
We have two repositories, one for newer Ubuntu and Debian versions (Ubuntu 14.04 and up, and Debian version 8 and higher), and a repository for the older Debian 6 and Debian 7 versions.
| Ubuntu 14.04 and up | deb https://packages.copernica.com/debian stable main |
| Debian 8 and up | deb https://packages.copernica.com/debian stable main |
| Debian 6 and Debian 7 | deb https://packages.copernica.com/debian legacy main |
You can enter the following instructions to add our repository to your software list, and to download and install MailerQ.
$ wget https://packages.copernica.com/copernica.key
$ sudo apt-key add copernica.key
$ rm copernica.key
$ echo "deb https://packages.copernica.com/debian stable main" | sudo tee /etc/apt/sources.list.d/copernica.list
$ sudo apt-get update
$ sudo apt-get install mailerqWatch out: if you're on Debian 6 or Debian 7, make sure you replace the word "stable" with "legacy"!
For Red Had based systems there are two repositories as well, one for the latest versions of these systems, and one for older releases.
| CentOS 7+, Red Hat 7+, Fedora 22+ | https://packages.copernica.com/rpm/copernica.repo |
| CentOS 6, Red Hat 6, Fedora 21 | https://packages.copernica.com/rpm-legacy/copernica.repo |
To install MailerQ on a new version of CentOS, Red Hat or Fedora, enter the following instructions in your shell:
sudo wget https://packages.copernica.com/rpm/copernica.repo -O /etc/yum.repos.d/copernica.repo
sudo yum update
sudo yum install mailerqOf course, if you're on an older system (CentOS 7, Red Hat 6 or Fedora 21), you should modify the repository URL to contain "rpm-legacy" instead of "rpm".
Once you've added the MailerQ repository to the list of software sources, you can always get the latest stable version of MailerQ with the instructions "sudo apt-get install mailerq" for Debian/Ubuntu based systems or "sudo yum install mailerq" for Red Hat based systems. If you prefer an older version over the current stable one, or when you want to try the bleeding edge development version, you should append a version number.
sudo apt-get install mailerq-4.4
sudo apt-get install mailerq-devsudo yum install mailerq-4.4
sudo yum install mailerq-devThe development version has a "-dev" postfix. This dev version is automatically recompiled every 24 hours and contains the latest fixes and changes. Although it is often not recommended to use it in production environments, it's a great way to keep an eye on MailerQ improvements.
Note that if you run a specific version of MailerQ, the location of the config file changes too. By default, MailerQ loads it configuration from "/etc/mailerq/config.txt", but if you install an explicit version the version number is included in the filename of the config file (for example "/etc/mailerq/config-4.4.txt" or "/etc/mailerq/config-dev.txt").
All repositories mentioned above contain MailerQ versions that are statically linked, which means that there are hardly any dependencies. However, due to licensing conditions, we are required to supply dynamically linked versions of MailerQ too. If you prefer using a dynamically linked version, add '-shared' to the package name.
sudo apt-get install mailerq-5.0-shared
sudo apt-get install mailerq-dev-sharedsudo yum install mailerq-5.0-shared
sudo yum install mailerq-dev-sharedThis example assumes that MailerQ is running and configured. It uses the .NET/C# AMQP client library provided by RabbitMQ.
The .NET/C# example consists of two files; one that takes care of placing messages on the outbox queue and one that takes sent messages from the result queue.
SendThis class connects to RabbitMQ and places a JSON encoded message on the outbox
message queue. To connect to RabbitMQ and to create the JSON encoded message,
the program uses the private variables defined inside the class. If you want to
use the .NET/C# example to test your MailerQ configuration, you will have to run
the Send class before running the Result class.
Send.cs// declare the namespaces
using System;
using RabbitMQ.Client;
class Send {
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static string hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static string username = "guest";
// same as the rabbitmq-password option in the config file
private static string password = "guest";
// same as the rabbitmq-vhost option in the config file
private static string vhost = "/";
// same as the rabbitmq-outbox option in the config file
private static string outbox = "outbox_test";
/**
* Settings for sending out the test email
*/
// domain where the test message should be delivered
private static string recipientDomain = "example.org";
// email where the test message should be delivered
private static string recipientEmail = "testmail@example.org";
// address where the email was sent from
private static string fromAddress = "this_is_my_email@my_domain.org";
/**
* Main method
*/
public static void Main()
{
// create the connection factory
ConnectionFactory factory = new ConnectionFactory();
// set the hostname of RabbitMQ
factory.HostName = hostname;
// set the user name to connect to RabbitMQ
factory.UserName = username;
// set the password to connect to RabbitMQ
factory.Password = password;
// set the virtual host to RabbitMQ
factory.VirtualHost = vhost;
// create the connection
using (IConnection connection = factory.CreateConnection())
// create the channel
using (IModel channel = connection.CreateModel()) {
// declare the queue
channel.QueueDeclare(outbox, true, false, false, null);
// put the domain inside the JSONMessage
string message = "{ "\"envelope\": \"" + fromAddress + "\", "
+ "\"recipient\": \"" + recipientEmail + "\", "
+ "\"mime\": \"" + "From: " + fromAddress + "\r\nTo: "
+ recipientEmail
+ "\r\nSubject: Example subject\r\n\r\n"
+ "This is the example message text\" }";
// get the byte array value of the message
byte[] mime = System.Text.Encoding.UTF8.GetBytes(message);
// publish the message on the outbox queue
channel.BasicPublish("exchange1", "key1", null, mime);
// send some output to the console
Console.WriteLine("Sent", message);
}
}
}ResultThis class connects to the RabbitMQ server and gets the messages that were
placed on the outbox message queue by the Send class back from the result
message queue. The result messages from the result queue are shown to the user.
The Result class can only output any relevant information, if it is executed
after the Send class was run. The Result program will keep running, waiting for
new messages until it is terminated (use Ctrl-C). You could run the Send program
from an other terminal, to test the effect.
Result.cs// declare the namespaces
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
class Result {
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static string hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static string username = "username";
// same as the rabbitmq-password option in the config file
private static string password = "password";
.NET/C#
// same as the rabbitmq-vhost option in the config file
private static string vhost = "/";
// same as the rabbitmq-outbox option in the config file
private static string resultbox = "results_test";
public static void Main() {
// create the connection factory
ConnectionFactory factory = new ConnectionFactory();
// set the hostname of RabbitMQ
factory.HostName = hostname;
// set the user name to connect to RabbitMQ
factory.UserName = username;
// set the password to connect to RabbitMQ
factory.Password = password;
// set the virtual host to RabbitMQ
factory.VirtualHost = vhost;
// create the connection
using (IConnection connection = factory.CreateConnection())
// create the channel
using (IModel channel = connection.CreateModel()) {
// declare the queue
channel.QueueDeclare(resultbox, true, false, false, null);
// start the consumer
QueueingBasicConsumer consumer = new QueueingBasicConsumer(channel);
// start the consumer
channel.BasicConsume(resultbox, true, consumer);
// message telling the user that the script is waiting for messages
// that will be placed on the result queue by MailerQ
System.Console.WriteLine("Waiting for messages." + "To exit press CTRL+C");
// keep waiting for new messages
while(true) {
// get a delivery from the result queue
BasicDeliverEventArgs ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
// get the message from the result message queue as a
// byte array
byte[] data = ea.Body;
// convert the message into a string
string message = System.Text.Encoding.UTF8.GetString(data);
// write some output to the console
System.Console.WriteLine("Received:");
System.Console.WriteLine(message);
System.Console.WriteLine("------------------------------------------");
}
}
}
}Provided .NET/C# is working and you installed the .NET/C# AMPQ library in the
/lib/ directory, you can test your configuration as follows
$ gmcs -r:/lib/bin/RabbitMQ.Client.dll Send.cs
$ gmcs -r:/lib/bin/RabbitMQ.Client.dll Result.cs
Run the Send.exe file to put a message on the queue.
$ MONO_PATH=/lib/bin mono Send.exe
Run the Result.exe file to get the result messages from the result message queue.
$ MONO_PATH=/lib/bin mono Result.exe
MailerQ relies on the DNS resolver functions that are standard installed on your system. The bad thing about these standard functions is that they are not very good, and they were never intended for high performance MTAs that do many DNS lookups at the same time. There is however also a nice thing about using the standard resolver functions: the DNS lookups of MailerQ return exactly the same results as the lookups done by system utilities like telnet, ping, openssl, dig, et cetera.
To work around the bad aspects of the standard resolver functions, MailerQ runs all DNS operations in seperate threads. By doing this, a slow DNS operation can never block or slow down other mail deliveries. The number of threads that MailerQ uses for the DNS lookups can be set in the config file using the "dns-threads" variable:
dns-threads: 5The management console has a special DNS widget that you can use to monitor how many DNS operations are in progress. If you see that the number of DNS operations is almost always higher than the number of threads, it is best to increase the number of threads. This will immediately improve your MTA's performance.
For the DNS lookups of IP addresses, MailerQ uses the standard "getaddrinfo" library call that is available on all Linux systems. This function checks the /etc/hosts and /etc/gai.conf files for your server's DNS settings and it sends a DNS query to a DNS server if necessary. This is exactly what you want.
However, this standard "getaddrinfo" function does not expose the time-to-live (TTL) value to its caller, which is needed to know for how long DNS results can be cached. MailerQ is therefore conservative and assumes that all TTLs are set to 60 seconds. This low TTL value causes a lot of unnecessary DNS lookups, because DNS queries are repeated every single minute, while in reality most TTL are set to a much higher values (for example 24 or 48 hours).
To prevent all these extra DNS lookups, you can instruct MailerQ to forget about the "getaddrinfo" function and send queries to DNS server directly. If MailerQ does this, it will have access to the real TTL value, and can cache the DNS results for a much longer time. You can disable the "getaddrinfo" calls with the following config file setting:
dns-getaddrinfo: noThere is however a downside here. If the getaddrinfo function is not used, MailerQ will also not check the /etc/hosts and /etc/gai.conf file, and you run the risk that MailerQ's DNS lookups return different results than the lookups of system utilities.
There is a second downside to the real TTLs, because this may also be very low. Some domains specify TTLs in the range of 0-30 seconds. To mitigate this, the minimum respected TTL is 60 seconds by default, since some domains specify 0 TTL which can cause lookups for every message. This incurs a significant performance penalty. change this minimum value, the following config file option can be used:
dns-min-ttl: 120This will make sure that the mininum respected TTL is 120 seconds. Note that the minimum allowed value should be greater than zero, since zero is explicitly disallowed due to the performance penalty.
Every couple of minutes, MailerQ reloads all its information from the database, and validates all the DKIM keys in the database by matching the public key found in DNS with the private key in the database. If you have a lot of DKIM keys in your database, this could take a very long time because the DNS lookups are slow and MailerQ can only run one at a time. If you want to speed this up, you can start special DNS threads that are only used when DKIM keys are being reloaded. This allows MailerQ to run many lookups in parallel. The number of threads can be set in the config file:
database-threads: 10These threads only run during the short period of time that DKIM keys are loaded from the database.
MailerQ uses a relational database to store all sorts of configuration data and delivery settings. The data stored in the database includes the delivery throttles and DKIM keys. The database is read and updated by MailerQ, but you are free to write your own scripts and programs that also modify this data.
MailerQ supports multiple database platforms: MySQL(5.5+), MariaDB(5.5+), PostgreSQL(9.1+) and SQLite3. The SQLite database is by far the simplest to set up because it does not require a server process to run. All you need is the "sqlite3" library to be installed on your system. Although it is the simplest to set up, we advise you to not use an SQLite database in production, because it makes scaling up MailerQ a lot more difficult.
The other database systems, MySQL, MariaDB and PostgreSQL, take a little more effort, but they are not too difficult to configure either. This is the most future-proof, since it can be easily scaled or shared between instances. If you already have such a database server running, you only have to create a new database and put the address, login and password to that database in MailerQ's configuration file. MailerQ will do the rest: the tables will be created and filled with the settings that you enter via the management console.
Only one variable has to be set in the config file to connect to a database: the "database-address" variable:
database-address: sqlite:///path/to/database/file.sql
database-address: mysql://user:password@hostname/databasename
database-address: postgresql://user:password@hostname/databasenameSQLite is the simplest database to set up, just specify the path to a file on the MailerQ server (this file does not even have to exist). However, an SQLite database can not be used if you want to access the same database from multiple machines.
If you do use SQLite, note that you need three (!) slashes in the address: the "sqlite://" prefix, followed by the "/path/to/database". If you use a MySQL, MariaDB or PostgreSQL database, make sure that the database exists, and that MailerQ not only gets enough privileges to read and write from and to the database, but also to create and modify tables.
On startup, MailerQ automatically checks if the tables in your database exist, and whether the database structure is correct. Missing, broken or incomplete tables are reported.
From a performance perspective it does not really matter which database engine you choose to use. MailerQ periodically copies all settings from the database to main memory, and uses this in-memory cache for lookups. No realtime queries are executed, and no connections are kept open to the database in between these reloads. The speed of the database does therefore not have to be a factor in choosing the most appropriate engine. It is better to choose a database that you feel most comfortable with. Do you already use MySQL databases? Then it is best to use it for MailerQ too. Do you run a single MailerQ instance, and does data not have to be shared amongst multiple MailerQ instances? Then the SQLite database is sufficient.
MailerQ makes use of client libraries on your system to connect to the database. To connect to a MySQL or MariaDB database, you must make sure that either libmysqlclient or libmariadbclient is installed on your system. For PostgreSQL connections libpq has to be installed, and libsqlite3 is needed for SQLite3 databases. If these libraries are not available on the system, it is not possible to connect to the database.
As we mentioned, the database is not used for real time queries. Instead, all data is periodically copied into main memory. This ensures that the speed of the database is never a limiting factor in your mail deliveries, or that a slow query slows down your email deliveries.
MailerQ normally reloads the database every 10 minutes. However, this is a configurable interval. In the config file you can set the "database-ttl" variable to any possible interval you like. This variable should be set to the number of seconds that the data loaded from the database is kept in main memory. The default value is 600, which is 10 minutes.
database-ttl: 60If the database-ttl is set to 60 seconds, like we did above, the data is reloaded every minute.
The private DKIM keys that are stored in the database can be configured to be verified against the public keys found in DNS. If this is enabled, MailerQ will do a DNS query for each DKIM key to see if the private key indeed matches the published public key. Keys that do not match will not be used.
Running all these DNS queries may sometimes take a long time, especially if you have a lot of DKIM keys in your database. To speed things up, you can increase the number of threads that are used for these DNS queries. The "database-threads" config file option sets the number of threads to be used for DNS lookups that are triggered by a database reload:
database-threads: 10The default value for this variable is one. If you notice that it takes a lot of time to start MailerQ because of all these DKIM key checks, you can experiment with higher values.
When MailerQ starts, it normally first connects to the database and checks whether
all tables are in a valid state. If this test fails, MailerQ does not start and
reports an error instead. With the --repair-database command line switch,
you can instruct MailerQ to repair all broken tables. Tables and columns that
are missing will be created, and obsolete columns are removed,
$ mailerq --repair-databaseThis option tells MailerQ to create and alter missing or broken tables. The only time when you normally do this, is when you install MailerQ for the first time, or when you are upgrading from a previous version.
If you want to enforce that all tables in the database are dropped and replaced by brand new empty tables, you can start MailerQ with the "--purge-database" command line option:
$ mailerq --purge-databaseThis option tells MailerQ not to check and repair tables, but to drop them all and create new ones.
You may want MailerQ to skip the database consistency check on startup. For example if you have manually modified the tables and added a couple of extra columns. To skip the validation on startup, you can set the config file variables 'database-validate' to false:
database-validate: falseIt is possible to run multiple MailerQ instances that all connect to the same database. If you do this, we recommend setting up a cluster of MailerQ instances. The different MailerQ instances then notify each other every time the settings in the database are updated, so that each instance can update its cache.
MailerQ has a powerful web based MTA management console. This console gives you access to the database using simple web forms. During normal operations, it is therefore not necessary to run any queries on the database yourself. But if you do like to access the data, you are free to do so.
All tables in the database have a very straightforward structure. Because MailerQ is compatible with many different database systems, we do not use vendor specific SQL features and have kept all tables as simple as possible. The tables use well known data types, and no foreign keys or constraints. Booleans are stored as integers and "NULL" and "0" values have the same semantics. The value "0" is often used to set something to "unlimited", while "-1" values are used to fall back on the default setting.
All data is stored in the tables in UTF-8 encoding. If you use international domain names and/or international email addresses, you should store them in UTF-8 format in the database.
The following tables are created:
| Tables | Description |
|---|---|
| capacity | all delivery capacities, eg max number of connections |
| capacity_rules | capacity rules per domain / MTA IP |
| dkim_keys | DKIM keys to sign outgoing mails |
| dkim_patterns | rules that decide what DKIM keys to use |
| dns | overridden dns lookups |
| errors | Forced errors |
| flood_responses | alternative delivery limits to use when certain responses come in |
| ips | all MailerQ's IP addresses |
| locals | local email addresses |
| mtanames | alternative "EHLO" names |
| paused | deliveries on pause |
| response_actions | response action, eg send mail from a different IP |
| response_patterns | patterns of responses that has to match |
| response_patterns_actions | what kind of response actions must be executed |
| rewrite_actions | actions to perform when a message is loaded eg send from a different IP |
| rewrite_triggers | triggers to fire rewrite actions |
Most tables are pretty self-explanatory, and every table can be filled by using the management console. We'll give a short explanation of each table here.
All delivery capacities are stored in this "capacity" table. It for example holds the max number of connections that may be set up to a certain domain, or the max number of messages to send within a minute.
The "localip" and "domain" columns in this table can both be empty. If they are empty, the setting applies to all local ips and/or all domains. Only the most specific record is used. Thus, if a mail is sent from IP 1.2.3.4 to domain example.com, the record with the same localip and same domain is used. If no such record exists, MailerQ looks for a record that only matches the domain or the IP (and for which the other field is empty). If that record also does not exist, the record for which both the domain and the IP are empty is used.
The best way to get an understanding of the column in this table is to play around with the management console, and see what kind of data ends up in this table.
The "flood_responses" table has a similar layout to the capacity table, but with a "pattern" and a "type" column instead of an IP address and a domain. For each failed delivery, the answer message from the remote server is checked agains the patterns in this table. If there is a match, MailerQ will start using the delivery capacity set in this table.
For example, if you insert a record in this table with pattern "flood detected" and type "substr", and MailerQ receives an answer from a remote server with a message like "message rejected, flood detected from your IP address", MailerQ automatically switches the send capacity to the associated capacity.
Once again, the best way to get an idea of the meaning of the columns is to just experiment with the management console, and then look at the data in this table.
On startup, MailerQ detects all IP addresses that are linked to the server, and stores these IP addresses in a database table. MailerQ does not use this table for queries or for anything, it is just created so that other programs or scripts can find out which IP addresses are available for sending out emails.
All DKIM private keys that are used for signing outgoing mails are stored in the "dkim_keys" table. Only the domain, the selector and the expire time for the signature are stored; other properties are retrieved from DNS.
To decide what keys should be used for outgoing emails, there is a separate "dkim_patterns" table. The from address of every outgoing mail is compared to the records in this table, and whenever there is a match, the associated key is used for a DKIM signature.
The pattern type can be "regex", "fnmatch", "substr" or "exact", and decides whether the "sign" column holds a regular expression, or some other kind of pattern.
Incoming mails are compared with the addresses stored in the local email address table. When there is a match, the mail is not sent to the regular inbox RabbitMQ message queue, but to the locals queue instead. The "type" column can be set to "regex", "fnmatch", "substr" and "exact".
If you want to send mail from a different IP address than is stored in the message JSON, you can use the "delegates" table. For example, if you notice that "example.com" no longer accepts incoming connections from your "1.2.3.4" address, but that it does accept connections from "1.2.3.5", you can add a row to the "delegates" table. This row instructs MailerQ that it should send all mail that was originally planned to be sent from "1.2.3.4" to "example.com" to be sent from address "1.2.3.5" instead.
To pause deliveries from and/or to IP addresses or domains, you can insert records in the "paused" table. The "target" column in this table can hold IP addresses, domain names and empty values. The "localip" column is of course only used for IP addresses, but may also hold empty strings.
If you want to pause all deliveries from your own IP address "1.2.3.4" to "example.com", you can simply insert such a record in the "paused" table. An empty value in one of the columns of the "paused" table means that all deliveries are paused. For example, if you insert an empty domain and IP address "1.2.3.4", MailerQ pauses all deliveries from "1.2.3.4", no matter to what domain they should be sent. If you insert a record with an empty domain and IP "5.6.7.8" as target, all deliveries to IP address "5.6.7.8" will be put on hold.
By default, MailerQ does a reverse DNS lookup to find the hostname of its local IP addresses. This hostname is the name used in the "HELO/EHLO" SMTP handshake. If you want to use a different hostname instead, you can insert a row into this table.
To override the DNS resolver, you can add records to the "dns" table. MailerQ normally uses normal DNS queries to find out to which IP addresses mail should be sent. First it does an MX query to find the MX records, and then for each MX record MailerQ looks for the associated A or AAAA records to find the IP addresses.
However, in case you want to bypass these lookups for specific IP addresses, you can add records to the "dns" table. If matching records are available, MailerQ uses the IP addresses in these records instead of doing the DNS queries. This feature can for example be useful if you know that certain domains (like the ones that you own yourself) have special IP addresses on which they receive email, besides the ones that are announced in DNS.
MailerQ reads its configuration from the "/etc/mailerq/config.txt" config file. This file holds configuration options for the connection to RabbitMQ, NoSQL storage engine and database (MySql, Sqlite or PostgreSql) and other options for MailerQ itself.
The config file, and the documentation, has been split up in different sections. Please select the section that you're interested in or find the desired setting in the A-Z settings overview.
The config file is not the only place where MailerQ gets its configuration from. Most of the settings relevant for deliveries are not kept in the config file, but are loaded from a relational database. The web based management console allows you to read or change these settings, and you can of course also access the database directly to read or update these settings.
The MailerQ program can be started in two different modes: as a daemon process that runs all the time and that sends out the messages, or as a command line utility that reads a message from standard input and publishes it to the inbox queue. As a command line utility, MailerQ does not attempt to send out the mail, it just puts the message in the RabbitMQ message queue. You need a separate running MailerQ daemon process that takes care of the delivery.
When you normally start up MailerQ, it operates as a daemon process. To start
it as a command line utility for reading a message from standard input, you need
to pass extra arguments to the program; either the email addresses of the recipients,
or the option --extract-recipients to tell MailerQ that it should read in a
mime message from standard input and filter out the destination addresses from it.
$ mailerq recipient1@example.com recipient2@example.com
$ mailerq --extract-recipients
The message read in by MailerQ is converted into a JSON encoded email message and published to the inbox queue. Another MailerQ daemon process can then pick it up from there and take care of the delivery.
MailerQ reads in all data from standard input until it encounters a line with a single dot on it. It treats this line as the end-of-message marker and publishes all data up to this dot (but not including the dot) to the inbox message queue. This is the same behavior as "sendmail" and allows you to use MailerQ as drop-in replacement for sendmail.
This behavior could however be problematic if your email contains lines that start with a dot. Such lines should of course not be treated as end-of-message markers. There are two ways to prevent this; you can either stuff those lines with an extra dot in front. MailerQ will automatically recognize this and removes the extra dot, or you can add the command line option "--ignore-dot" to instruct MailerQ that dots do not have a special meaning, and that the message is ended with end-of-file instead.
$ mailerq --ignore-dot recipient1@example.com recipient2@example.com$ mailerq --ignore-dot --extract-recipientsThe envelope address that MailerQ uses for sending out an email is automatically extracted from the message MIME. It can be set in the custom "x-mq-envelope" header, or in the "return-path", "sender" or "from" header (in that specific order). However, you can also set the envelope address manually with the "--envelope" command line option:
$ mailerq --envelope myaddress@mydomain.com --extract-recipients
You can use regular unix shell pipes and I/O redirection operators to get the contents from a file into MailerQ.
$ cat email.txt | mailerq --extract-recipients --ignore-dot$ mailerq --extract-recipients --ignore-dot < email.txt If you run multiple MailerQ instances, you can set up a MailerQ cluster. All instances will be aware of the other running MailerQ servers, share information and hand over undeliverable messages. Setting up a cluster has many advantages:
Every MailerQ server in the cluster shares its configuration with the other instances. This allows the management consoles to link to each other, and more importantly, to trigger other instances to reload cached data when settings change. For example, when you add a DKIM key via the management console of your first MailerQ server, the other MailerQ servers will also have direct access to this new DKIM key.
Another advantage of setting up a cluster is that email messages automatically end up in the right outbox queue. In a normal cluster setup, you run multiple servers and on each server you run a single MailerQ instance. Each instance reads from its own outbox queue (instead of having all instances consume from the same queue, which is not recommended). If you want to send out an email from a specific IP address, you therefore must make sure that the message ends up in the right outbox queue. However, if you do publish a message to a wrong queue, the MailerQ inter-cluster communication ensures that the message will be moved to the right outbox queue.
There are two config file settings relevant for setting up a cluster:
cluster-address: amqp://login:passwd@host/vhost
cluster-exchange: clusterBoth cluster settings are optional. If you leave them empty, the exchange "cluster" is used, and the same server address as is set in the "rabbitmq-address" setting.
It is important that every running MailerQ instance uses the same cluster settings. Even when your instances all have their own private RabbitMQ server, the instances must still share the same values for the "cluster-address" and "cluster-exchange" settings.
It is also recommended to use the same relational database for each instance. By doing this, the instances use the same delivery throttles and DKIM keys.
If your RabbitMQ server supports secure connections, you can configure MailerQ to connect to an "amqps://" address instead. The communication between MailerQ and RabbitMQ will then be encrypted.
cluster-address: amqps://guest:guest@hostname/vhost
cluster-verify: falseMailerQ checks the server certificate when it connects to a secured RabbitMQ server. If this certificate can not be verified with one of the known openssl certificate authorities, MailerQ refuses to set up a connection. If you use a self-signed certificate, you may want to skip this extra test, and set the config file option "cluster-verify" to "false".
All instances create a private message queue in RabbitMQ and bind this private queue to the cluster exchange. By doing this, each MailerQ instance receives all messages that are published by the other instances. Each instance also periodically sends its local IP addresses and some other configuration settings to this exchange. The announcement messages end up in all the private queues of the instances, and everyone in the cluster receives information about everyone.
When a change is made via one of the management consoles, a message is sent to the cluster exchange, so that each server in the cluster can reload the delivery limits from the database.
Be aware that both RabbitMQ and MailerQ support clustering, but that from a technical standpoint this clustering has a whole different meaning. Clustering for RabbitMQ means that queues are mirrored across different RabbitMQ instances, so that no messages are lost when one of the RabbitMQ servers crashes.
For MailerQ clustering means something different: the individual servers communicate and share data with each other, but it does not necessarily mean that emails will still be delivered when one of the servers crash. If no other server has access to the same IP addresses as the crashed server, there is no way that any other server can take over the deliveries. If you do want to achieve this, you have to set up a heartbeat daemon so that other servers can automatically jump in and take over when one of the servers fails.
To find out if the cluster is working, you can simply go to the management console of one of your MailerQ instances. If the console shows hyperlinks to your other MailerQ instances, you know that the instances have identified each other over the shared RabbitMQ exchange. You can also start MailerQ with the "--list-cluster" command line argument to find out if the cluster is working:
$ mailerq --list-clusterIf you start MailerQ like this, it will connect to the cluster and wait for one second. All the MailerQ instances that announce themselves within that second are displayed. Normally, all instances in the cluster report within this timeout. However, if your servers or internal network is overloaded, the 1 second period could be too limited. In that case you can change the "cluster-timeout" variable to use a different timeout. This setting can be set in the config file or via the command line:
$ mailerq --list-cluster --cluster-timeout=5You can also write your own scripts or programs to access the cluster. You can for example write an application that connects to RabbitMQ, and that publishes a single JSON message to the cluster exchange to instruct all running MailerQ instances that they should reload the database:
{ "reload": true; }If the above instruction is received by a MailerQ instance, it will immediately connect to the database and reload all settings from it.
It's also possible to intercept all cluster traffic. For this you simply have to create a queue in RabbitMQ and bind it to the shared cluster exchange. All messages that are sent to the cluster will then also end up in your queue. By reading out the messages from the queue, you will exactly see what is going on.
Tip: normally all MailerQ instances report once every couple of minutes their status on the cluster. If you do not want to wait that long, you can send a single "hello" json message:
{ "hello": true; }Some mail servers initially seem to accept a message, but send back a bounce email later on in which they inform you that the mail could not be delivered after all. Such asynchronous bounces are sent back to the envelope address of the initial message. There is sadly often no way to find out whether a server is going to send back notifications, and what these messages are going to look like.
However, there is a formal standard for these bounce messages: the Delivery Status Notification (DSN) format. And there is an extension to the SMTP protocol that allows mail servers to exchange with each other whether they respect this standardized format, if the original sender would like to receive such bounce messages, and what kind of information he or she wants to receive. This standard make it much easier to link incoming bounces to the original sent email.
MailerQ supports this standard and SMTP extension: when communicating with a receiving server, DSN parameters can be passed to tell the receiver what kind of bounces should be returned. If MailerQ is configured to receive mail on a SMTP port, it understands these kind of parameters too. MailerQ can also be configured to send out, receive and recognize DSN messages.
Delivery Status Notifications are sent back to the envelope address of your messages. This is the address that is used in the "MAIL FROM" command of the SMTP handshake, and that is stored in the "envelope" property of the input JSON. If you want to receive DSN messages, you therefore first have to make sure that your envelope address is valid, and that it indeed points back to your server.
Besides the envelope address that is used to send back DSN messages, some mail servers (but not all!) have implemented the SMTP DSN extension, and allow extra parameters to be passed to instruct them what kind of DSN messages you would like to receive back. You can for example specify that you want to see the full original mail message in the notification, or just the headers. You can also specify whether you want to receive notifications on failure, on success and/or on a delayed delivery.
{
"envelope": "youraddress@yourdomain.com",
"recipient": "info@example.com",
"mime": "....",
"dsn": {
"notify": "FAILURE",
"ret": "HDRS",
"envid": "unique-identifier",
"orcpt": "info@example.com"
}
}The above JSON sets "youraddress@yourdomain.com" to be the envelope address. If something goes wrong, the notification will be sent to this address. If the receiving mail server also supports the DSN SMTP extension, MailerQ uses the "dsn" property in the input JSON to pass additional DSN parameters to the receiver. You can use the following four DSN settings:
| notify | comma separated events that trigger a notification (FAILURE, DELAY, SUCCESS, NEVER) |
| ret | should the notification hold the full original mail or just the headers (FULL, HDRS) |
| orcpt | the address to be included in the notification as "original-recipient" |
| envid | unique identifier to be included in the notification as "original-envelope-id" |
The "notify" variable specifies what sort of events trigger the Delivery Status Notification. Possible values are "NEVER", "SUCCESS", "FAILURE" and "DELAY". If you set it to "FAILURE", you will only receive bounces on failure. It is also allowed to set this to a comma separated list of values. If you set the "notify" property to "SUCCESS, FAILURE", a bounce message will be sent on successful delivery as well as on failure.
The bounce message holds a full delivery report, as well as the full original message. If you want to save a lot of network bandwidth, you can set the "ret" property to the value "HDRS". By doing so, you instruct MTAs not to include the full MIME message in the status notification, but only the original MIME headers.
The property "orcpt" is an optional property and holds the original recipient address. This original recipient address is included in the delivery report. This property is optional; if you leave it out the actual recipient will be used. This property is only useful if you want the DSN to contain a different "original-recipient" value than the actual recipient.
The last property that you can include in the "dsn" object, is "envid". This is a message specific message identifier, and you can set it to whatever you like. It will be included in the bounce message as "original-envelope-id", and you can use it to match the bounces with a sent message.
Keep in mind that not every mail server supports the DSN extension, and that even if they do support it, they may not always respect your parameters. There is no guarantee that you receive the DSN that you wanted. For example, if you specify that you want to receive the full original mail in the bounce (you set "ret" to "FULL"), it still is possible that you do not receive a bounce at all, or a bounce with just the headers.
In the above example, we've demonstrated the JSON properties for passing DSN parameters. If you inject mails using a non-json format (via SMTP, the spool directory or via MailerQ's command line interface), you can use MIME headers instead. The following MIME headers are recognized, and do the same thing as the JSON settings:
x-mq-dsn-notify: FAILURE
x-mq-dsn-ret: HDRS
x-mq-dsn-envid: your_unique_identifier
x-mq-dsn-orcpt: email@example.comIf one or more of the DSN settings are missing, the default values from the global config file are used. These defaults can be set with the following config file variables:
dsn-notify: FAILURE
dsn-ret: HDRSYou can only set defaults for the "notify" and "ret" settings, because the "envid" and "orcpt" are normally only used on a per-message setting and it is not meaningful to have default values that apply to all emails.
MailerQ has its own ways to report the results: it publishes JSON result objects to the result queues, and it writes results to log files. These reporting methods are powerful, and easy to integrate. Most of the MailerQ users therefore do no let MailerQ send out its own DSN messages back to the envelope address. The only DSN messages that are normally sent back to your envelope address are notifications from MTAs that initially accepted the mail.
However, MailerQ can be configured to send out DSN messages too. If you set up a RabbitMQ DSN queue (using the "rabbitmq-dsn" config file property), MailerQ starts sending out DSN messages. Every time that a message cannot be delivered, an outgoing bounce mail is constructed and published to this DSN queue. If you assign the outbox to this "rabbitmq-dsn" setting, the bounces are even automatically picked up and delivered.
We wrote that a bounce is sent on every failed delivery. This is technically not completely correct. To be precise, a bounce is created everytime that the message specific "dsn.notify" setting matches. Thus, if "notify" is set to "SUCCESS", MailerQ sends out a bounce message on successful delivery, and when it is set to "FAILURE", bounces are sent on failure. The "notify" setting is normally set to "FAILURE" so in normal circumstances bounces are sent on failure.
If you set the "rabbitmq-dsn" setting to an empty string (which is the default), no Delivery Status Notifications are sent by MailerQ. But MailerQ does forward the DSN settings to the receiving MTA, so you could still receive DSNs from MTAs further up the chain.
If you inject mail into MailerQ using SMTP, you can pass DSN parameters with the "MAIL FROM" and "RCPT TO" commands.
MAIL FROM:<envelope@example.com> RET=HDRS ENVID=unique_identifier
RCPT TO:<info@example.com> NOTIFY=FAILURE ORCPT=rfc822;info@example.comThese parameters are directly copied to the "dsn" property in the JSON that is published to the inbox queue. To enable or disable this feature, use the "dsn-advertise" setting in the config file.
dsn-advertise: 1If you set dsn-advertise to 1, MailerQ announces in its SMTP connection handshake that it supports the DSN extension, and converts the DSN parameters that are passed to the "MAIL FROM" and "RCPT TO" commands to JSON properties. If you set it to 0, no DSN parameters can be passed to "MAIL FROM" and "RCPT TO".
MailerQ can also be used to recognize incoming DSN messages. If you submit a DSN message to the SMTP port of MailerQ, and MailerQ detects that both the recipient address is on the list of local email addresses and that the message holds a report message, it publishes this report to the queue for incoming reports. The name for this queue can be set in the config file with the "rabbitmq-reports" variable.
This means that you can not only use MailerQ to send out mass mailings, but that you can also run a MailerQ instance to process the delivery reports that come back. If you use "reports@yourdomain.com" as envelope address, you can run a MailerQ instance to process all mails for the yourdomain.com domain. Via the management console you add the "reports@yourdomain.com" local email address, so that all these incoming messages are accepted and sent to either the "local" message queue for normal email messages, and the "reports" message queue for emails that were recognized as reports.
MailerQ does not only recognize DSN messages, but also other types of reports like DMARC reports and disposition notifications.