Cloudmark Plugin for CommuniGate Pro |
| |
|
|
The Cloudmark spam filtering Plugin runs as an External Filter and calculates a spam "score" for each message being processed. Unlike tools with statically defined patterns for spam messages, the Cloudmark Plugin dynamically retrieves new patterns from Cloudmark Network thus providing greater accuracy for new spam messages.
The score ranges from 0 to 100; the higher the message score the more likely the message is spam. The score info is added to message headers so it can be processed by Server-Wide, Domain-Wide and Account Rules.
By default the added header lines look like this:
X-Junk-Score: 92 [XXXX] X-Cloudmark-Score: 92 [XXXX] X-Alert: possible spam! X-Color: redBesides the digital score value, the header field contains a "bar score" to simplify automated message processing: the more 'X' characters the higher the score. The following ratios between the digital and bar scores are used by default:
| Digital score range | Bar score |
| 0 | [] |
| 1-39 | [X] |
| 40-80 | [XX] |
| 81-90 | [XXX] |
| 91-95 | [XXXX] |
| 96-99 | [XXXXX] |
| 100 | [XXXXXX] |
Every day at midnight the Plugin generates a report message about the number of mails processed and their spam scores. By default the report message is mailed to postmaster address from the CommuniGate main domain.
Note: The Cloudmark Plugin is available only for some platforms supported with the CommuniGate Pro server software. Before you order the Cloudmark Plugin License, make sure the plugin is available for your CommuniGate Pro Server platform.
Note: The Cloudmark Plugin requires CommuniGatePro version 5.2.3 or later.
Hardware requirements:Pentium-II or above (UltraSparc for Solaris), Multi-processor systems and multi-core CPUs are encouraged (but not Intel HyperThreading). 512MB of RAM, 200MB of hard disk space.
Note:The Cloudmark plugin requires outbound TCP port 80 network access to Cloudmark's external 208.83.136.0/22 CIDR range to access lvc.cloudmark.com and microupdates.cloudmark.com servers. It downloads about 100MB of updates daily.
Cloudmark plugins are available for certain platforms only.
| Operating System | CPU | Download | |
|---|---|---|---|
| via http |
via ftp |
||
| Linux (RedHat, Debian, SuSE) | x86 |  |
|
| x86_64 | |
|
|
| Sun Solaris | Sparc | |
|
| x86_64 | |
|
|
| FreeBSD 6.2+ | x86 | |
|
| FreeBSD 7.0+ | x86 | |
|
| x86_64 | |
|
|
| Microsoft Windows NT/2000/XP/2003 Microsoft Windows 95/98/ME |
x86 | |
|
The current version of the Plugin is 2.0
| Library Name | Location |
|---|---|
| libpthread.so.0 | /lib |
| libstdc++.so.5 | /usr/lib |
| libm.so.6 | /lib |
| libgcc_s.so.1 | /lib |
| libc.so.6 | /lib |
| libdl.so.2 | /lib |
| librt.so.1 | /lib |
| ld-linux.so.2 | /lib |
| Library Name | Location) |
|---|---|
| libpthread.so.0 | /lib64 |
| libstdc++.so.6 | /usr/lib64 |
| libm.so.6 | /lib64 |
| libgcc_s.so.1 | /lib64 |
| libc.so.6 | /lib64 |
| libdl.so.2 | /lib64 |
| librt.so.1 | /lib64 |
| ld-linux-x86-64.so.2 | /lib64 |
| Library Name | Location |
|---|---|
| libpthread.so.2 | /lib or /usr/lib |
| libc.so.6 | /lib or /usr/lib |
| libm.so.4 | /lib or /usr/lib |
| libstdc++.so.5 | /usr/lib |
| Library Name | Location |
|---|---|
| libthr.so.3 | /lib |
| libc.so.7 | /lib |
| libm.so.5 | /lib |
| libgcc_s.so.1 | /lib |
| libstdc++.so.6 | /usr/lib |
| Library Name | Location |
|---|---|
| libnsl.so.1 | /usr/lib |
| libc.so.1 | /usr/lib |
| libsocket.so.1 | /usr/lib |
| libresolv.so.2 | /usr/lib |
| libdl.so.1 | /usr/lib |
| librt.so.1 | /usr/lib |
| libpthread.so.1 | /usr/lib |
| libCstd.so.1 | /usr/lib |
| libmp.so.2 | /usr/lib |
| libaio.so.1 | /usr/lib |
| libCrun.so.1 | /usr/lib |
| libthread.so.1 | /usr/lib |
| libc_psr.so.1 | /usr/platform/$(uname -i)/lib |
| libCstd_isa.so.1 | /usr/lib/cpu/sparcv8plus |
| Library Name | Location |
|---|---|
| libnsl.so.1 | /lib/64 |
| libc.so.1 | /lib/64 |
| libsocket.so.1 | /lib/64 |
| libresolv.so.2 | /lib/64 |
| libdl.so.1 | /lib/64 |
| librt.so.1 | /lib/64 |
| libpthread.so.1 | /lib/64 |
| libCstd.so.1 | /usr/lib/64 |
| libmp.so.2 | /lib/64 |
| libmd5.so.1 | /lib/64 |
| libscf.so.1 | /lib/64 |
| libaio.so.1 | /lib/64 |
| libCrun.so.1 | /usr/lib/amd64 |
| libm.so.2 | /usr/lib/amd64 |
| libdoor.so.1 | /lib/64 |
| libuutil.so.1 | /lib/64 |
| Library Name | Location |
|---|---|
| MSVCR71.DLL | {SystemDir}\system32 |
| MSVCP71.DLL | {SystemDir}\system32 |
When updating the Cloudmark Plugin to a newer version, do the following steps:
On a Unix System:
On a MS Windows System:
Note: the distributive contains no spam definition databases and on the first launch the Engine will download them and store some files into micro_updates subdirectory; and while the databases are not loaded completely the accuracy may be degraded. To change this behavior you can switch the "download micro-updates before init" in cartridge.cfg file to "yes", however it will make the engine initialization to take quite long time.
Note: the test spam message from test.msg file may become obsolete and can be removed from spam definitions databases; so this is not a malfunction it it's scored 0. Try replacing it with a fresh spam.
Open the General page in the Settings section of the WebAdmin Interface and click the Helpers link. Create a Helper for the Cloudmark Plugin:
| ||||||||||||
Note: For a MS Windows system the Program Path should be CGPCloudmark\CGPCloudmark.exe with the back slash "\" as the path separator.
Note: On some versions of FreeBSD system you may need to specify the full path to the program, i.e. /var/CommuniGate/CGPCloudmark/CGPCloudmark
To invoke the SpamCathcer Helper you should create a Server-Wide
Rule
with "ExternalFilter Cloudmark" action. The Scanning Rule will apply Cloudmark to the
message and the spam score will be added to the message header.
Note: It must be a Server-Wide Rule, not Domain-Wide or Account-level.
The recommended Scanning Rule is as follows:
| Data | Operation | Parameter |
|---|---|---|
| Action | Parameter | |
This rule skips messages from the MAILER-DAEMON address (such as non-delivery reports, return-receipts, etc.), skips messages from Client IP Addresses and from authenticated senders, and includes only messages for local accounts and mailing lists.
Note: The unlicensed installation of Cloudmark Plugin is limited to 5 messages per hour. If the E-mail traffic exceeds the limit, the Plugin will let the messages go through unrated.
Cloudmark by itself doesn't block spam, it only assigns a spam score to the messages. To actually block spam you need to create yet another Rule which blocks messages according to their spam score. There are many scenarios possible:
Scenario #1: suitable for small companies where you can assign one person (e.g. postmaster) to look through the spam messages daily to check for false positives, and if any false positives found - redirect them to the appropriate persons.
Create a Server-Wide Rule with the following contents:
| Data | Operation | Parameter |
|---|---|---|
| Action | Parameter | |
This Rule moves the incoming messages with score 96 and greater to the "spam_box" mailbox
of the postmaster@domain.com account.
The "Discard" action is required to prevent the message from going to the
initially intended destination (INBOX mailbox).
Note in the example above, the "*" in [XXXXX* is necessary to filter all messages
scored above 5 X's. Without it, the rule will only filter out messages with 5 X's.
Note: The priority of this Server-Wide Rule must be lower than the priority of the Scanning Rule.
Scenario #2: suitable for large companies and ISPs. Let users to deal with spam on their own.
Create one Domain-Wide rule or many Account-level rules for each account with the following contents:
| Data | Operation | Parameter |
|---|---|---|
| Action | Parameter | |
This Rule moves the incoming messages with score 96 and greater to the "Junk" mailbox of the original recipeint account. The users should regularly check their "Junk" mailboxes and purge them. The "Discard" action is required to prevent the message from going to the initially intended destination (INBOX mailbox). Note in the example above, the "*" in [XXXXX* is necessary to filter all messages scored above 5 X's. Without it, the rule will only filter out messages with 5 X's.
The "Junk" mailbox from the above example must exist in every account in the domain. Otherwise the Rule will fail and the message will be delivered into the user's INBOX.
Alternatively, you can use "Junk Mail Control" simplified Rule on domain or account level:
| High probability: | Medium probability: | Low probability: |
Scenario #3: suitable for large companies and ISPs for users who don't have access to mailboxes other than INBOX, e.g. POP3 users.
Create one Domain-Wide rule or many Account-level rules for each account with the following contents:
| Data | Operation | Parameter |
|---|---|---|
| Action | Parameter | |
This Rule marks subjects of spam messages with [SPAM] prefix.
Scenario #4:suitable for companies with relatively small input traffic, available from CommuniGate Pro version 5.1 and greater.
In CommuniGate Pro version 5.1 and greater you can enqueue messages synchronously. Use the WebAdmin Interface to configure the Enqueuer component. Open the Queue page in the Settings->Mail realm. Clear off the checkbox of the "Enqueue Asynchronously" option:
| Log Level: | Processors: | ||
| Hop Counter Limit: | Enqueue Asynchronously |
Please see the details in CommuniGate Manual.
Create a Server-Wide Rule with the following contents: <
| Data | Operation | Parameter |
|---|---|---|
| Action | Parameter | |
When enqueueing synchronously, when a message is rejected with a Server-Wide Rule it is rejected on SMTP level with 5xx error code, rather than accepted and bounced.
In any scenario it's not recommend to discard spam messages blindly without saving them because of the possible false positives. It's either highly not recommended to automatically reject spam (unless you're in synchronous mode using scenario#4) because usually the return addresses are forged and the rejection notice message will go to an innocent person or a spamtrap, which may result in your server to become blacklisted. When rejecting in syncronous mode, the sending host will get an error during SMTP transaction and there will be no bounce message generated by your server.
The recommended threshold (the score you start treating messages as spam) is 96. If not enough spam is caught then lower the threshold to 90; if there too many false positives, raise the threshold to 100.
On startup the Cloudmark Plugin reads the contents of the CGCloudmark.cfg file from the current directory. The format of the file data elements is described in http://www.communigate.com/CommuniGatePro/Data.html. The description of the data elements you may find in the CGCloudmark.cfg file. The default CGPCloudmark.cfg is available here.
In initialization time the Cloudmark Engine reads configuration options specified in the cartridge.cfg file and the whitelisting options in whitelist.cfg file.
Micro-updates is the mechanism that allows the Cloudmark Cartridge to regularly download the latest fingerprint data used to identify abusive messages. The fingerprint data is provided by highly trusted reporters and analysis by the Trust Evaluation System in the Cloudmark Collaborative Security Network in near real-time. Therefore, proper use of the micro-update mechanism is critical in maintaining cartridge accuracy as new types of spam, phishing, and virus messages are reported.
See the details in the default cartridge.cfg file.
Micro-updates are downloaded using standard HTTP requests. If an HTTP proxy is not enabled, then at the specified interval, a download will be attempted over port 80. If this fails, the most recent offline version of the data file in the micro_updates directory will be used until the next micro-updates interval before attempting to download again.
The micro-update files containing data are compressed and encrypted. Data that is not encrypted with the correct key will be ignored. As a result, DNS or IP spoofing is not a concern. The contents of the data files are read into memory at startup, as well as each time a new file is downloaded.
By default, the Cloudmark Cartridge sends cartridge configuration information and message scanning statistics back to Cloudmark. By collecting this information, Cloudmark can more effectively detect potential accuracy issues and proactively address them before there is a need for the customer to contact Cloudmark. If your organization has special privacy concerns, you have the option to disable statistics reporting.
Statistics are always collected at each cartridge installation. Collected statistics are not written to disk and they are not accessible at an installation. By default, statistics are reported to Cloudmark at the following URL: http://<configured micro-update hostname>/cmstats The default hostname is microupdates.cloudmark.com. If the cartridge is configured to use a HTTP proxy for micro-update downloads, statistics will be sent using the same HTTP proxy. By default, the cartridge communicates statistics back to Cloudmark every hour. The statistic reporting interval is managed by Cloudmark; Cloudmark may adjust the frequency of statistic reporting interval when assisting customers with accuracy issues.
In the cartridge.cfg file in "customer id = communigate-customername@feedback.cloudmark.com" please change the "customername" to your company name.
A whitelist is a list of trusted senders from whom you always accept email, or email characteristics which indicate a trusted message. This feature of the Cloudmark Cartridge minimizes the filtering of legitimate messages and allows system administrators to conveniently manage the receipt of messages from known safe senders.
See the details in the default whitelist.cfg file.
The Plugin is so-called text-only application which can be launched from a command shell, and it accepts some command line options. When used with CommuniGate Pro as a helper applicaiton the Plugin does not require any command line options.
The Plugin can be used to calculate the spam scores of a number of message files in a directory.
The syntax of the program is: CGPCloudmark RATE [options] <directory>
The <directory> must contain message files in RFC822 format or in CommuniGate format (RFC822 with the envelope info), there must be one message per file. You can use messages form CommuniGate mailbxes in .mdir format. Messages in .EML, .MSG, .mbox and other formats must be converted to the RFC822 format. Be careful not to alter the messages. For example, "Received" headers are often accidentally removed. Note that mail clients very often modify the messages when they save them.
Example: ./CGPCloudmark RATE /var/CommuniGate/Queue
The Cloudmark Network Feedback System is an automatic feedback mechanism that allows Cloudmark ISP and enterprise customers (or CNFS users) to submit misclassified messages directly into the Cloudmark Service. Cloudmark Network Feedback System enables CNFS users to immediately become members of the Cloudmark Collaborative Security Network, joining the millions of users currently providing real-time feedback to the Cloudmark Service on the latest email threats. Cloudmark will immediately begin tracking the reputation of each reporter and gauge their feedback appropriately. Feedback from trusted reporters will lead to near real-time accuracy improvement at the gateway where Cloudmark is filtering spam.
The feedback messages should be mailed to one of the following addresses:
Feedback messages that are not submitted as an RFC822 attachment will not be forwarded into the Cloudmark Service for evaluation. However, these feedback messages will be tracked for statistical analysis purpose.