CommuniGate Pro
Version 6.1
Storage
 
 
Mailboxes

Mailboxes

CommuniGate Pro Accounts contain one or several Mailboxes. Each Mailbox has its own unique name and it can contain zero or more messages. The POP, IMAP, MAPI, XIMSS, WebUser Interface, and Real-Time Application modules provide access to Account Mailboxes.

Several storage formats can be used for CommuniGate Pro Mailboxes. A multi-Mailbox Account can contain Mailboxes stored in different formats.

Each Account always has the INBOX Mailbox. Any message delivered to a CommuniGate Pro Account is stored in its INBOX Mailbox - unless some Automated Processing Rules instruct the Server to store the message in a different Mailbox.

Mailbox Names

When an Account is created, its INBOX Mailbox is automatically created. The System and/or Domain Administrator can specify additional Mailboxes to be created at that time.

A user can create a Mailbox using an IMAP, MAPI, or XIMSS mailer application or using the WebUser Interface.

Mailboxes can be "nested": for any Mailbox "A" you can create a sub-Mailbox "B" - in the same way as you can created a file directory inside some other file directory. The CommuniGate Pro Server uses the slash (/) symbol as the hierarchy separator: INBOX/important is the name of the sub-Mailbox important "inside" the INBOX Mailbox.

CommuniGate Pro allows you to store messages in some Mailbox X and at the same time you can create sub-Mailboxes X/Y, X/Z for that Mailbox. This feature is implemented by providing two "invisible" Mailbox entities - one for storing messages, one - for serving as a "directory" for the nested Mailboxes. The "directory" entity is created automatically, as soon as you try to create the first sub-Mailbox. You can, though, create the "directory" entity without creating the "mail storage" entity: use the ABCDEF/ name as the new Mailbox name to create only the directory entity with the ABCDEF name. The name ABCDEF will be listed, but will not be "selectable" - and you will not be able to store messages in the ABCDEF Mailbox. You can later create the regular ABCDEF Mailbox and the "storage" entity for your ABCDEF Mailbox name will be added.

It is impossible to delete the INBOX Mailbox. You can rename the INBOX Mailbox, though. In this case a new empty INBOX Mailbox will be created automatically.

Mailbox names are case-sensitive. Some file systems (NTFS, for example) provide case-insensitive file naming conventions. When these file systems are used for CommuniGate Pro Account/Mailbox storage, the Mailbox names are still case-sensitive, but you cannot create two Mailboxes with names that differ in case only. The INBOX Mailbox name is an exception: it is always a case-insensitive name.


Message Flags

Messages in Mailboxes have individual flags. These flags can be set when the message is being stored in the Mailbox, and they can be updated using Mailbox access protocols and methods, such as IMAP, MAPI, XIMSS, WebUser Interface, Real-Time Applications.

Some flags are set automatically, even when the access protocol used does not support flag modification. For example, the Seen flag is set automatically when the message is being read using the POP protocol RETR command.

Several components (such as Automated Rule, CG/PL programs, etc.) can access message flags by name. They can also use "negative names" to instruct the server to reset a certain flag or to look for messages that do not have that flag set.

The following table lists the supported message flags along with their IMAP and Negative names:
NameDescriptionIMAP NameNegative Name
Seen This flag is set when the message was read by a client. It can be set automatically as a result of certain Mailbox access operations, and it can be set and reset explicitly with mail client applications. \SeenUnseen
Read same as Seen  Unread
Answered This flag is set when a reply was sent for this message. This flag is explicitly set and reset with mail client applications. \AnsweredUnanswered
Flagged This flag is set to attach a "flag" to the message (for example, a mail client can show this message to the user as an important one). This flag is explicitly set and reset with mail client applications. \FlaggedUnflagged
Draft This flag is set for messages that have not been sent yet. It tells a mail client that it can open and edit this message. This flag is explicitly set and reset with mail client applications. \DraftUndraft
Deleted This flag is set for messages that were marked for deletion. Some mail clients allow users to mark some Mailbox messages first, and then delete ("expunge") all marked messages from the Mailbox. This flag is explicitly set and reset with mail client applications. \DeletedUndeleted
Redirected This flag is set when a copy of the message was sent (redirected) to someone. This flag is explicitly set and reset with mail client applications. $ForwardedNotRedirected
MDNSent This flag is set when an MDN ("read report") for the message has been sent. This flag helps mail clients to send only one MDN report for each message. This flag is explicitly set and reset with mail client applications. $MDNSentNoMDNSent
Hidden Messages with this flag set are visible only to the Mailbox Account owner and to those users who have the Admin Access Right for this Mailbox.
This flag allows users to grant access to their Mailboxes to others while keeping certain messages private (hidden).
$HiddenNotHidden
Service Messages with this flag set are not visible to IMAP or POP clients.
MAPI clients can use this flag to create service items invisible to users (such as Mailbox forms).
$ServiceNotService
Media If this flag is set, the message is treated as containing some "media" (audio/video) data. $MediaNotMedia
Junk If this flag is set, the message is treated as "junk" (spam). JunkNotJunk
Label1 Custom flag. $Label1NotLabel1
Label2 Custom flag. $Label2NotLabel2
Label3 Custom flag. $Label3NotLabel3

Mailbox Access Rights

The CommuniGate Pro Server maintains an Access Control List (ACL) for every Mailbox it creates.

The Access Control Lists are used to control the Foreign Mailbox Access feature that allows Account users to access Mailboxes in other Accounts.

A Server Administrator with the All Users access right has all access rights to all Mailboxes in all Server or Cluster Accounts.

Domain Administrators with the CanViewMailboxes access right have all access rights for all Mailboxes in their Domains.

The Mailbox Account owner can grant certain limited access rights to other users, using the Access Control Lists.

The following Mailbox access rights are supported:
l (Lookup)
If you grant a user the Lookup access right, that user will be able to see this Mailbox when it asks the Server to list all Mailboxes in your Account.
r (Read/Select)
If you grant a user the Read access right, that user will be able to open (select) this Mailbox and see (read) the messages in this Mailbox.
s (Seen)
If you grant a user the Seen access right, that user will be able to mark messages as read (seen). Usually a message is automatically marked as seen when a user reads it. But if this access right is not granted to a user reading the Mailbox, the Mailbox message "seen" status will not be changed.
w (Write/Flags)
If you grant a user the Write access right, that user will be able to set message flags: i.e. to mark messages as answered or "flagged", and to reset the message flags.
d (Delete)
If you grant a user the Delete access right, that user will be able to mark messages as deleted and to compress the Mailbox, removing all its messages marked as deleted.
i (Insert)
If you grant a user the Insert access right, that user will be able to append messages to this Mailbox and to copy messages from other Mailboxes into this one.
p (Post)
This access right is not used by modern mailers.
c (Create)
If you grant a user the Create access right, that user will be able to create new Mailboxes "inside" this Mailbox.
a (Administer)
If you grant a user the Administer access right, that user will be able:
  • to modify the Mailbox ACL
  • to modify the Mailbox meta-data (such as the Mailbox Class)
  • to see Hidden Mailbox messages

When a sub-Mailbox is created, it inherits the ACL of the outer (parent) Mailbox. This means that if you create the INBOX/sales Mailbox, it is created with the same ACL as specified for the INBOX Mailbox.

To delete a foreign Mailbox, a user should have:

To rename a foreign Mailbox, a user should have:

To create a foreign Mailbox, a user should have the Create access right for the outer (parent) Mailbox of the Mailbox to be created.

When the target Mailbox is a "top" one in an Account, and thus there is no outer (parent) Mailbox, the "CreateMailboxes" Account Access Right is checked instead.

The Mailbox Access Control Lists can be set and modified using the WebUser Interface, a XIMSS, a MAPI, or a decent IMAP client.


Mailbox Formats

CommuniGate Pro stores received messages in Account Mailboxes. The server supports several Mailbox formats, and the Mailbox type is defined by the Mailbox file (or directory) name extension.

For single-Mailbox Accounts, the Mailbox type is specified when the Account is created.

Each multi-Mailbox Account has a setting that specifies the default type for all new Mailboxes created in this Account. A user can explicitly specify the Mailbox type creating a Mailbox in a multi-Mailbox Account: if the Mailbox name is specified as name.extension, then the Mailbox name of the extension type is created.

The Text Mailbox (.mbox) Format

The Mailbox files with this extension store messages in the legacy BSD mailbox format. Each message in the Mailbox is preceded with the a From-line:
From <>(flags-UID-origUID) time stamp
This is the same format as one used in legacy mail systems, but with a "comment" added after the return-path part. The .mbox format remains compatible with legacy applications (local mailers), and at the same time it allows the CommuniGate Pro Server to store the required message information (message status flags, the unique Mailbox message ID, and, optionally, the "permanent" message UID).

If a Mailbox file has been copied from an old system, or when it is used as an Legacy INBOX and old applications can add messages to this Mailbox, some messages may have no "comment;" part. CommuniGate Pro allows a user to work with such messages, but it does not store message flags if they were modified, and it does not remember the message UIDs between sessions. The simplest solution is to copy such messages to a different Mailbox and then copy them back to the original Mailbox - the copy operation places the correct information into the From-line.

When a message is being stored in the .mbox-type Mailbox, all message lines are checked. If there is an empty line followed with the line starting with the letters From, the '>' symbol is inserted before the letter F.

The Text Mailboxes become less effective as their size grows. When a Text Mailbox is being opened, it has to be parsed, in order to detect message boundaries and retrieve the UID, flags, and other per-message information. When some messages are being deleted from the middle of a Text Mailbox, the Server has to copy the remaining messages data, compressing the Mailbox. To make these processes more efficient, the CommuniGate Pro server can deal with Mailbox data in large chunks. A special semaphore object limits the number of buffers allocated for large Mailbox processing. Changing this parameter can change the overall large Mailbox access (you may want to increase or decrease it, depending on the OS and file system you use).

To improve Text Mailbox opening speed, the CommuniGate Pro can maintain a Mailbox index (.bdx) file alongside the Text Mailbox Mailbox file. If the index file exists, the Server reads it instead of parsing the entire Mailbox file. CommuniGate Pro automatically creates an index file when it the Mailbox file size exceeds the specified limit. The Server removes the index file if the Mailbox becomes smaller than that limit.
The Index file is created when any message in the Mailbox is modified or deleted. If new messages have been added to the Mailbox, but the Mailbox has not been opened, or it has been only read without any flag modification, the Index file may not be created.

To store Message Attributes, special "invisible" messages are created in the Text Mailbox.

Use the WebAdmin Interface to specify the Text Mailbox Manager settings. Open the General pages in the Settings realm, and find the Text Mailbox Manager panel on the Others page:

Text Mailbox Manager
Concurrently used large buffers:
Index Mailboxes larger than:
Concurrently used large buffers
Use this setting to specify how many concurrent operations (parsing, deletion) the Text Mailbox Manager can perform on large Mailboxes.
Index Mailboxes larger than
Use this setting to specify the minimal size for Mailboxes that need indexing.

The MailDir Mailbox (.mdir) Format

Mailboxes with this extension are file directories. Each Mailbox message is stored as a separate file in the Mailbox directory.

The message file name has the following format:
iiiiOoooo-flags-timestamp-nLines
where iiii is the message unique ID, Ooooo is the message permanent UID (optional), flags are the message status flags, the timestamp is the message internal time stamp - the time (GMT) when the message was added to the Mailbox, in the yyyymmddhhmmss format, and nLines is the number of text lines in the message.

Note:On the Unix platforms, the MailDir Mailboxes implement the shared storage model: if the same message is directed to many Accounts/Mailboxes, only one message file is created, and a hard link to that file is placed into each Mailbox directory.
When a message is removed from all Mailboxes, the file is automatically deleted by the OS.
This feature is disabled for the Accounts that have the Erase Deleted Messages Setting enabled.

To store Message Attributes, special files with Ooooo names are created inside the Mailbox directory.

Note: most freeware mail systems use either the mbox-like or mdir-like formats, and designers of those systems make various claims about the advantages of the formats they have selected. It is important to remember that:

Note: the Text format is more efficient than the MailDir format in most cases, this is why this format is used as the default one. The MailDir format is recommended only for those Mailboxes that contain many (100 or more) large (1MB or more) messages. If a user has a Proposals Mailbox where she stores all messages with attached documents, each 0.5-5MB in size, then this Mailbox may work faster if it is created in the .mdir format.

The Sliced Mailbox (.mslc) Format

Mailboxes with this extension are file directories. These directories contain dataNNNN files, each file contains one or more messages, in the format similar to one used in Text Mailboxes.

The index.bdx file contains the mailbox index, and includes information for all messages stored in all data files.

As new messages are added to such a Mailbox, they are either appended to the existing data files, or stored in a newly created data file, depending on the message size and the size and number of messages stored in the existing data files.

When messages are removed, data files can be merged to keep the total number of data files low.


Mailbox Classes

Each Mailbox can have a Class attribute. This attribute specifies the type of the information this Mailbox is created for: Calendar, Contacts, Tasks, Notes, etc. If a Mailbox does not contain the Class attribute, it means that it is created to store regular E-mail messages.

The Mailbox Class does not restrict the types of data that can be stored in the Mailbox: E-mail and Contacts messages can be stored in Mailboxes with the Tasks Class, Notes messages can be stored in Calendar Class Mailboxes, etc. The Mailbox Class information is used with the advanced user interfaces (WebUser, MAPI) to present the Mailbox content in the proper format.

When a Mailbox is created with an advanced client interface, the interface can set the Mailbox Class. Mailbox Classes can also be updated using the CommuniGate Pro CLI/API.


Special Mailboxes

Some Mailboxes have special meanings.

The INBOX Mailbox receives all incoming E-mail messages, unless these messages are explicitly directed to other Mailboxes with Rules or using Direct Mailbox Addressing.
When an Account is created, its INBOX Mailbox is created automatically.

Other special Mailboxes do not have fixed names. The Account user or an administrator can specify modify the name used for any special Mailbox. These names are stored in Account Preferences.

The CommuniGate Pro Server recognizes Special Mailbox names (starting and ending with the $ symbols) as references to special Mailboxes. This feature allows client applications to access special Mailboxes in any Account, without being reconfigured to use the actual name of those special Mailboxes. These Special names do not show up in the Account Mailbox lists.

Trash
This Mailbox is used to store messages to be deleted. Clients can move messages from other Mailboxes to this Mailbox, using it as a "Tash can". Special name: $Trash$
Sent Messages
This Mailbox is used to store copies of sent E-mail messages. Special name: $Sent$
Drafts
This Mailbox is used to store E-mail message drafts. Special name: $Drafts$
Junk
This Mailbox is used to store junk E-mail messages. Special name: $Junk$
History
This Mailbox is used to store "log"-type messages, such as copies of the IM sessions. Special name: $History$
Default Calendar
This Mailbox is used as the Account main Calendar Mailbox. This Mailbox is expected to be of the Calendar class. Special name: $Calendar$
Default Task List
This Mailbox is used as the Account main Tasks Mailbox. This Mailbox is expected to be of the Tasks class. Special name: $Tasks$
Default Contacts
This Mailbox is used as the Account main Contacts Mailbox. This Mailbox is expected to be of the Contacts class. Special name: $Contacts$
Default Notes
This Mailbox is used as the Account main Notes Mailbox. This Mailbox is expected to be of the Notes class. Special name: $Notes$

Locked Mailboxes

Each Mailbox can have the Locked attribute. If this attribute is set, the Mailbox cannot be deleted or renamed.

A locked Mailbox can be deleted or renamed together with its parent Mailbox, if the parent Mailbox itself is not locked.

You can specify the Locked attribute for the Mailboxes created using the Account Template. The Mailbox Locked attribute can also be updated using the CommuniGate Pro CLI/API.


Creating Mailboxes

Every Account has a setting that specifies the default format for new Mailboxes that can be created in this Account.

The Account user can explicitly specify the storage format for a new Mailbox by adding the format extension to the new Mailbox name. If a user tells the CommuniGate Pro Server to create the newmailbox.mdir Mailbox, the .mdir-formatted Mailbox newmailbox is created.


Mailbox Subscription

The CommuniGate Pro Server allows an Account user to subscribe to some Mailboxes. The Account Mailbox subscription is a simple list of Mailbox names. This list is not used by the Server itself - the Server just stores one subscription list for each Account.

Many IMAP mailers use the Account subscription list and show only the Mailboxes the Account is subscribed to. The WebUser Interface can also be configured to show only the subscribed Mailboxes.

You can modify the Account subscription either via a decent IMAP mailer, or using the WebUser Interface.

You can use the Account Mailbox subscription to make some not-so-decent IMAP mailers access foreign Mailboxes: make sure that your IMAP client is configured to use the Account Mailbox subscription, and add the desired foreign Mailbox name into the subscription list.

Note:Some IMAP mailers tend to rebuild Account subscription lists: they empty the subscription, and then subscribe you to all Mailboxes in your own Account.

The Account Mailbox subscription is stored in the Account .info service file.


Mailbox Aliases

Many IMAP clients (such as Microsoft Outlook and Outlook Express) and AirSync clients (such as Windows Mobile, Apple iPhone, Nokia PDAs, etc.) cannot handle foreign Mailboxes directly, and they cannot use the Account Mailbox subscription to access foreign Mailboxes.

Mailbox Aliases can be used to let these clients access foreign Mailboxes.

A Mailbox Alias is a name associated with some [foreign] Mailbox name. For example, you can create a Mailbox Alias salesBox for the ~sales/INBOX Mailbox name. You will see the salesBox Mailbox in your client application, but in reality this will be the INBOX Mailbox in the sales Account.

Mailbox Aliases can be created only on the topmost level of the Account Mailbox hierarchy, that means that the Mailbox Alias name cannot contain the slash (/) symbol.

A Mailbox Aliase provides access to the Mailbox it is associated with, and to all its sub-mailboxes (subject to Access Control List restrictions).

A Mailbox Alias can contain just a foreign Account name (~accountName). Such an Alias provides access to all accessible Mailboxes in that foreign Account. The Mailbox Alias itself is presented as an unselectable Mailbox name.

Sample configuration:

The owner of the Account chief has granted "lookup" and other access rights for his Mailboxes INBOX and Pending to the assistant Account.

The user assistant has created the Mailbox Alias boss pointing to ~j.smith.

When the user assistant connects to her Account using any IMAP or XIMSS client, or the WebUser Interface, she sees all her own Mailboxes, the unselectable Mailbox boss, and also the boss/INBOX and boss/Pending Mailboxes.

If the user j.smith creates a new Mailbox Urgent in his Account and grants access rights for that Mailbox to the assistant Account, the user assistant will immediately see the new Mailbox as the boss/Urgent Mailbox.


Simultaneous Access

The CommuniGate Pro Server allows several client applications to connect, open the same Mailbox, and read and modify the Mailbox data at the same time.

The CommuniGate Pro multithreaded design allows the Server to synchronize client activities without using OS-level file locks and it does not require a client to wait till all other clients close the Mailbox.

Simultaneous Access means that:

Clients accessing the same Mailbox can use the same or different Mailbox access protocols - POP, IMAP, MAPI, AirSync, WebUser, or XIMSS Interface.

Simultaneous Access is supported for all Mailbox types implemented in the CommuniGate Pro software.

This feature allows you to work with your Mailbox from several workstations and devices, and it lets a group of people (i.e. the sales department) process messages in one centralized Mailbox.


Foreign and Public Mailboxes

The CommuniGate Pro allows an Account user to access Mailboxes in other Accounts.
Access to these foreign Mailboxes (also called shared Mailboxes) is controlled via the Mailbox Access Rights.

To access a Mailbox in a different Account, the Mailbox name should be specified as ~accountname/mailboxname. For example, to access the INBOX Mailbox in the Boss Account, the Mailbox name should be specified as ~Boss/INBOX .

If there are several local Domains on the Server, Mailboxes in a different Domain can be accessed by specifying full Account names. To access the LIST/reports Mailbox in the Account ListMaster in the client.com Domain, the Mailbox name should be specified as ~ListMaster@client.com/LIST/reports.

Account names specified after the "~" sign are processed with the Router, so Account Alias names can be used instead of the real Account names, and all Routing Table rules are applied.

Very often Foreign Mailboxes are used:

CommuniGate Pro can provide "public" Mailboxes, too. This can be done by creating an Account public, and assigning public Access rights to its Mailboxes. Usually, each group of public Mailboxes is managed by some administrator, who is not required to be a CommuniGate Pro administrator.

A CommuniGate Pro Server administrator should create the public Account, log into that Account using the WebUser Interface or a decent IMAP client, create some public Mailboxes, and grant administration rights to regular users that will administer these public Mailboxes. Those users will then grant access rights to other users, create sub-Mailboxes, and perform other administrative tasks.

For example, a public Mailbox administrator can use Automated Rules to copy certain incoming messages directly into some public Mailbox.

Some IMAP clients (such as Microsoft Outlook and Outlook Express), and most AirSync clients do not support foreign Mailboxes at all. To let those clients access shared Mailboxes in other Accounts, Mailbox Aliases can be used.


Legacy Mailboxes

On some systems users have direct (login) access to the mail server computer, and some of them get used to Local Mailers - mail, elm, and others. Local Mailers do not use any network protocol to access mailboxes. Instead, those programs read and modify mailbox files directly, via the file system.

The CommuniGate Pro allows you to create Accounts with Legacy INBOX Mailboxes. These Mailboxes are stored not inside the CommuniGate Pro base directory, but in the system directory known to the legacy mailer applications.

Since these INBOX files can be read and modified directly, bypassing the CommuniGate Pro protocols and modules, the Server needs to synchronize its activity with legacy mail applications using OS file locking features - either FileLevel locks or FileRange locks.

On Unix systems the FileLevel locks are known as flock operations, and RangeLevel locks are known as fcntl operations. Check with your OS manual to see which method the legacy mailers use on your system, and configure the CommuniGate Pro Server to use that method. For systems that support only one file locking mechanism (MS Windows, Sun Solaris, and some other systems), selecting either method selects that mechanism.

You should use Legacy Mailboxes only when absolutely necessary, because:

If you have to support Local Mailer compatibility for all or some Accounts in a Domain (usually - in the Main Domain), you should specify the Legacy INBOX settings for that Domain.

When you create an Account that has an Legacy INBOX, the Server checks if the Account INBOX file already exists in the specified location and creates one if the Mailbox file is absent.

When you delete an Account that has an Legacy INBOX, the Server does NOT remove the INBOX Mailbox file.


CommuniGate® Pro Guide. Copyright © 1998-2014, Stalker Software, Inc.