Neat Little Project - GMail IMAP syncer

Posted by james on Sept. 23, 2011

Problem: there are multiple people that handle incoming emails. They need to coordinate responses, so that any one person can receive & respond to any of the incoming email. They need to be able to see each other's responses to make sure they don't duplicate responses. Also, they should be able to use their existing email (gmail) accounts, and not have to log into a separate specific account.

So in this example, we have our own business with 2 employees. Both need to be able to see the incoming emails and respond to them.

1) They could use a generic email address to send/receive, like some businesses do. The emails are forwarded to both of them. Then they can both receive incoming email.

2) They could BCC the other on *every* outgoing email. This would allow the other to see all outgoing messages, but is very inconvenient and really easy to forget (even just one).

3) They could use a separate email account solely for this, but that's also very inconvenient.

Solution - a program that logs into both accounts, searches for specific messages, and ensures they exist on both accounts. We can use IMAP for this. The benefit is that this takes care of not only incoming emails, but outgoing as well. Also, both employees can have their own email addresses (which is always a plus, since recipients know who they're talking to). And the employees can just read & respond to emails as they normally do, without needing to remember to BCC anything. Also, with separate email accounts they can mark items as read when *they* read it, whereas on a shared account it would be marked read when either of them read it.

The program itself is relatively simple in theory:
-Log into both accounts via IMAP
-Search for messages matching a certain criteria: "from or to: since [some date in the past]"
-Find list of messages on each server that are not on the other
-Copy the messages from one server to the other

The most complicated part of this is working around the limitations of the IMAP protocol. Namely, that there are ID's, UID's, Message ID's, etc.

ID - this is a folder (mailbox) -specific number that changes as messages are deleted/added. Very transient & unreliable. It is part of the IMAP protocol.

UID - this is a folder-specific number that does not change. It is part of the IMAP protocol.

Message ID - this is a message-specific ID that identifies the message, no matter where it lives. It is not account specific. We can treat this as a global UID, since it works across accounts as well. This is part of the email message headers, and is not part of IMAP.

The biggest problem is that most IMAP functions return IDs or UIDs, but not MessageIds. That makes sense, since MessageIds are not IMAP. However, that means the process is more like this:

-Search for messages matching the criteria (returns a list of UIDs)
-Convert that list of UIDs into MessageIds
-Search for MessageIds on the other server (returns a list of UIDs)
-Convert *those* UIDs into MessageIds
-That list to the original server's list to give a list of MessageIds that need to be copied
-Convert this list of MessageIds back into UIDs on server1 so we have a list of messages to copy
-Finally, copy the actual content of the messages from server1 to server2

Really, it's an inefficient bunch of converting UIDs <-> MessageIds, but it works. It'd be nice if all IMAP was based off MessageIds, but those are two separate protocols so it makes sense that they'd be incompatible. And the decoupling of IMAP and email headers means we don't have to update every email server on the planet in order to add another header. So it's annoying, but at least we're not in lowest-common-denominator land.