Email Address Internationalization K. Hurtta (EAI) March 17, 2007 Internet-Draft Intended status: Informational Expires: September 18, 2007 Message Store requirements for Internationalized Email draft-hurtta-eai-messagestore-00 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 18, 2007. Copyright Notice Copyright (C) The IETF Trust (2007). Abstract The Email Address Internationalization (EAI) is implemented by allowing UTF-8 characters in SMTP envelope and mail headers. UTF8SMTP extension of ESMTP takes care that mails with UTF-8 characters in SMTP envelope and mail headers are not delivered to EAI non-compliant SMTP servers. This document describes mechanism how to keep messages with UTF-8 characters on mail headers separated from EAI non-compliant Mail User Agents. This document also describes Hurtta Expires September 18, 2007 [Page 1] Internet-Draft EAI Message Store March 2007 general requirements for UTF8SMTP Message Store. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Message Store with IMAP and POP access . . . . . . . . . . 5 2.2. Message Store with direct file system access . . . . . . . 6 3. Message Store requirements . . . . . . . . . . . . . . . . . . 6 4. Mail Delivery Agent requirements . . . . . . . . . . . . . . . 7 5. Final delivery MTA requirements . . . . . . . . . . . . . . . 8 6. Traditional Unix mailboxes . . . . . . . . . . . . . . . . . . 8 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 8. Security Considerations . . . . . . . . . . . . . . . . . . . 10 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 10 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 10 10.1. Normative References . . . . . . . . . . . . . . . . . . . 10 10.2. Informative References . . . . . . . . . . . . . . . . . . 11 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 11 Intellectual Property and Copyright Statements . . . . . . . . . . 13 Hurtta Expires September 18, 2007 [Page 2] Internet-Draft EAI Message Store March 2007 1. Introduction Internationalized email [ietf-eai-framework] includes UTF-8 characters [RFC3629] on email headers [ietf-eai-utf8headers]. Internationalized email is incompatible with EAI unaware Mail Transport and User Agents. Therefore it is required that Internationalized email are separated from EAI unaware Mail Agents. UTF8SMTP extension [ietf-eai-smtpext] of ESMTP takes care that mails with UTF-8 characters in SMTP envelope and mail headers are not delivered to EAI non-compliant SMTP servers. EAI compliant Message Store need take care that mails with UTF-8 characters in mail headers are not seen by EAI non-compliant Mail User Agents. IMAP protocol extension [ietf-eai-imap-utf8] and POP protocol extension [ietf-eai-pop] takes care that mails with UTF-8 characters in mail headers are not seen by EAI non-compliant Mail User Agents when served via IMAP and POP protocol. Therefore this this document focus more traditional Unix mailboxes, where Mail User Agents access mail via file system access. Focus of this document is on requirements of Message Stores and not actual implementation choice although some suggestions are given. IMAP and POP protocol assumes existence of Message Store. Requirements given on this document apply also Message Stores used by POP and IMAP servers. Mail submission is not discussed on this document. 2. Model Terms "Mail Transfer Agents" (MTA), "Mail User Agent" (MUA), "Message Delivery Agent" (MDA) and "Message Store" (MS) are used according of [crocker-email-arch]. Term "final delivery MTA" is used according of [ietf-eai-framework]. Term "UTF8SMTP message" are used for messages which follow [ietf-eai-utf8headers]. Term "ASCII header message" are used for messages which follow [RFC2822]. Hurtta Expires September 18, 2007 [Page 3] Internet-Draft EAI Message Store March 2007 This document assumes following model for mail architecture. +----------+ +---------+ -- SMTP ---> | final | -----> | | (a) | delivery | (b) | MDA | | MTA | | | +----------+ +---------+ | | write \ / (c) | (---------) ( ) ( MS ) ( ) (---------) | (d) / \ access | | | | +---------+ | | | MUA | | | +---------+ (a) Mail arrives via SMTP [RFC2821]. (b) Final MTA passes mail to MDA with proprietary method or via LMTP [RFC2033]. (c) MDA writes message to MS. (d) MUA access messages on MS with some means. This model is divided to two special model: 1. Case where MUA access MS with POP and IMAP protocol, and 2. Case where MUA access MS via file system access. Hurtta Expires September 18, 2007 [Page 4] Internet-Draft EAI Message Store March 2007 2.1. Message Store with IMAP and POP access This model assumes that MUAs do not access MS via file system access. +----------+ -- SMTP ---> | final | (a) | delivery | | MTA | +----------+ | | (b) | v (----) +-----------------------+ (c) ( ) | MDA | ----- ( ) |.......................| ( ) | | ( MS ) | mail server | (d) ( ) | | ----- ( ) | | ( ) +-----------------------+ (----) | | | (e) IMAP (f)POP (g)HTTP | | | | v | +---------+ +-------+ +--------+ | | | | | WWW | | MUA | | MUA | | browser| | | | | +--------+ +---------+ +-------+ | (-------) ( MUA's ) (folders) (-------) (a)-(c) As on general model. (d) Mail server access messages on MS with some means. (e) MUA access mail server (IMAP server) with IMAP [RFC3501]. (f) MUA access mail server (POP server) with POP [RFC1939]. (g) WWW browser access mail server with HTTP [RFC2616]. Hurtta Expires September 18, 2007 [Page 5] Internet-Draft EAI Message Store March 2007 2.2. Message Store with direct file system access This model assumes that MUA access Message Store with file system access. +----------+ +---------+ -- SMTP ---> | final | -----> | | (a) | delivery | (b) | MDA | | MTA | | | +----------+ +---------+ || (c) write || vv (----------------------------) ( MS [incoming mailboxes] ) (----------------------------) | | | | (d) read,write | | +-------+ | | | MUA | | | +-------+ | (-------) ( MUA's ) (folders) (-------) (a)-(c) As on general model. (d) MUA access MS via file system access. Traditionally Unix incoming mailboxes are files /var/mail/ or /var/ spool/mail/ directory or similar places. User's incoming mail is located on file which name correspond to username of user. These files are on "mbox" format [RFC4155]. NOTE: Also other arrangements and formats exists. 3. Message Store requirements Requirements for UTF8SMTP aware Message Store are following: o UTF8SMTP messages must be accessible to UTF8SMTP aware MUAs on UTF8SMTP form without information lost. Hurtta Expires September 18, 2007 [Page 6] Internet-Draft EAI Message Store March 2007 o UTF8SMTP ignorant MUAs must not see UTF8SMTP messages (messages can be hidden or downgraded [ietf-eai-downgrade] for UTF8SMTP ignorant MUAs). o All messages should be accessible to UTF8SMTP aware MUAs on that form which MS received them. This requirements is needed that possible signatures and hashes calculated for message can be verified. There is two mail reason why UTF8SMTP ignorant MUAs must not see UTF8SMTP messages: o Earlier standards allows only ASCII header fields. Therefore UTF-8 may cause malfunction for MUA. o Address syntax on UTF8SMTP header fields includes fallback address. Therefore UTF8SMTP ignorant MUAs are not able to parse these header fields. Notes: o It is not required UTF8SMTP ignorant MUA can see UTF8SMTP message on downgraded form. When MUAs access MS via file system this is seen to difficult (or wasteful) requirement. o Although this document suggest that messages with ASCII [ASCII] header fields and UTF8SMTP messages are handled separately it is allowed that UTF8SMTP aware MS handles all messages as UTF8SMTP messages. ASCII header message can be treated as subset of UTF8SMTP messages. This implies that UTF8SMTP ignorant MUA does not see any message (if downgrading is not provided). o On some environment it is common that Subject: and some other header fields are not encoded sometimes, but use some random 8-bit (presumably system local) character set. If used character set is not UTF-8, these are not UTF8SMTP messages. Therefore these messages can not be treated as UTF8SMTP messages as ASCII header messages can be treated. o First requirement (messages accessible on UTF8SMTP form) practically forbids storing messages only on downgraded form. 4. Mail Delivery Agent requirements Requirements for UTF8SMTP aware Mail Delivery Agent are following: o MDA must not deliver UTF8SMTP messages to UTF8SMTP unaware Message Store. Notes: o In practice this means that if Mail Delivery Agent is UTF8SMTP, MS must be constructed that way that requirements previous section (Section 3) are fulfilled. Hurtta Expires September 18, 2007 [Page 7] Internet-Draft EAI Message Store March 2007 o UTF8SMTP messages does not include label, which which tells which messages are UTF8SMTP messages and which messages are ASCII header messages. To discover message is UTF8SMTP message may require that all message header fields (including header fields from MIME body parts) are parsed. Because MDAs are not expected to parse messages, it is suggested that final delivery MTA pass that information to MDA. o On some legacy environments it is common that Subject: and some other header fields are not encoded. Therefore presence of 8-bit bytes itself does not indicate that message is UTF8SMTP message. Test is also needed that sequence of 8-bit bytes forms UTF-8 characters. 5. Final delivery MTA requirements UTF8SMTP aware MTAs must fill requirements of [ietf-eai-smtpext]. For UTF8SMTP aware final delivery MTA there is following additional requirements: o Final delivery MTA must not deliver UTF8SMTP messages to UTF8SMTP unaware mail delivery agent. Notes: o If final delivery MTA is UTF8SMTP aware, it is recommended that MDA is arranged that way that it is UTF8SMTP aware. o If communication between final delivery MTA and MDA use LMTP, UTF8SMTP response to LHLO command tells that MDA is UTF8SMTP aware. o In general final delivery MTA can be expected to parse messages, so it knows when them are UTF8SMTP messages. Passing that information to MDA may require that LMTP is extended. 6. Traditional Unix mailboxes On this section suggestion to how handle traditional Unix mailboxes is given. Let's assume that incoming mail for user is stored to /var/mail/ {username} file on UTF8SMTP unaware MS by using "mbox" format [RFC4155], where {username} is user's account name. To make MS UTF8SMTP aware, MS is modified following way: o UTF8SMTP messages are stored to /var/mail/{username}:UTF8 file by MDA (for UTF8SMTP messages ':UTF8' is appended to name of file.) If environment is fully UTF8SMTP aware, all messages are stored to /var/mail/{username}:UTF8 file by MDA. Hurtta Expires September 18, 2007 [Page 8] Internet-Draft EAI Message Store March 2007 o ASCII header messages can be still stored to /var/mail/{username} file. o MDA is allowed (but not required) to store copy of UTF8SMTP messages on downgraded form to /var/mail/{username} file. This means that if downgrading is provided, storage space requirements are doubled. If downgrading fails, MDA must not reject message, but instead just store original UTF8SMTP message to /var/mail/ {username}:UTF8 file. o On some cases it is common that Subject: and some other header fields are not encoded and use some random 8-bit character set (presumably local character set of system). If UTF-8 is not used on header fields, these messages are not UTF8SMTP messages. Therefore they are not stored to /var/mail/{username}:UTF8 file. Handling of these messages is not changed. It is very unlikely that Unix account names includes ':' characters, therefore it is expected that ':UTF8' suffix does not conflict with user's account names. UTF8SMTP aware MUA needs to do following: o When incoming mailbox is opened, both files /var/mail/{username} and /var/mail/{username}:UTF8 are parsed. o If message with same Message-ID header field is presented both on /var/mail/{username} and /var/mail/{username}:UTF8 file, only message from /var/mail/{username}:UTF8 file is presented. o UTF8SMTP aware MUA must not write UTF8SMTP messages to /var/mail/ {username} file. It is recommended that configuration for MDA is provided, which allows specify o which users accepts only non-UTF8SMTP messages, and o which users accepts also UTF8SMTP messages. In lack of configuration, following heuristic is suggested: o Presence of /var/mail/{username}:UTF8 file indicates that user accepts UTF8SMTP messages. o If file /var/mail/{username} exists and /var/mail/{username}:UTF8 file do not exists, that can be treated that user accepts only non-UTF8SMTP messages. o Presence of /var/mail/{username}:UTF8 file without /var/mail/ {username} file can be treated that all messages should be treated as UTF8SMTP. o If either /var/mail/{username} or /var/mail/{username}:UTF8 file exists, that can be treated as temporary error condition. 7. IANA Considerations There is no IANA considerations in this document. Hurtta Expires September 18, 2007 [Page 9] Internet-Draft EAI Message Store March 2007 8. Security Considerations If user uses UTF8SMTP unaware MUA, UTF8SMTP messages may look for his/her that they go to bit bucket although they appear to be delivered as far sender is considered. See "Security considerations" section in [ietf-eai-framework] for more discussion. 9. Acknowledgements Various requirements and ideas are suggested on IMA mailing list discussions. 10. References 10.1. Normative References [RFC2822] Resnick, P., Ed., "Internet Message Format", RFC 2822, April 2001. [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", RFC 3629, STD 63, November 2003. [ietf-eai-framework] Klensin, J. and Y. Ko, "Overview and Framework for Internationalized Email", draft-ietf-eai-framework-05 (work in progress), February 2007. [ietf-eai-utf8headers] Yeh, J. and Abel, "Internationalized Email Headers", draft-ietf-eai-utf8headers-04 (work in progress), March 2007. [ietf-eai-downgrade] YONEYA, Y., Ed. and K. Fujiwara, Ed., "Downgrading mechanism for Email Address Internationalization", draft-ietf-eai-downgrade-03 (work in progress), March 2007. [ietf-eai-smtpext] Yao, J., Ed. and W. Mao, Ed., "SMTP extension for internationalized email address", draft-ietf-eai-smtpext-04 (work in progress), March 2007. [crocker-email-arch] Hurtta Expires September 18, 2007 [Page 10] Internet-Draft EAI Message Store March 2007 Crocker, D., "Internet Mail Architecture", draft-crocker-email-arch-06 (work in progress), March 2007. 10.2. Informative References [ASCII] American National Standards Institute (formerly United States of America Standards Institute), "USA Code for Information Interchange", ANSI X3.4-1968, 1968. ANSI X3.4-1968 has been replaced by newer versions with slight modifications, but the 1968 version remains definitive for the Internet. [RFC1939] Myers, J. and M. Rose, "Post Office Protocol - Version 3", RFC 1939, STD 53, May 1996. [RFC2033] Myers, C., "Local Mail Transfer Protocol", RFC 2033, October 1996. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC2821] Klensin, J., Ed., "Simple Mail Transfer Protocol", RFC 2821, April 2001. [RFC3501] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1", RFC 3501, March 2003. [RFC4155] Hall, E., "The application/mbox Media Type", RFC 4155, September 2005. [ietf-eai-imap-utf8] Resnick, P. and C. Newman, "IMAP Support for UTF-8", draft-ietf-eai-imap-utf8-01 (work in progress), March 2007. [ietf-eai-pop] Newman, C., "POP3 Support for UTF-8", draft-ietf-eai-pop-01 (work in progress), January 2007. Hurtta Expires September 18, 2007 [Page 11] Internet-Draft EAI Message Store March 2007 Author's Address Kari Hurtta Kala-Matti 4 B 24 02230 Espoo FI Email: hurtta-ietf@elmme-mailer.org URI: http://iki.fi/keh/ Hurtta Expires September 18, 2007 [Page 12] Internet-Draft EAI Message Store March 2007 Full Copyright Statement Copyright (C) The IETF Trust (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Acknowledgment Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA). Hurtta Expires September 18, 2007 [Page 13]