DCNET Mail Plan
D.L. Mills and M.J. O'Connor COMSAT Laboratories 23-Jul-81
The Distributed Computer Network (DCNET) is an
experimental research network in use at COMSAT and elsewhere. It includes a number of PDP11-compatible hosts connected to each other and to ARPANET, SATNET and other networks accessable to the DARPA Internet Project. The network and its hosts are used for research in computer network protocols and for general-purpose software development. One of the principal functions of the network is to support electronic mail, including the capability to construct and edit messages on-line, forward them to one or more hosts on DCNET, ARPANET or elsewhere and to retrieve and archive incoming messages from these hosts. These capabilities have, of course, been available and extensively used for some time on ARPANET hosts and on commercial services such as HERMES, ONTYME and TELEMAIL.
All DCNET hosts presently support both the Internet
Protocol (IP) and Transmission Control Protocol (TCP), which have been implemented on many computers and operating systems and have been adopted as DoD standards . High-level protocol modules allow DCNET hosts to connect to ARPANET hosts as virtual terminals and to perform mail functions using the ARPANET hosts in the ordinary way. In addition, files can be exchanged between DCNET and ARPANET hosts so that, in principal, messages arriving at ARPANET hosts can be relayed to DCNET hosts for furthur processing and archiving.
One of the tasks addressed in our present Internet
Project activities is to investigate mechanisms with which mail functions can be performed directly in small hosts, rather than requiring support from larger ARPANET service hosts. Besides reducing the network resources required and providing potentially better performance, such mechanisms would greatly simplify integration of speech and facsimile media into the message system. We have been working to define and develop such mechanisms for some time now and have completed a version suitable for general use. We believe that this establishes the feasibility of performing nearly all mail functions in small hosts of the LSI-11 variety and yet maintain complete compatibility with existing hosts and their protocols. The remainder of this memorandum describes the architecture of this system and demonstrates its use. DCNET Mail Plan PAGE 2
2. DCNET Functions and Features
The DCNET was conceived as a prototype and testbed for
distributed network architectures. All DCNET hosts use variants of a common operating system called the Basic Operating System (BOS), which includes the usual supervisor services together with the capability to emulate the DEC RT-11 operating system environment and run ordinary RT-11 system and application programs. Support for IP and TCP is embedded in the BOS together with an interface to application programs, including a set of high-level protocol modules supporting virtual-terminal (TELNET), file-transfer (FTP, NIFTP) and various utilities (XNET, PING, NAME and WATCH).
The most common application of a DCNET host is in
single-user mode. Although the software can support simultaneous access by a number of users, the typical hardware configuration includes only a modest amount of on-line storage and can support only a limited set of applications in multi-user mode. In the case of those hosts equipped with dual floppy-disk drives, for example, the usual mode of operation is for each user to mount a personal disk containing private files on one of the drives and a system disk containing public files on the other.
All DCNET hosts participate in network functions such
as routing, multiplexing, network monitoring, timekeeping and various utility "fake host" functions useful for testing with other internet hosts. Some hosts are given more specific responsibiliites. For instance, two LSI-11 machines are presently used as multiplexors for a number of other machines and as gateways to ARPANET and SATNET. Another instance of specialization involves a host equipped with digital-facsimile and digital-speech peripherals which are used in experiments in multi-media message systems.
3. The DCNET Mail System
The most essential component in any electronic mail
system is on-line storage. It has been common experience that rather a lot of it is required for even a modest number of users involved in an active research community such as the Internet Project. To be useful, a modest amount of this storage should be reachable from other internet hosts at all times, since those hosts holding unsent mail typically attempt to forward it immediately upon receipt. We are currently using disks with a capacity of between 10 and 20 megabytes for this purpose and believe this sufficient for the volume of mail expected, as well as for a general DCNET data and archive base. One of the disks is attached to a DCNET host expected to be available for mail access substantially all the time; however, this host may DCNET Mail Plan PAGE 3
occasionally be unavailable for short periods due to program development. For subsequent reference this host will be called the mail host.
The mail host serves as a DCNET post office and
forwarding depot, but is not ordinarily used for general-purpose application programs. The remaining hosts are used for these programs by various individuals on an intermittent basis. In the typical scenario, a user mounts his personal disk on one of the local hosts, contacts the mail host and interrogates its data base for his new mail. Upon inspection of the mail, the user disposes of it in one of several ways, including: (1) deletes it, in which case it is gone forever; (2) forwards it to the local host for later processing; (3) copies it to a file or printer at the mail host, local host or some other host. Implicit in this scenario is the expectation that the volume of mail will require that each user individually archive his mail on his cache of personal disks as required. Also implicit is the requirement that some forms of mail, in particular multi-media speech and facsimile, will require access to a host with the required peripheral equipment. We expect eventually to provide automatic forwarding features that do not require direct interrogation of the mail host.
In order to send mail, the user constructs and edits
each message at the local host, perhaps incorporating messages and files from other hosts including the mail host. Once the message has been prepared and prefixed with a list of recipients in a standard header, the user initiates transmission in one of two ways: (1) opens connections with each recipient internet host and transmits the message directly or (2) forwards the message to the mail host for later onward relay. The user would naturally elect the former if speed was important and the latter if the recipient host was not responding at that time.
4. Mail Protocols
The mechanisms designed and implemented in DCNET hosts
to support the above scenarios must be compatible with those used elsewhere in the internet community. The existing ARPANET mail system evolved as a feature of the File Transfer Protocol (FTP) used to transport files between ARPANET hosts. The original FTP was described in a working document called RFC-542 and the format of the mail message itself in RFC-733 . The FTP described in RFC-542 is, however, not compatible with the present internet protocols and therefore is unsuitable for use outside the ARPANET. A version of FTP compatible with TCP has been proposed in RFC-765 ; however, there are few servers operational at present which conform to this protocol. DCNET Mail Plan PAGE 4
In order to provide mail support for systems using TCP
and for interworking with the existing ARPANET systems, a new protocol called the Mail Transfer Protocol (MTP)  was developed and described in RFC-780. The MTP is designed to operate with current internet protocols and, in addition, to provide for onward relay of mail into networks that do not conform to these protocols, such as MMDF and NITS . Preliminary versions of MTP have been implemented at ISI for TOPS-20 hosts, at MIT for Multics hosts, at DCEC for Unix hosts and at COMSAT for DCNET hosts.
The MTP is regarded as an intermediate step between the
ARPANET-specific FTP-based mail and a proposed new system called the Internet Message Protocol (IMP) , which provides much greater operational flexibility together with the capability to process multi-media data types. The MTP can provide that now only through ad-hoc protocol extensions. However, it is likely that the MTP will be used for some time until the necessary software can be constructed for the ARPANET hosts. The IMP, which is described in RFC-759, is now being implemented by ISI for TOPS-20 hosts and is being planned for DCNET hosts.
In order to evaluate these protocols and assess their
feasibility for use in the DCNET environment, we have constructed protocol modules to support MTP and have tested them with other hosts on the ARPANET, including those at ISI, DCEC and MIT. Although yet to be thoroughly debugged, our intitial experience indicates this to be a practical way to join the ARPANET mail community. We intend this implementation to be an intermediate step; however, and expect to proceed to the IMP and multi-media data types in the future.
5. Data Structures
In the RFC-733 model messages are sent by a user to a
specified recipent in the format <user>@<host>, where <host> is the name of a host and <user> is the name of an user known to that host. The implied address, usually called a mailbox, is typically associated with a mail file belonging to the recipient and is easily generalized to include organizations as well as users and networks as well as hosts. As messages arrive at a host the mail server dispatches them to the various mailboxes by appending them, together with an appropriate header, following the messages already in the mailbox.
Since most of our interactions with internet hosts have
been with TOPS-20 machines, we have tried to maintain a degree of compatability with the mail file formats used by that system. The file is line-structured, with each line terminated by the ASCII sequence <CR><LF>, and contains only ASCII printing characters and format effectors. Messages DCNET Mail Plan PAGE 5
consist of a file header, which contains a character count, followed by the message text. Messages are stored one after the other with the last followed by an ASCII <SUB> character for compatibility with other RT-11 components. Figure 1 shows the format of a typical message.
29-May-81 01:01:14,342;000000000000 MRCP to:<OConnor@ISIE> MRCP to:<@Multics,Mills@ISIE> MRCP to:<Mills@COMSAT> MAIL from:<Mills@COMSAT-DLM> Date: 29-May-81 01:00:42-UT From: Mills@COMSAT-DLM Subject: Mail example To: OConnor@ISIE cc: <@Multics,Mills@ISIE>,Mills@COMSAT
This message demonstrates RFC-733 and RFC-780 formats.
The first line is the file header, including the date,
time and count of characters in the message text, and followed by an array of twelve flag characters used by the MSG program described below. The message text begins immediately following the <CR><LF> which terminates this line and includes first the transport (RFC-780) header followed by the message (RFC-733) header and, finally, the body of the message itself. In the above example the lines beginning with MRCP and MAIL belong to the transport header and the lines following that up until the blank line belong to the message header.
Mail files can be created in three ways: (1) by copying
a mail file from a TOPS-20 or DCNET host as-is to a DCNET host, (2) by creating and appending a new message locally and (3) by receiving and appending a message from another host. Case (1) has been found useful during testing and in cases involving large amounts of mail which can be bulk-transferred using more efficient file-transfer protocols like FTP and NIFTP. However, TOPS-20 files do not include the transport header, so that messages sent from these files require manual intervention. Case (2) is implemented by an interactive program called SNDMSG, which operates much like the TOPS-20 program of the same name to construct and edit messages and append them, along with their transport and message headers, onto a specified mail file. Case (3) is implemented by the MTPSRV server program, which listens for messages from the network and appends them DCNET Mail Plan PAGE 6
onto a well-known mail file. All three cases produce identical file formats, so that a common message display program can be used. This interactive program, called MSG, is the focus of most mail operations and is used in the same fashion as its TOPS-20 counterpart of the same name.
6. Mail Operations
Display and editing of mail file contents is performed
by the MSG program. This program includes the capability to select messages and groups of messages and to display them on the operator's terminal and/or append them to other mail files. Since DCNET files and devices can be accessed in the same way, this amounts to the capability to print them and even to display them on the Dacom facsimile machine or speak them on the LPCM speech decoder (assuming the correct source encoding, of course). When a message is copied to a file its headers are copied as well, so that copying a message to a mail file results in a mail file in correct format.
A message is constructed using the SNDMSG program,
which builds the transport and message headers shown in Figure 1 according to an interactive dialog and then edits the text as specified by the user. Note that, as illustrated in Figure 1, separate MRCP lines are created for each recipient and a MAIL line is created for the sender. These lines are edited by the MTP user program to indicate the delivery status for each recipient. Features in the present SNDMSG implementation provide for the inclusion of data files, such as might be produced by a standard text editor, and address lists.
Mail is sent to recipients at the various internet
hosts by the MTP user program. This program first searches a specified mail file and constructs a data structure including a set of pointers to the MRCP transport-header lines, along with the internet address associated with each recipient host. It then sorts this structure by host and constructs a source-route string, if necessary, as specified by the operator. Finally, it connects to each host in turn and sends its messages in either text-first or recipients-first format, as required by the host. As delivery to each recipient is confirmed, the MTP user program overwrites the corresponding MRCP string with the string SENT. Other strings are possible in case of errors.
It may happen that some messages sent to a host may
specify recipients not at that host, in which case these messages must be forwarded to the final destination as required by RFC-780. This would be the case when an operator at a local host wishes to stage a batch of messages at the mail host for later relay to other hosts not on-line at the moment. In addition, forwarding is also required DCNET Mail Plan PAGE 7
when the final destination host supports some transport protocol other than TCP, so that an intermediary supporting both protocols is required. The present system supports two operational modes, one in which mail is sent automatically either directly to the destination or via an intermediate relay, as directed by internal tables, and the other in which it is sent manually according to a source route specified by the operator.
Mail is ordinarily received automatically at a DCNET
host by the MTPSRV server program. This program appends each message as it is received to a public, controlled-access mail file called UNSENT.MSG. For those messages addressed to a recipient at the receiving host, the corresponding MRCP string is overwritten with the string DLVD; the remainder are left for later relay by the MTP user program.
8. On Hosts, Users and Mailboxes
Upon reflection on the above it may be noted that no
mention is made of the concept of a DCNET mailbox. This is intentional and reflects the fact that each user is identified in fact only by his personal data disk and an informal convention of assigned user names. Matters become considerably more complicated when DCNET virtual hosts are involved, for this mechanism (described in detail elsewhere) provides the capability to associate one or more internet addresses with a single physical host and to move this association from time to time. Thus, the mail host may pop up in various physical hosts depending upon any of several considerations; however, the internet addressing will be transparent to this and the routing will be performed automatically.
For these reasons we have chosen to identify a
particular internet address with the mail host, to be called simply COMSAT and the remaining hosts as COMSAT-xxx where xxx corresponds to the number (or name) of each of the other virtual hosts. Ordinarily, mail for all DCNET users is sent to mailboxes such as <user>@COMSAT. It would then remain at the mail host for later inspection by <user>. If, on the other hand, it is known that <user> is active on some DCNET host at the time the mail is to be sent, then it could be sent directly to that host.
In order to preserve privacy when accessing messages at
the mail host via virtual-terminal connection from a local host, a feature has been incorporated into the MSG program normally used for this purpose. Ordinarily, all messages received at the mail host are saved in a public file called UNSENT.MSG, while the messages belonging to each user are saved in private files. MSG normally operates with a DCNET Mail Plan PAGE 8
private file as specified by the user, with access granted to UNSENT.MSG only by means of a keyword, normally the recipient's name. A special MSG command provides for searching UNSENT.MSG for messages which have been "delivered" (marked DLVD) to the recipient name matching the keyword. These messages can then be appended to the user's private file and forwarded to his local host for further processing or archiving, if required.
9. Internet Name Domains
In the long run, it will not be practicable for every
internet host to include all internet hosts in its name-address tables. Even now, with over four hundred names and nicknames in the combined ARPANET-DCNET tables, this has become awkward. Some sort of hierarchical name-space partitioning can easily be devised to deal with this problem; however, it has been wickedly difficult to find one compatible with the known mail systems throughout the community. The one described here, which has been partially implemented in the DCNET mail system, is the product of several discussions and meetings and is believed both compatible with existing systems and extensible for future systems involving thousands of hosts.
We first observe that every internet host is uniquely
identified by one (or more) 32-bit internet addresses and that the entire system is fully connected. For the moment, the issue of protocol compatibility will be ignored, so that all hosts can be assumed MTP-competent. We next impose a topological covering on the space of all internet addresses with a set of so-called name domains. In the natural model, name domains would correspond to institutions such as ARPA, USC and COMSAT, and would not be necessarily disjoint or complete. While in principle name domains could be hierarchically structured, we will assume in the following only a single-level structure.
Every name domain is associated with one (or more)
internet hosts called mail forwarders and the name of that domain is a name or nickname for any of these hosts. Each mail forwarder is expected to maintain name-address tables containing all other hosts in its domain and, in addition, at least one mail forwarder for every other domain. All hosts other than mail forwarders are expected to maintain name-address tables containing at least one mail forwarder for every domain together with additional hosts as convenient. Following current practice, several nicknames may be associated with the principal name of a host in any domain and the names and nicknames need not be unique in any other domain. Furthermore, hosts can be multi-homed, that is, respond to more than one address. For the purpose of mail forwarding and delivery, we will assume that any of DCNET Mail Plan PAGE 9
these addresses can be used without prejudice.
In its most general form, an internet mailbox name has
where <user> is the name of a user known at the host <host> in the name domain <domain>. This syntax is intended to suggest a three-level hierarchically structured name (reading from the right) which is unique throughout the internet system. However, hosts within a single domain may agree to adopt another structure, as long as it does not conflict with the above syntax and as long as the forwarders for that domain are prepared to make the requisite transformations. For instance, let the name of a domain including DCNET be COMSAT and the name of one of its hosts be COMSAT-DLM with Mills a user known to that host. From within the COMSAT domain the name Mills@COMSAT-DLM uniquely identifies that mailbox as could, for example, the name Mills.DLM@COMSAT from anywhere in the internet system. However, Mills@COMSAT-DLM is not necessarily meaningful anywhere outside the COMSAT domain (but it could be).
The functions required of the forwarder are
straightforward. When it receives a message for <user>.<host>@<domain>, it transforms <host> as necessary and forwards the message to its address found in the name-address table for <domain>. Note that a single host can be a mail forwarder for several independent domains in this model and that these domains can intersect. Thus, the names Mills@USC-ISIE, Mills.USC-ISIE@ARPA and Mills.USC-ISIE@COMSAT can all refer to the same mailbox and can, conceivably, all be entries in the same name-address table of some host. In this example the ARPANET host USC-ISIE appears as a domain with a null host name. Such use would be permissable only in case the name USC-ISIE was unique and known to all forwarders in the internet system.
In order for this scheme to work properly, it is
necessary that messages transiting forwarders always contain complete internet mailbox names. When this is not feasible, as in the current ARPANET mail system, the forwarder must be able to determine which domain the message came from and edit the names accordingly. This would be necessary in order to compose a reply to the message in any case.
10. Current Status and Unresolved Issues
The present system is working with DCNET hosts and
certain other ARPANET hosts mentioned above, although some minor problems remain to be worked out. The experience gained has guided the implementation of features recently DCNET Mail Plan PAGE 10
incorporated into MSG and SNDMSG. Additional features are to be incorporated into MTP and MTPSRV as the result of current discussions and revisions of RFC-780.
There are a number of system-specific issues which need
to be resolved before the DCNET implementation could be considered user-fortified, among them the following:
1. There are no provisions to resolve conflicting accesses
to public files such as UNSENT.MSG which might occur if a message arrives while SNDMSG is active on the same file.
2. There are no provisions to compact UNSENT.MSG
automatically as messages are forwarded or processed by the recipients.
3. The MTP user program must be initiated manually, rather
than automatically as the result of a timeout, for example.
4. Forwarder mailbox name transformations as described
above are not supported.
5. A facility is needed to re-address misrouted messages
and to return failure messages to the sender in case they cannot be forwarded.
This memorandum has described the environment and
features required of the DCNET mail system, as well as an interim plan for compatability with other developing internet mail systems. The interim plan focusses on the Mail Transfer Protocol of RFC-780 and on the transition of the existing DCNET mail system to be compatible with it. We believe this to be a useful and reasonably easy thing to do and that it will both shake the bugs from existing internet mail transport systems as well as smooth the way for the eventual implementation of the Internet Message Protocol and multi-media data types. DCNET Mail Plan PAGE 11
1. Crocker, D., J. Vittal, K. Pogran, and D. Henderson.
Standard for the Format of ARPA Network Text Messages. Request for Comments RFC-733, NIC 41952, November 1977. Also in: Feinler, E. and J. Postel, eds. ARPANET Protocol Handbook. NIC 7104, for the Defense Communications Agency by SRI International, Menlo Park, California, Revised January 1978.
2. Postel, J., ed. File Transfer Protocol. Request for
Comments RFC-765, Internet Experiment Note IEN-149. USC Information Sciences Institute, Marina del Rey, California, 1980.
3. Postel, J., and S. Sluizer. Mail Transfer Protocol.
Request for Comments RFC-780. USC Information Sciences Institute, Marina del Rey, California, May 1981.
4. Postel, J. Internet Message Protocol. Request for
Comments RFC-759, Internet Experiment Note IEN-113. USC Information Sciences Institute, Marina del Rey, California, August 1980.
5. Postel, J., ed. DoD Standard Transmission Control
Protocol. Request for Comments 761, Internet Experiment Note 129, NTIS ADA082609. USC Information Sciences Institute, Marina del Rey, California, January 1980. Also in: ACM Computer Communication Review 10, 4 (October 1980).
6. PSS/SG3. A Network Independent Transport Service.
Study Group 3, The Post Office PSS Users Group, February 1980. Available from: DCPU, National Physical Laboratory, Teddington, UK.