Go to the first, previous, next, last section, table of contents.


The Mimestring module

Collection of auxiliary functions to parse MIME headers

val scan_header : 
       ?unfold:bool ->
       string -> start_pos:int -> end_pos:int -> 
         ((string * string) list * int)

let params, i2 = scan_header s i0 i1:

DESCRIPTION

Scans the MIME header that begins at position i0 in the string s and that must end somewhere before position i1. It is intended that in i1 the character position following the end of the body of the MIME message is passed. Returns the parameters of the header as (name,value) pairs (in params), and in i2 the position of the character following directly after the header (i.e. after the blank line separating the header from the body). The following normalizations have already been applied: - The names are all in lowercase - Newline characters (CR and LF) have been removed (unless ?unfold:false has been passed) - Whitespace at the beginning and at the end of values has been removed (unless ?unfold:false is specified) The rules of RFC 2047 have NOT been applied. The function fails if the header violates the header format strongly. (Some minor deviations are tolerated, e.g. it is sufficient to separate lines by only LF instead of CRLF.)

OPTIONS:

unfold: If true (the default), folded lines are concatenated and returned as one line. This means that CR and LF characters are deleted and that whitespace at the beginning and the end of the string is removed. You may set ?unfold:false to locate individual characters in the parameter value exactly.

ABOUT MIME MESSAGE FORMAT:

This is the modern name for messages in "E-Mail format". Messages consist of a header and a body; the first empty line separates both parts. The header contains lines "param-name: param-value" where the param-name must begin on column 0 of the line, and the ":" separates the name and the value. So the format is roughly:

param1-name: param1-value ... paramN-name: paramN-value

body

This function wants in i0 the position of the first character of param1-name in the string, and in i1 the position of the character following the body. It returns as i2 the position where the body begins. Furthermore, in 'params' all parameters are returned that exist in the header.

DETAILS

Note that parameter values are restricted; you cannot represent arbitrary strings. The following problems can arise: - Values cannot begin with whitespace characters, because there may be an arbitrary number of whitespaces between the ':' and the value. - Values (and names of parameters, too) must only be formed of 7 bit ASCII characters. (If this is not enough, the MIME standard knows the extension RFC 2047 that allows that header values may be composed of arbitrary characters of arbitrary character sets.) - Header values may be broken into several lines, the continuation lines must begin with whitespace characters. This means that values must not contain line breaks as semantical part of the value. And it may mean that ONE whitespace character is not distinguishable from SEVERAL whitespace characters. - Header lines must not be longer than 76 characters. Values that would result into longer lines must be broken into several lines. This means that you cannot represent strings that contain too few whitespace characters. - Some gateways pad the lines with spaces at the end of the lines.

This implementation of a MIME scanner tolerates a number of deviations from the standard: long lines are not rejected; 8 bit values are accepted; lines may be ended only with LF instead of CRLF. Furthermore, header values are transformed: - leading and trailing spaces are always removed - CRs and LFs are deleted; it is guaranteed that there is at least one space or tab where CR/LFs are deleted. Last but not least, the names of the header values are converted to lowercase; MIME specifies that they are case-independent.

COMPATIBILITY WITH THE STANDARD

This function can parse all MIME headers that conform to RFC 822. But there may be still problems, as RFC 822 allows some crazy representations that are actually not used in practice. In particular, RFC 822 allows it to use backslashes to "indicate" that a CRLF sequence is semantically meant as line break. As this function normally deletes CRLFs, it is not possible to recognize such indicators in the result of the function.

The following types and functions allow it to build scanners for structured MIME values in a highly configurable way.

WHAT ARE STRUCTURED VALUES?

RFC 822 (together with some other RFCs) defines lexical rules how formal MIME header values should be divided up into tokens. Formal MIME headers are those headers that are formed according to some grammar, e.g. mail addresses or MIME types. Some of the characters separate phrases of the value; these are the "special" characters. For example, '' is normally a special character for mail addresses, because it separates the user name from the domain name. RFC 822 defines a fixed set of special characters, but other RFCs use different sets. Because of this, the following functions allow it to configure the set of special characters. Every sequence of characters may be embraced by double quotes, which means that the sequence is meant as literal data item; special characters are not recognized inside a quoted string. You may use the backslash to insert any character (including double quotes) verbatim into the quoted string (e.g. "He said: \"Give it to me!\""). The sequence of a backslash character and another character is called a quoted pair. Structured values may contain comments. The beginning of a comment is indicated by '(', and the end by ')'. Comments may be nested. Comments may contain quoted pairs. A comment counts as if a space character were written instead of it. Control characters are the ASCII characters 0 to 31, and 127. RFC 822 demands that MIME headers are 7 bit ASCII strings. Because of this, this function also counts the characters 128 to 255 as control characters. Domain literals are strings embraced by '\[' and '\]'; such literals may contain quoted pairs. Today, domain literals are used to specify IP addresses. Every character sequence not falling in one of the above categories is an atom (a sequence of non-special and non-control characters). When recognized, atoms may be encoded in a character set different than US-ASCII; such atoms are called encoded words (see RFC 2047).

EXTENDED INTERFACE:

In order to scan a string containing a MIME value, you must first create a mime_scanner using the function create_mime_scanner. The scanner contains the reference to the scanned string, and a specification how the string is to be scanned. The specification consists of the lists 'specials' and 'scan_options'.

The character list 'specials' specifies the set of special characters. These characters are returned as Special c token; the following additional rules apply:

- Spaces: If ' ' in specials: A space character is returned as Special ' '. Note that there may also be an effect on how comments are returned (see below). If ' ' not in specials: Spaces are ignored.

- Tabs, CRs, LFs: If '\t' in specials: A tab character is returned as Special '\t'. If '\t' not in specials: Tabs are ignored.

If '\r' in specials: A CR character is returned as Special '\r'. If '\r' not in specials: CRs are ignored.

If '\n' in specials: A LF character is returned as Special '\n'. If '\n' not in specials: LFs are ignored.

- Comments: If '(' in specials: Comments are not recognized. The character '(' is returned as Special '('. If '(' not in specials: Comments are recognized. How comments are returned, depends on the following: If Return_comments in scan_options: Outer comments are returned as Comment (note that inner comments count but are not returned as tokens) If otherwise ' ' in specials: Outer comments are returned as Special ' ' Otherwise: Comments are recognized but ignored.

- Quoted strings: If '"' in specials: Quoted strings are not recognized, and double quotes are returned as Special '"'. If '"' not in specials: Quoted strings are returned as QString tokens.

- Domain literals: If '\[' in specials: Domain literals are not recognized, and left brackets are returned as Special '\['. If '\[' not in specials: Domain literals are returned as DomainLiteral tokens.

Note that the rule for domain literals is completely new in netstring-0.9. It may cause incompatibilities with previous versions if '\[' is not special.

The general rule for special characters: Every special character c is returned as Special c, and any additional scanning functionality for this character is turned off.

If recognized, quoted strings are returned as QString s, where s is the string without the embracing quotes, and with already decoded quoted pairs.

Control characters c are returned as Control c.

If recognized, comments may either be returned as spaces (in the case you are not interested in the contents of comments), or as Comment tokens. The contents of comments are not further scanned; you must start a subscanner to analyze comments as structured values.

If recognized, domain literals are returned as DomainLiteral s, where s is the literal without brackets, and with decoded quoted pairs.

Atoms are returned as Atom s where s is a longest sequence of atomic characters (all characters which are neither special nor control characters nor delimiters for substructures). If the option Recognize_encoded_words is on, atoms which look like encoded words are returned as EncodedWord tokens. (Important note: Neither '?' nor '=' must be special in order to enable this functionality.)

After the mime_scanner has been created, you can scan the tokens by invoking scan_token which returns one token at a time, or by invoking scan_token_list which returns all following tokens.

There are two token types: s_token is the base type and is intended to be used for pattern matching. s_extended_token is a wrapper that additionally contains information where the token occurs.

SIMPLE INTERFACE

Instead of creating a mime_scanner and calling the scan functions, you may also invoke scan_structured_value. This function returns the list of tokens directly; however, it is restricted to s_token.

EXAMPLES


scan_structured_value "user@domain.com" [ '@'; '.' ] []
= [ Atom "user"; Special '@'; Atom "domain"; Special '.'; Atom "com" ]

scan_structured_value "user @ domain . com" [ '@'; '.' ] []
= [ Atom "user"; Special '@'; Atom "domain"; Special '.'; Atom "com" ]

scan_structured_value "user(Do you know him?)@domain.com" [ '@'; '.' ] []
= [ Atom "user"; Special '@'; Atom "domain"; Special '.'; Atom "com" ]

scan_structured_value "user(Do you know him?)@domain.com" [ '@'; '.' ] 
[ Return_comments ]
= [ Atom "user"; Comment; Special '@'; Atom "domain"; Special '.'; 
Atom "com" ]

scan_structured_value "user (Do you know him?) @ domain . com" 
[ '@'; '.'; ' ' ] []
= [ Atom "user"; Special ' '; Special ' '; Special ' '; Special '@'; 
Special ' '; Atom "domain";
Special ' '; Special '.'; Special ' '; Atom "com" ]

scan_structured_value "user (Do you know him?) @ domain . com" 
[ '@'; '.'; ' ' ] [ Return_comments ]
= [ Atom "user"; Special ' '; Comment; Special ' '; Special '@'; 
Special ' '; Atom "domain";
Special ' '; Special '.'; Special ' '; Atom "com" ]

scan_structured_value "user @ domain . com" [ '@'; '.'; ' ' ] []
= [ Atom "user"; Special ' '; Special '@'; Special ' '; Atom "domain";
Special ' '; Special '.'; Special ' '; Atom "com" ]

scan_structured_value "user(Do you know him?)@domain.com" ['@'; '.'; '(']
[]
= [ Atom "user"; Special '('; Atom "Do"; Atom "you"; Atom "know";
Atom "him?)"; Special '@'; Atom "domain"; Special '.'; Atom "com" ]

scan_structured_value "\"My.name\"@domain.com" [ '@'; '.' ] []
= [ QString "My.name"; Special '@'; Atom "domain"; Special '.';
Atom "com" ]

scan_structured_value "=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?=" 
[ ] [ ] 
= [ Atom "=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?=" ]

scan_structured_value "=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?=" 
[ ] [ Recognize_encoded_words ] 
= [ EncodedWord("ISO-8859-1", "Q", "Keld_J=F8rn_Simonsen") ]
type s_token =
    Atom of string
  | EncodedWord of (string * string * string)
  | QString of string
  | Control of char
  | Special of char
  | DomainLiteral of string
  | Comment
  | End

- Words are: Atom, EncodedWord, QString. - Atom s: The character sequence forming the atom is contained in s - EncodedWord(charset, encoding, encoded_string) means: charset is the (uppercase) character set encoding is either "Q" or "B" encoded_string: contains the text of the word; the text is represented as octet string following the conventions for character set charset and then encoded either as "Q" or "B" string. - QString s: Here, s are the characters inside the double quotes after decoding any quoted pairs (backslash + character pairs) - Control c: The control character c - Special c: The special character c - DomainLiteral s: s contains the characters inside the brackets after decoding any quoted pairs - Comment: if the option Return_comments is specified, this token represents the whole comment. - End: Is returned after the last token

type s_option =
    No_backslash_escaping

Do not handle backslashes in quoted string and comments as escape characters; backslashes are handled as normal characters. For example: "C:\dir\file" will be returned as QString "C:\dir\file", and not as QString "C:dirfile". - This is a common error in many MIME implementations.

  | Return_comments

Comments are returned as token Comment (unless '(' is included in the list of special characters, in which case comments are not recognized at all). You may get the exact location of the comment by applying get_pos and get_length to the extended token.

  | Recognize_encoded_words

Enables that encoded words are recognized and returned as EncodedWord(charset,encoding,content) instead of Atom.

type s_extended_token

An opaque type containing s_token plus: - where the token occurs - RFC-2047 access functions

val get_token : s_extended_token -> s_token

Return the s_token within the s_extended_token

val get_decoded_word : s_extended_token -> string

val get_charset : s_extended_token -> string

Return the decoded word (the contents of the word after decoding the "Q" or "B" representation), and the character set of the decoded word (uppercase). These functions not only work for EncodedWord: - Atom: Returns the atom without decoding it - QString: Returns the characters inside the double quotes, and decodes any quoted pairs (backslash + character) - Control: Returns the one-character string - Special: Returns the one-character string - DomainLiteral: Returns the characters inside the brackets, and decodes any quoted pairs - Comment: Returns "" The character set is "US-ASCII" for these tokens.

val get_pos : s_extended_token -> int

Return the byte position where the token starts in the string (the first byte has position 0)

val get_line : s_extended_token -> int

Return the line number where the token starts (numbering begins usually with 1)

val get_column : s_extended_token -> int

Return the column of the line where the token starts (first column is number 0)

val get_length : s_extended_token -> int

Return the length of the token in bytes

val separates_adjacent_encoded_words : s_extended_token -> bool

True iff the current token is white space (Special ' ', Special '\t', Special '\r' or Special '\n') and the last non-white space token was EncodedWord and the next non-white space token will be EncodedWord. Such spaces do not count and must be ignored by any application.

type mime_scanner

val create_mime_scanner : 
      specials:char list -> 
      scan_options:s_option list -> 
      ?pos:int ->
      ?line:int ->
      ?column:int ->
      string -> 
        mime_scanner

Creates a new mime_scanner scanning the passed string. specials: The list of characters recognized as special characters. scan_options: The list of global options modifying the behaviour of the scanner pos: The position of the byte where the scanner starts in the passed string. Defaults to 0. line: The line number of this byte. Defaults to 1. column: The column number of this byte. Default to 0.

The optional parameters pos, line, column are intentionally after scan_options and before the string argument, so you can specify scanners by partially applying arguments to create_mime_scanner which are not yet connected with a particular string: let my_scanner_spec = create_mime_scanner my_specials my_options in ... let my_scanner = my_scanner_spec my_string in ...

val get_pos_of_scanner : mime_scanner -> int

val get_line_of_scanner : mime_scanner -> int

val get_column_of_scanner : mime_scanner -> int

Return the current position, line, and column of a mime_scanner. The primary purpose of these functions is to simplify switching from one mime_scanner to another within a string:

let scanner1 = create_mime_scanner ... s in ... now scanning some tokens from s using scanner1 ... let scanner2 = create_mime_scanner ... ?pos:(get_pos_of_scanner scanner1) ?line:(get_line_of_scanner scanner1) ?column:(get_column_of_scanner scanner1) s in ... scanning more tokens from s using scanner2 ...

RESTRICTION: These functions are not available if the option Recognize_encoded_words is on. The reason is that this option enables look-ahead scanning; please use the location of the last scanned token instead. It is currently not clear whether a better implementation is needed (costs a bit more time).

Note: To improve the performance of switching, it is recommended to create scanner specs in advance (see the example my_scanner_spec above).

val scan_token : mime_scanner -> (s_extended_token * s_token)

Returns the next token, or End if there is no more token.

val scan_token_list : mime_scanner -> (s_extended_token * s_token) list

Returns all following tokens as a list (excluding End)

val scan_structured_value : string -> char list -> s_option list -> s_token list

This function is included for backwards compatibility, and for all cases not requiring extended tokens.

It scans the passed string according to the list of special characters and the list of options, and returns the list of all tokens.

val specials_rfc822 : char list

val specials_rfc2045 : char list

The sets of special characters defined by the RFCs 822 and 2045.

CHANGE in netstring-0.9: '' and '' are no longer special because there is now support for domain literals. '?' and '=' are not special in the rfc2045 version because there is already support for encoded words.

Widely used scanners:

val scan_encoded_text_value : string -> s_extended_token list

Scans a "text" value. The returned token list contains only Special, Atom and EncodedWord tokens. Spaces, TABs, CRs, LFs are returned unless they occur between adjacent encoded words in which case they are ignored.

val scan_value_with_parameters : string -> s_option list ->
                                   (string * (string * string) list)

let name, params = scan_value_with_parameters s options: Scans phrases like name ; p1=v1 ; p2=v2 ; ... The scan is done with the set of special characters ';', '='.

val scan_mime_type : string -> s_option list ->
                       (string * (string * string) list)

let name, params = scan_mime_type s options: Scans MIME types like text/plain; charset=iso-8859-1 The name of the type and the names of the parameters are converted to lower case.

Scanners for MIME bodies

val scan_multipart_body : string -> start_pos:int -> end_pos:int -> 
                            boundary:string ->
                            ((string * string) list * string) list

let params1, value1; params2, value2; ... = scan_multipart_body s i0 i1 b

Scans the string s that is the body of a multipart message. The multipart message begins at position i0 in s and i1 the position of the character following the message. In b the boundary string must be passed (this is the "boundary" parameter of the multipart MIME type, e.g. multipart/mixed;boundary="some string" ). The return value is the list of the parts, where each part is returned as pair (params, value). The left component params is the list of name/value pairs of the header of the part. The right component is the RAW content of the part, i.e. if the part is encoded ("content-transfer-encoding"), the content is returned in the encoded representation. The caller must himself decode the content. The material before the first boundary and after the last boundary is not returned.

MULTIPART MESSAGES

The MIME standard defines a way to group several message parts to a larger message (for E-Mails this technique is known as "attaching" files to messages); these are the so-called multipart messages. Such messages are recognized by the major type string "multipart", e.g. multipart/mixed or multipart/form-data. Multipart types MUST have a boundary parameter because boundaries are essential for the representation. Multipart messages have a format like

...Header... Content-type: multipart/xyz; boundary="abc" ...Header...

Body begins here ("prologue") --abc ...Header part 1...

...Body part 1... --abc ...Header part 2...

...Body part 2 --abc ... --abc-- Epilogue

The parts are separated by boundary lines which begin with "--" and the string passed as boundary parameter. (Note that there may follow arbitrary text on boundary lines after "--abc".) The boundary is chosen such that it does not occur as prefix of any line of the inner parts of the message. The parts are again MIME messages, with header and body. Note that it is explicitely allowed that the parts are even multipart messages. The texts before the first boundary and after the last boundary are ignored. Note that multipart messages as a whole MUST NOT be encoded. Only the PARTS of the messages may be encoded (if they are not multipart messages themselves).

Please read RFC 2046 if want to know the gory details of this brain-dead format.

val scan_multipart_body_and_decode : string -> start_pos:int -> end_pos:int -> 
                                        boundary:string ->
                                        ((string * string) list * string) list

Same as scan_multipart_body, but decodes the bodies of the parts if they are encoded using the methods "base64" or "quoted printable". Fails, if an unknown encoding is used.

val scan_multipart_body_from_netstream
    : Netstream.t ->
      boundary:string ->
      create:((string * string) list -> 'a) ->
      add:('a -> Netstream.t -> int -> int -> unit) ->
      stop:('a -> unit) ->
      unit

scan_multipart_body_from_netstream s b create add stop:

Reads the MIME message from the netstream s block by block. The parts are delimited by the boundary b.

Once a new part is detected and begins, the function 'create' is called with the MIME header as argument. The result p of this function may be of any type.

For every chunk of the part that is being read, the function 'add' is invoked: add p s k n. Here, p is the value returned by the 'create' invocation for the current part. s is the netstream. The current window of s contains the read chunk completely; the chunk begins at position k of the window (relative to the beginning of the window) and has a length of n bytes.

When the part has been fully read, the function 'stop' is called with p as argument.

That means, for every part the following is executed: - let p = create h - add p s k1 n1 - add p s k2 n2 - ... - add p s kN nN - stop p

IMPORTANT PRECONDITION: - The block size of the netstream s must be at least String.length b + 3

EXCEPTIONS: - Exceptions can happen because of ill-formed input, and within the callbacks of the functions 'create', 'add', 'stop'. - If the exception happens while part p is being read, and the 'create' function has already been called (successfully), the 'stop' function is also called (you have the chance to close files).

THREAD-SAFETY: The functions are thread-safe as long as the threads do not share values.


Go to the first, previous, next, last section, table of contents.