HtmlPrag: Pragmatic Parsing and Emitting of HTML using SXML and SHTML

License: LGPL 3   Web: http://www.neilvandyke.org/htmlprag/

(require (planet neil/htmlprag:1:=7))

1 Introduction

Note: This package has been obsoleted by newer SXML/xexp tools of the author.

HtmlPrag provides permissive HTML parsing and emitting capability to Scheme programs. The parser is useful for software agent extraction of information from Web pages, for programmatically transforming HTML files, and for implementing interactive Web browsers. HtmlPrag emits “SHTML,” which is an encoding of HTML in SXML, so that conventional HTML may be processed with XML tools such as SXPath. Like Oleg Kiselyov’s SSAX-based HTML parser, HtmlPrag provides a permissive tokenizer, but also attempts to recover structure. HtmlPrag also includes procedures for encoding SHTML in HTML syntax.

The HtmlPrag parsing behavior is permissive in that it accepts erroneous HTML, handling several classes of HTML syntax errors gracefully, without yielding a parse error. This is crucial for parsing arbitrary real-world Web pages, since many pages actually contain syntax errors that would defeat a strict or validating parser. HtmlPrag’s handling of errors is intended to generally emulate popular Web browsers’ interpretation of the structure of erroneous HTML. We euphemistically term this kind of parse “pragmatic.”

HtmlPrag also has some support for XHTML, although XML namespace qualifiers are currently accepted but stripped from the resulting SHTML. Note that valid XHTML input is of course better handled by a strict XML parser.

HtmlPrag requires R5RS, SRFI-6, and SRFI-23. This version of HtmlPrag is specific to PLT Scheme, due to a transition period in how portability is handled, but the exceedingly portable version 0.16 is available at: http://www.neilvandyke.org/htmlprag/htmlprag-0-16.scm

2 SHTML and SXML

SHTML is a variant of SXML, with two minor but useful extensions:

• The SXML keyword symbols, such as *TOP*, are defined to be in all uppercase, regardless of the case-sensitivity of the reader of the hosting Scheme implementation in any context. This avoids several pitfalls.

• Since not all character entity references used in HTML can be converted to Scheme characters in all R5RS Scheme implementations, nor represented in conventional text files or other common external text formats to which one might wish to write SHTML, SHTML adds a special & syntax for non-ASCII (or non-Extended-ASCII) characters. The syntax is (& val), where val is a symbol or string naming with the symbolic name of the character, or an integer with the numeric value of the character.

shtml-comment-symbol : any/c

shtml-decl-symbol : any/c

shtml-empty-symbol : any/c

shtml-end-symbol : any/c

shtml-entity-symbol : any/c

shtml-pi-symbol : any/c

shtml-start-symbol : any/c

shtml-text-symbol : any/c

shtml-top-symbol : any/c

These variables are bound to the following case-sensitive symbols used in SHTML, respectively: *COMMENT*, *DECL*, *EMPTY*, *END*, *ENTITY*, *PI*, *START*, *TEXT*, and *TOP*. These can be used in lieu of the literal symbols in programs read by a case-insensitive Scheme reader.

shtml-named-char-id : any/c

shtml-numeric-char-id : any/c

These variables are bound to the SHTML entity public identifier strings used in SHTML *ENTITY* named and numeric character entity references.

(make-shtml-entity val) → any/c

val : any/c

Yields an SHTML character entity reference for val. For example:

(make-shtml-entity "rArr")                  ==> (& rArr)

(make-shtml-entity (string->symbol "rArr")) ==> (& rArr)

(make-shtml-entity 151)                     ==> (& 151)

(shtml-entity-value obj) → any/c

obj : any/c

Yields the value for the SHTML entity obj, or #f if obj is not a recognized entity. Values of named entities are symbols, and values of numeric entities are numbers. An error may raised if obj is an entity with system ID inconsistent with its public ID. For example:

(define (f s) (shtml-entity-value (cadr (html->shtml s))))

(f " ")  ==> nbsp

(f "ߐ") ==> 2000

3 Tokenizing

The tokenizer is used by the higher-level structural parser, but can also be called directly for debugging purposes or unusual applications. Some of the list structure of tokens, such as for start tag tokens, is mutated and incorporated into the SHTML list structure emitted by the parser.

(make-html-tokenizer in normalized?) → any/c

in : any/c

normalized? : any/c

Constructs an HTML tokenizer procedure on input port in. If boolean normalized? is true, then tokens will be in a format conducive to use with a parser emitting normalized SXML. Each call to the resulting procedure yields a successive token from the input. When the tokens have been exhausted, the procedure returns the null list. For example:

(define input (open-input-string "<a href=\"foo\">bar</a>"))

(define next  (make-html-tokenizer input #f))

(next) ==> (a (@ (href "foo")))

(next) ==> "bar"

(next) ==> (*END* a)

(next) ==> ()

(next) ==> ()

(tokenize-html in normalized?) → any/c

in : any/c

normalized? : any/c

Returns a list of tokens from input port in, normalizing according to boolean normalized?. This is probably most useful as a debugging convenience. For example:

(tokenize-html (open-input-string "<a href=\"foo\">bar</a>") #f)

==> ((a (@ (href "foo"))) "bar" (*END* a))

(shtml-token-kind token) → any/c

token : any/c

Returns a symbol indicating the kind of tokenizer token: *COMMENT*, *DECL*, *EMPTY*, *END*, *ENTITY*, *PI*, *START*, *TEXT*. This is used by higher-level parsing code. For example:

(map shtml-token-kind

(tokenize-html (open-input-string "<a<b>><c</</c") #f))

==> (*START* *START* *TEXT* *START* *END* *END*)

4 Parsing

Most applications will call a parser procedure such as html->shtml rather than calling the tokenizer directly.

(parse-html/tokenizer   tokenizer             normalized?)   →   any/c

tokenizer : any/c

normalized? : any/c

Emits a parse tree like html->shtml and related procedures, except using tokenizer as a source of tokens, rather than tokenizing from an input port. This procedure is used internally, and generally should not be called directly.

(html->sxml-0nf input) → any/c

input : any/c

(html->sxml-1nf input) → any/c

input : any/c

(html->sxml-2nf input) → any/c

input : any/c

(html->sxml input) → any/c

input : any/c

(html->shtml input) → any/c

input : any/c

Permissively parse HTML from input, which is either an input port or a string, and emit an SHTML equivalent or approximation. To borrow and slightly modify an example from Kiselyov’s discussion of his HTML parser:

(html->shtml

"<html><head><title></title><title>whatever</title></head><body>\n<a href=\"url\">link</a><p align=center><ul compact style=\"aa\">\n<p>BLah<!-- comment <comment> --> <i> italic <b> bold <tt> ened</i>\nstill &lt; bold </b></body><P> But not done yet...")

==>

(*TOP* (html (head (title) (title "whatever"))

(body "\n"

(a (@ (href "url")) "link")

(p (@ (align "center"))

(ul (@ (compact) (style "aa")) "\n"))

(p "BLah"

(*COMMENT* " comment <comment> ")

" "

(i " italic " (b " bold " (tt " ened")))

"\n"

"still < bold "))

(p " But not done yet...")))

Note that in the emitted SHTML the text token "still < bold" is not inside the b element, which represents an unfortunate failure to emulate all the quirks-handling behavior of some popular Web browsers.

The procedures html->sxml-nnf for n 0 through 2 correspond to 0th through 2nd normal forms of SXML as specified in SXML, and indicate the minimal requirements of the emitted SXML.

html->sxml and html->shtml are currently aliases for html->sxml-0nf, and can be used in scripts and interactively, when terseness is important and any normal form of SXML would suffice.

5 Emitting HTML

Two procedures encoding the SHTML representation as conventional HTML, write-shtml-as-html and shtml->html. These are perhaps most useful for emitting the result of parsed and transformed input HTML. They can also be used for emitting HTML from generated or handwritten SHTML.

(write-shtml-as-html   shtml             out             foreign-filter)   →   any/c

shtml : any/c

out : any/c

foreign-filter : any/c

Writes a conventional HTML transliteration of the SHTML shtml to output port out. If out is not specified, the default is the current output port. HTML elements of types that are always empty are written using HTML4-compatible XHTML tag syntax.

If foreign-filter is specified, it is a procedure of two argument that is applied to any non-SHTML (“foreign”) object encountered in shtml, and should yield SHTML. The first argument is the object, and the second argument is a boolean for whether or not the object is part of an attribute value.

No inter-tag whitespace or line breaks not explicit in shtml is emitted. The shtml should normally include a newline at the end of the document. For example:

(write-shtml-as-html

'((html (head (title "My Title"))

(body (@ (bgcolor "white"))

(h1 "My Heading")

(p "This is a paragraph.")

(p "This is another paragraph.")))))

outputs:

<html><head><title>My Title</title></head><body bgcolor="whi

te"><h1>My Heading</h1><p>This is a paragraph.</p><p>This is

another paragraph.</p></body></html>

(shtml->html shtml) → any/c

shtml : any/c

Yields an HTML encoding of SHTML shtml as a string. For example:

(shtml->html

(html->shtml

"<P>This is<br<b<I>bold </foo>italic</ b > text.</p>"))

==> "<p>This is<br /><b><i>bold italic</i></b> text.</p>"

Note that, since this procedure constructs a string, it should normally only be used when the HTML is relatively small. When encoding HTML documents of conventional size and larger, write-shtml-as-html is much more efficient.

6 History

• Version 0.20 — 2011-08-22 — PLaneT (1 7)

  Document that HtmlPrag has been obsoleted.

• Version 0.19 — 2009-11-08 — PLaneT (1 6)

  Whitespace after a < in a context that would otherwise start a tag is no longer considered the start of a tag. This behavior is consistent with, e.g., Firefox 3, and we have found a major site relying on it. Three regression tests were changed to match the new desired behavior.

• Version 0.18 — 2009-11-07 — PLaneT (1 5)

  The p element can be a child of the li element.

• Version 0.17 — 2009-08-16 — PLaneT (1 4)

  License is now LGPL3. Converted to author’s new Scheme management system. Revamped high-level parser to not use mutable pairs, for PLT Scheme 4.x compatibility. Until the new portability mechanism is in place, the previous portable version of HtmlPrag is available at: http://www.neilvandyke.org/htmlprag/htmlprag-0-16.scm

• Version 0.16 — 2005-12-18

  Documentation fix.

• Version 0.15 — 2005-12-18

  In the HTML parent element constraints that are used for structure recovery, div is now always permitted as a parent, as a stopgap measure until substantial time can be spent reworking the algorithm to better support div (bug reported by Corey Sweeney and Jepri). Also no longer convert to Scheme character any HTML numeric character reference with value above 126, to avoid Unicode problem with PLT 299/300 (bug reported by Corey Sweeney).

• Version 0.14 — 2005-06-16

  XML CDATA sections are now tokenized. Thanks to Alejandro Forero Cuervo for suggesting this feature. The deprecated procedures sxml->html and write-sxml-html have been removed. Minor documentation changes.

• Version 0.13 — 2005-02-23

  HtmlPrag now requires syntax-rules, and a reader that can read the at-sign character as a symbol. SHTML now has a special & element for character entities, and it is emitted by the parser rather than the old *ENTITY* kludge. shtml-entity-value supports both the new and the old character entity representations. shtml-entity-value now yields #f on invalid SHTML entity, rather than raising an error. write-shtml-as-html now has a third argument, foreign-filter. write-shtml-as-html now emits SHTML & entity references. Changed shtml-named-char-id and shtml-numeric-char-id, as previously warned. Testeez is now used for the test suite. Test procedure is now the internal %htmlprag:test. Documentation changes. Notably, much documentation about using HtmlPrag under various particular Scheme implementations has been removed.

• Version 0.12 — 2004-07-12

  Forward-slash in an unquoted attribute value is now considered a value constituent rather than an unconsumed terminator of the value (thanks to Maurice Davis for reporting and a suggested fix). xml: is now preserved as a namespace qualifier (thanks to Peter Barabas for reporting). Output port term of write-shtml-as-html is now optional. Began documenting loading for particular implementation-specific packagings.

• Version 0.11 — 2004-05-13

  To reduce likely namespace collisions with SXML tools, and in anticipation of a forthcoming set of new features, introduced the concept of “SHTML,” which will be elaborated upon in a future version of HtmlPrag. Renamed sxml-x-symbol to shtml-x-symbol, sxml-html-x to shtml-x, and sxml-token-kind to shtml-token-kind. html->shtml, shtml->html, and write-shtml-as-html have been added as names. Considered deprecated but still defined (see the “Deprecated” section of this documentation) are sxml->html and write-sxml-html. The growing pains should now be all but over. Internally, htmlprag-internal:error introduced for Bigloo portability. SISC returned to the test list; thanks to Scott G. Miller for his help. Fixed a new character eq? bug, thanks to SISC.

• Version 0.10 — 2004-05-11

  All public identifiers have been renamed to drop the “htmlprag:” prefix. The portability identifiers have been renamed to begin with an htmlprag-internal: prefix, are now considered strictly internal-use-only, and have otherwise been changed. parse-html and always-empty-html-elements are no longer public. test-htmlprag now tests html->sxml rather than parse-html. SISC temporarily removed from the test list, until an open source Java that works correctly is found.

• Version 0.9 — 2004-05-07

  HTML encoding procedures added. Added htmlprag:sxml-html-entity-value. Upper-case X in hexadecimal character entities is now parsed, in addition to lower-case x. Added htmlprag:always-empty-html-elements. Added additional portability bindings. Added more test cases.

• Version 0.8 — 2004-04-27

  Entity references (symbolic, decimal numeric, hexadecimal numeric) are now parsed into *ENTITY* SXML. SXML symbols like *TOP* are now always upper-case, regardless of the Scheme implementation. Identifiers such as htmlprag:sxml-top-symbol are bound to the upper-case symbols. Procedures htmlprag:html->sxml-0nf, htmlprag:html->sxml-1nf, and htmlprag:html->sxml-2nf have been added. htmlprag:html->sxml now an alias for htmlprag:html->sxml-0nf. htmlprag:parse has been refashioned as htmlprag:parse-html and should no longer be directly. A number of identifiers have been renamed to be more appropriate when the htmlprag: prefix is dropped in some implementation-specific packagings of HtmlPrag: htmlprag:make-tokenizer to htmlprag:make-html-tokenizer, htmlprag:parse/tokenizer to htmlprag:parse-html/tokenizer, htmlprag:html->token-list to htmlprag:tokenize-html, htmlprag:token-kind to htmlprag:sxml-token-kind, and htmlprag:test to htmlprag:test-htmlprag. Verbatim elements with empty-element tag syntax are handled correctly. New versions of Bigloo and RScheme tested.

• Version 0.7 — 2004-03-10

  Verbatim pair elements like script and xmp are now parsed correctly. Two Scheme implementations have temporarily been dropped from regression testing: Kawa, due to a Java bytecode verifier error likely due to a Java installation problem on the test machine; and SXM 1.1, due to hitting a limit on the number of literals late in the test suite code. Tested newer versions of Bigloo, Chicken, Gauche, Guile, MIT Scheme, PLT MzScheme, RScheme, SISC, and STklos. RScheme no longer requires the “(define get-output-string close-output-port)” workaround.

• Version 0.6 — 2003-07-03

  Fixed uses of eq? in character comparisons, thanks to Scott G. Miller. Added htmlprag:html->normalized-sxml and htmlprag:html->nonnormalized-sxml. Started to add close-output-port to uses of output strings, then reverted due to bug in one of the supported dialects. Tested newer versions of Bigloo, Gauche, PLT MzScheme, RScheme.

• Version 0.5 — 2003-02-26

  Removed uses of call-with-values. Re-ordered top-level definitions, for portability. Now tests under Kawa 1.6.99, RScheme 0.7.3.2, Scheme 48 0.57, SISC 1.7.4, STklos 0.54, and SXM 1.1.

• Version 0.4 — 2003-02-19

  Apostrophe-quoted element attribute values are now handled. A bug that incorrectly assumed left-to-right term evaluation order has been fixed (thanks to MIT Scheme for confronting us with this). Now also tests OK under Gauche 0.6.6 and MIT Scheme 7.7.1. Portability improvement for implementations (e.g., RScheme 0.7.3.2.b6, Stalin 0.9) that cannot read the at-sign character as a symbol (although those implementations tend to present other portability issues, as yet unresolved).

• Version 0.3 — 2003-02-05

  A test suite with 66 cases has been added, and necessary changes have been made for the suite to pass on five popular Scheme implementations. XML processing instructions are now parsed. Parent constraints have been added for colgroup, tbody, and thead elements. Erroneous input, including invalid hexadecimal entity reference syntax and extraneous double quotes in element tags, is now parsed better. htmlprag:token-kind emits symbols more consistent with SXML.

• Version 0.2 — 2003-02-02

  Portability improvements.

• Version 0.1 — 2003-01-31

  Dusted off author’s old Guile-specific code from April 2001, converted to emit SXML, mostly ported to R5RS and SRFI-6, added some XHTML support and documentation. A little preliminary testing has been done, and the package is already useful for some applications, but this release should be considered a preview to invite comments.

7 Legal

Copyright (c) 2003 – 2011 Neil Van Dyke. This program is Software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License (LGPL 3), or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See http://www.gnu.org/licenses/ for details. For other licenses and consulting, please contact the author.

Standard Documentation Format Note: The API signatures in this documentation are likely incorrect in some regards, such as indicating type any/c for things that are not, and not indicating when arguments are optional. This is due to a transitioning from the Texinfo documentation format to Scribble, which the author intends to finish someday.