#lang scribble/doc
@(require "common.ss"
          scribble/struct)

@(define (convert-table . rows)
  (let* ([to-flow (lambda (x)
                    (make-flow (list
                                (make-paragraph (list x)))))]
         [spacer (to-flow (hspace 3))])                      
   (make-blockquote
    #f
    (list
   (make-table
    #f
    (cons
     (list (to-flow 'nbsp)
           spacer
           (to-flow "ASCII")
           spacer
           (to-flow "output"))
     (map (lambda (l)
            (list (to-flow (car l))
                  spacer
                  (to-flow (cadr l))
                  spacer
                  (to-flow (caddr l))))
          rows)))))))

@title[#:tag "decode"]{Decoding ASCII Text}

The @lit["@"] notation supports local text transformations and
mark-up, but it does not directly address some other problems in
organizing a document's source:

@itemize[

 @item{We would like to group section content implicitly via
       @scheme[section], @scheme[subsection], etc. declarations,
       instead of explicitly nesting constructions of @scheme[part]
       instances.}

 @item{We would like paragraphs to determined by a line of whitespace
       in the in source text, instead of explicitly constructing
       @scheme[paragraph] instances.}

 @item{We would like a handful of ASCII character sequences to be
       converted automatically to more sophisticated typesetting
       elements:

        @convert-table[
         (list @elem{left doublequote} @lit["``"]  @elem{``})
         (list @elem{right doublequote} @lit["''"]  @elem{''})
         (list @elem{apostrophe} @lit["'"]  @elem{'})
         (list @elem{em-dash} @lit["---"]  @elem{---})
         (list @elem{en-dash} @lit["--"]  @elem{--})
        ]}

]

A @defterm{decode} layer in Scribble provides these additional
facilities. For example, @scheme[bold] and @scheme[emph] apply
@scheme[decode-content] to their arguments to perform ASCII
transformations, and @scheme[item] calls @scheme[decode-flow] to both
transform ASCII sequences and form paragraphs between empty lines. In
contrast, @scheme[tt] and @scheme[verbatim] do not call the decode
layer, and they instead typeset text exactly as it is given.

The section-grouping portion of the decoder is applied the entire body
of a document through the @code-elem{#lang scribble/doc} prefix on a
document. This prefix actually performs three jobs: it installs the
Scribble @lit["@"]-notation extension of the Scheme reader to read the
document body, it lifts definitions and imports out of the document
body, and it wraps the document body in a call to
@scheme[decode-part]. The result of @scheme[decode-part] is bound to a
@schemeidfont{doc} identifier and exported.

For example, the source document

@code-block|{
#lang scribble/doc
@(require scribble/manual)

@title{All About @bold{Tubers}}

@section{Problem}

You say ``potato.''

I say ``potato.''

@section{Solution}

Call the whole thing off.
}|

is equivalent to the following module:

@code-block|{
#lang scheme/base
(require scribble/decode
         scribble/manual)

(provide doc)

(define doc
 (decode-part
  (list
   (title "All About " (bold "Tubers")) "\n"
   "\n"
   (section "Problem") "\n"   
   "\n"
   (emph "You") " say ``potato.''" "\n"
   "\n"
   (emph "I") " say ``potato.''" "\n"
   "\n"
   (section "Solution") "\n"   
   "\n"
   "Call the whole thing off." "\n")))
}|

The @scheme[decode-part] function recognizes the result of
@scheme[title] as a declaration of the document's title, and it
recognizes the result of each @scheme[section] call as a declaration
for a new section, so that the above is also equivalent to the
following:

@code-block|{
#lang scheme/base
(require scribble/decode
         scribble/manual)

(provide doc)

(define doc
 (make-part
  (decode-elements (list "All About " 
                         (bold "Tubers"))) ....
  (list
   (decode-part
    "Problem"
    (list (emph "You") " say ``potato.''" "\n"
          "\n"
          (emph "I") " say ``potato.''" "\n"))
   (decode-part
    "Solution"
    (list "Call the whole thing off.")))))
}|

The @scheme[decode-part] function, in turn, recognizes an empty line
as a paragraph separator, so that the example is equivalent to the
following:

@code-block|{
#lang scheme/base
(require scribble/decode
         scribble/manual)

(provide doc)

(define doc
 (make-part
  (decode-elements (list "All About " 
                         (bold "Tubers"))) ....
  (list
   (make-part
    "Problem" ....
    ;; Two paragraphs
    (list
     (decode-paragraph
      (list (emph "You") " say ``potato.''"))
     (decode-paragraph
      (list (emph "I") " say ``potato.''"))))
   (make-part
    "Solution" ....
    ;; One paragraph
    (list 
     (decode-paragraph
      (list "Call the whole thing off.")))))))
}|

Each call of @scheme[decode-paragraph] sends text on to
@scheme[decode-elements], which finally converts the quotes around
``potato'' to curly quotes.

The @scheme[decode-elements] function inspects only arguments that are
strings. An @scheme[element] created by @scheme[bold] or @scheme[emph]
thus left alone by @scheme[decode-elements], and any decoding needed
within the element is the job of @scheme[bold] or @scheme[emph] before
constructing the element. For example, the @scheme[bold] function is
implemented as follows:

@schemeblock[
(define (bold . content)
  (make-element 'bold
                (decode-elements content)))
]

In much the same way that @scheme[decode-elements] leaves an
@scheme[element] value as-is, @scheme[decode-part] recognizes
@scheme[part] instances and accepts them as implicit sub-parts of the
current section. The @scheme[include-section] form expands to import
@scheme[doc] from a given module, and then references the @scheme[doc]
in place, so that the document implemented by the module is
incorporated as a subsection. The above document can be arranged into
separate files as follows:

@code-block|{
;; In tubers.scrbl:
#lang scribble/doc
@(require scribble/manual)

@title{All About @bold{Tubers}}
@include-section["problem.scrbl"]
@include-section["solution.scrbl"]
}|

@code-block|{
;; In problem.scrbl:
#lang scribble/doc
@(require scribble/manual)

@title{Problem}

You say ``potato.''

I say ``potato.''
}|

@code-block|{
;; In solution.scrbl:
#lang scribble/doc
@(require scribble/manual)

@title{Solution}

Call the whole thing off.
}|

Note that as the ``Problem'' and ``Solution'' sections are moved to
their own files, the @scheme[section] command changes to
@scheme[title]. This change is similar to the change that is needed
when shifting sections to different levels in @|latex|. In Scribble,
however, the @scheme[title] and @scheme[section] are relative to the
enclosing module. As a result,
@scr:code-elem|{@include-section["potato.scrbl"]}| can be used at any
section level, in which case ``Problem'' is always incorporated as a
section that is one level deeper in the hierarchy.