doc.txt

R6RS Macros and R6RS libraries (van Tonder)

_R6RS Macros and R6RS libraries (van Tonder)_
_r6rs-expander-vantonder_

David Van Horn
<dvanhorn@cs.brandeis.edu>

This collection provides the van Tonder portable implementation of
R6RS Macros and R6RS libraries, version 6.0.2 (dec-27-07):

   http://www.het.brown.edu/people/andre/macros/

This library currently consists of a module _expander.ss_ that exports
the procedures _ex:repl_, _ex:expand-file_, _ex:expand-r5rs-file_,
etc. as defined in the van Tonder implementation, and a module
_repl.ss_ that launches an interactive REPL session.

Future versions of this library may provide support for integrating
R6RS libraries into MzScheme modules.

To run the interactive REPL from the command line:

   > mzscheme -p repl.ss dvanhorn r6rs-expander-vantonder.plt
   R6RS>

To run the interactive REPL from MzScheme:

   > (require (planet "repl.ss" ("dvanhorn" "r6rs-expander-vantonder.plt" 3 2)))
   R6RS>

To import the expander library:

   (require (planet "expander.ss" ("dvanhorn" "r6rs-expander-vantonder.plt" 3 2)))


Integrated REPL
===============

An integerated R6RS REPL is provided by requiring the _repl.ss_
module.  It is intended to be required from the command line as in:

   > mzscheme -p repl.ss dvanhorn r6rs-expander-vantonder.plt
   R6RS> (import (rnrs))
   R6RS> (define x 1)
   R6RS> x
   1
   R6RS> ^D
   >

Supported Libraries
-------------------

The interactive REPL fully supports a number of libraries:

   - rnrs base: Base library
   - rnrs unicode: Unicode
   - rnrs bytevectors: Bytevectors
   - rnrs lists: List utilities
   - rnrs sorting: Sorting
   - rnrs control: Control structures
   - rnrs records: Records 
   - rnrs io simple: Simple I/O
   - rnrs files: File system
   - rnrs programs: Command-line access and exit values
   - rnrs arithmetic bitwise: Exact bitwise arithmetic
   - rnrs syntax-case: syntax-case
   - rnrs: Composite library
   - rnrs eval: eval
   - rnrs mutable-pairs: Mutable pairs
   - rnrs mutable-strings: Mutable strings
   - rnrs r5rs: R5RS compatibility
   - srfi n5: A compatible let form with signatures and rest arguments 
   - srfi n6: Basic String Ports
   - srfi n8: receive: Binding to multiple values
   - srfi n9: Defining Record Types
   - srfi n11: Syntax for receiving multiple values
   - srfi n14: Character-set Library (Unicode) (*)
   - srfi n16: Syntax for procedures of variable arity
   - srfi n17: Generalized set!
   - srfi n26: Notation for Specializing Parameters without Currying
   - srfi n28: Basic Format Strings
   - srfi n31: A special form `rec' for recursive evaluation
   - srfi n39: Parameter objects
   - srfi n42: Eager Comprehensions
   - srfi n54: Formatting
   - srfi n60: Integers as Bits
   - srfi n63: Homogeneous and Heterogeneous Arrays
   - srfi n66: Octet Vectors
   - srfi n69: Basic hash tables
   - srfi n74: Octet-Addressed Binary Blocks
   - srfi n78: Lightweight testing
   - srfi n95: Sorting and Merging
   - slib record: SLIB Records
          http://www-swiss.ai.mit.edu/~jaffer/slib_7.html#SEC204
   - explicit-renaming: Explicit renaming mechanism included in van
     Tonder implementation.
   - ubik include: Source code inclusion mechanism for libraries.
   - ubik define-values: Definition form for multiple values.
   - ubik inversion-lists: Inversion list representation of sets of integers (*)

(*) New this release.

Test suites: 

   - srfi n5 tests
   - srfi n6 tests
   - srfi n9 tests
   - srfi n14 tests (*)
   - srfi n42 tests
   - srfi tests
   - ubik inversion-lists tests (*)
   - ubik tests (*)
   - tests (*)

To run any of the test suites, import the relevant library.

The REPL currently lacks support for the following R6RS libraries:

   - rnrs hashtables 
   - rnrs enumerations
   - rnrs arithmetic fixnums
   - rnrs arithmetic flonums
   - rnrs io ports
   - rnrs exceptions
   - rnrs conditions

Primitives
----------

MzScheme bindings can be imported by using the _primitives_ import
clause (an extension to R6RS).  For example, we can import the
MzScheme primitive add1 procedure as follows:

   R6RS> (import (primitives add1))
   R6RS> (add1 1)
   2   

Primitives include any bindings provided by MzScheme.  Additional
primitives can be made available through the _#!mzprim_ lexeme which
must be followed by a require specification.  The specified module is
namespace-required and the new bindings can be imported using
primitives.  For example:

   R6RS> #!mzprim(lib "list.ss")
   R6RS> (import (primitives cons?))
   R6RS> (cons? 1)
   #f

Note that macros may not be imported with the primitives clause.

See documentation on _ex:repl_ for details on the semantics of REPL
evaluation.

Reader
------

The interactive REPL uses a modified reader that is intended to be a
conformant R6RS reader.  It falls short in a number of ways currently,
however it supports the following features:

   - -> identifier syntax
   - #!r6rs comments
   - #vu8 syntax, although it will raise an error since this is
      unsupported by the expander.
   - prohibits intial vertical bars in symbols and identifiers.

Additional libraries
--------------------

The interactive REPL has a number of libraries available beyond what
is distributed with the van Tonder reference implementation.

   - (rnrs programs (6))
   


REPL
====

  > (ex:repl quoted-top-level-sequence-of-forms)

  A REPL is provided that is integrated with the r6rs library system.
  Libraries and programs can be defined at, imported into and run in
  the REPL toplevel.  The ex:repl procedure expands and evaluates a
  sequence of forms one by one.

  The REPL goes beyond r6rs to allow incremental development in a
  toplevel environment.  The developer can freely change, replace and
  make new toplevel definitions, evaluate toplevel expressions, enter
  libraries and <toplevel programs> at the prompt, as well as import
  libraries into the toplevel environment.

Example
-------

Here is a simple example use of the REPL:

  > (require (planet "expander.ss" ("dvanhorn" "r6rs-expander-vantonder.plt" 3 1)))
  > (ex:repl '( 1 ))
  prints: 1

In order to access any bindings such as +, define, lambda, etc., you
must first import the appropriate library.  So the following fails:

  > (ex:repl '( (+ 1 1) ))
  error: reference to undefined identifier: ~+

But importing the base library provides the binding for +, as in:

  > (ex:repl '( (import (rnrs base))
                (+ 1 1) ))
  prints: 2

Note that the ex:repl procedure is stateful.  Once a library has been
imported into the top-level, subsequent uses of ex:repl will have
access to those library bindings.  So, for example:

   > (ex:repl '( (+ 1 1) ))
   prints: 2

An extensive example REPL session is given in:

   http://www.het.brown.edu/people/andre/macros/aug-14-07/examples.scm

This file is also included in the private/src/ directory.  

Expander
========

  > (ex:expand-file source-file destination-file dependency-file ...)

  Use this to expand a file containing libraries and/or toplevel
  programs before loading into an r5rs-type system or feeding result
  to an r5rs-type compiler.

  The expander expands everything to r5rs toplevel definitions
  and expressions, so the expanded code should be compilable
  with an r5rs compiler.

Example
-------

Source code for the standard libraries is included in
private/src/standard-libraries.scm, which contains a series of library
forms.  This file can be expanded as follows (assuming it is in your
current path):

   (ex:expand-file "standard-libraries.scm" "standard-libraries.exp")

The file standard-libraries.exp contains an R5RS top-level sequence
for the fully expanded standard libraries.  The program can then be
run by loading it.

Libraries must be expanded given the names of all the expanded
libraries on which they depend.

See the examples file for more involved compilation examples.


The original readme follows.

---
;;;===============================================================================
;;;
;;; R6RS Macros and R6RS libraries:
;;;
;;;   Copyright (c) 2006 Andre van Tonder
;;;
;;;   Copyright statement at srfi.schemers.org/srfi-process.html
;;;
;;;===============================================================================
 
----------------------------------------------------------------------
TO RUN : execute examples.scm
----------------------------------------------------------------------

Should run out of the box on any R5RS-compliant system that 
provides (interaction-environment) and load.

Tested on: MzScheme     (compat file provided)
           Petite Chez  (compat file provided)
           Larceny      (compat file provided)
           Scheme48     (compat file provided)

R6RS COMPLIANCE NOTES:
======================

SYNTAX-CASE: 
------------   

 FULLY R6RS-COMPLIANT, WITH THE FOLLOWING OPTIONAL OR RECOMMENDED FEATURES:
  
   * Expansion algorithm (r6rs optional):
   
       We use a renaming algorithm instead of the mark-antimark
       algorithm described in the draft.  The difference will not 
       matter for r6rs-conforming macros.

   * Wrapped = Unwrapped (r6rs optional):
   
       A wrapped syntax object is the same as an unwrapped syntax 
       object, and can be directly manipulated using car, cdr, ... 
       without syntax-case deconstruction.  
       This should make porting of legacy low-level macros,
       e.g., explicit renaming or define-macro, easier.
       Macros depending on this feature wil not be 
       portable to all r6rs implementations.  
                 
   * Lexical scoping violation detection (r6rs recommended):
   
       We treat a violation of the following restriction in chapter 8
       as a syntax violation:
       
       A definition in the sequence of forms must not define any 
       identifier whose binding is used to determine the meaning of the
       undeferred portions of the definition or any definition that 
       precedes it in the sequence of forms.       

LIBRARIES:
----------

 FULLY R6RS-COMPLIANT WITH ONE EXCEPTION (SEE BELOW) WITH THE FOLLOWING 
 OPTIONAL FEATURES: 
   
   * Phase semantics (r6rs optional):
   
       - A syntax violation is raised at expand-time if a binding is
         referenced outside its declared levels.  We do this for 
         both variable references and macro references.  As we are 
         entitled to do by R5.97RS, we also check levels for some
         references that occur when free-identifier=? is called.
         Specifically: 
          
         A syntax error is raised if free-identifier=? succeeds 
         but either argument is outside its declared level.  This is sufficient 
         to ensure that literals such as ... in syntax-case are used at the 
         correct level.  See the examples file for more discussion, and for 
         an explanation of why the levels should not be checked if the 
         comparison fails.  
         
         Note: This choice raises a syntax violation when MzScheme and Chez
               would disagree, thus ensuring that a program blessed by this
               expander will be maximally portable.  
                
       - Any given library is visited at most once per session, and 
         is invoked at most once per session, unless the library is redefined.
         Thus, any state the library may have will persist during the session
         until it is redefined.  

         This behaviour, which is R6RS-compliant, avoids possible quadratic 
         growth in the number of library invocations when importing many libraries 
         with lots of dependencies.  
       
       - While expanding a library form, we use an instance/visit at any phase as 
         an instance/visit at any other phase.     
                   
       - We may create instances/visits of more libraries than are 
         strictly required to satisfy references, but only in the case - of 
         which the semantics is unspecified in r6rs - where a library is imported 
         but none of its bindings is used.  
         Specifically:
                
         During expansion of a library or program, each library imported 
         at phase n > 0 is instantiated once, and each library imported 
         at phase n >= 0 is visited once.   
                
         During execution of a program, each library imported 
         at phase n = 0 is instantiated once.
       
 KNOWN R6RS-NONCOMPLIANCE:       
       
   * Version name and reference are syntax-checked, but otherwise ignored. 
 
ADDITIONAL FEATURES:
==================== 

   * Separate and joint code organization and expansion:
   
       Libraries and programs may be in separate files or in the same file,
       and may be separately or jointly expanded. 
       See example scripts in examples.scm.

   * Repl:
   
       A REPL is provided that is integrated with the r6rs library system.  
       Libraries and programs can be defined at, imported into and run in the REPL toplevel. 
       The ex:repl procedure expands and evaluates a sequence of forms one by one. 
       See example REPL session examples.scm.

   * Dynamic loading of libraries:

       A non-standard library (rnrs load) is provided containing LOAD, which can be
       used for dynamic loading of any code, including dynamic loading 
       of R6RS libraries.
       See example in examples.scm.
       
   * Reflection:
    
       A library (rnrs eval reflection (6)) is provided.  
       It exports the procedure (environment-bindings <environment>).
       Displays for each binding whether it is a variable or macro, its
       source library, and its levels.
       Very useful for library development and debugging. 
       See examples of use in examples.scm.  
       
   * Explicit renaming:
     
       A compatible explicit renaming library (explicit-renaming) has 
       been included.  Although such a library is definable in syntax-case
       implementations that follow the algorithm described in the r6rs draft, 
       it would be inherently inefficient, suffering from an unavoidable
       increase in complexity class over comparable syntax-case macros.  
       Our algorithm does not have this problem, is in fact nothing but
       explicit renaming behind the scenes, and allows us to expose
       explicit renaming efficiently.
       See further documentation in standard-libraries.scm and 
       tutorial in examples.scm.  
 
CHANGELOG:
========== 

CHANGES SINCE VERSION OF NOV 6, 2007:
-------------------------------------

- ONCE PER SESSION SEMANTICS:
  Libraries are now visited/invoked at most ONCE PER SESSION unless
  they get redefined.  This prevents possible quadratic growth in the 
  number of library invocations when importing many libraries with
  lots of dependencies.  
- LOAD ON IMPORT: 
  Fixed issue preventing expander from being called reentrantly by
  ex:load, ex:compile-file, etc.  These can now be used, for example, 
  to compile/load libraries automatically on import.  
  See example code in ex:lookup-library in runtime.scm
  for how to do this.  
- Fixed bug preventing certain R6RS template extensions from working.  
  Now the following will work correctly: 
   (syntax-case '((1 2 3) (4 5 6) (7 8 9)) ()
     (((Var ...) 
       (Var2 ...) ...)
      (syntax
       ((Var ... Var2 ...) ...))))  ;==> ((1 2 3 4 5 6) (1 2 3 7 8 9)) 
- Improved definition of procedure LOAD in (rnrs load).
- Now raises proper error message when using R6RS-invalid syntax
  (define-syntax (foo x) ----) instead of (define-syntax foo (lambda (x) ----))
- Removed identifiers ending in ":" so that code may run on Gambit.


CHANGES SINCE VERSION OF OCT 29, 2007:
--------------------------------------

- Fixed bug where lists denoted by pattern variables in 
  templates like ((x y) ...) were not being checked for being 
  equal in length 
- Fixed bug causing inconsistent/useless toplevel when syntax error
  happened while expanding library

CHANGES SINCE VERSION OF AUG 19, 2007:
--------------------------------------

- Added non-standard library (rnrs load) containing LOAD, which can be
  used for dynamic loading of any code, including dynamic loading 
  of R6RS libraries.
- Added (r5rs) non-standard library.
- Added ASSERT to (rnrs base) and (rnrs).
- Added missing stuff to (rnrs unicode).
- Added (rnrs sorting), (rnrs records procedural), (rnrs records inspection),
  (rnrs files), (rnrs arithmetic fixnums), (rnrs arithmetic flonums), 
  (rnrs arithmetic bitwise).
- Added additional bindings in these libraries to (rnrs).
- Removed mutable string primitives from (rnrs (6)).
- Fixed minor bug that prevented (environment ---) from executing
  correctly when evaluated in phase > 0.
- Updated compat-larceny.scm to latest version found at
  http://larceny.ccs.neu.edu/larceny-trac/browser/trunk/larceny_src/lib/R6RS

CHANGES SINCE VERSION OF AUG 14, 2007:
--------------------------------------

- Added support for Scheme48.
- R5RS compliance: EVAL now takes interaction-environment second argument
- R5RS compliance: Use of - on three arguments has been removed
- R5RS compliance: replaced non-portable identifier ..0 by ___
- Letrec now correctly sets variables to black hole instead of unspecified

CHANGES SINCE VERSION OF JULY 31, 2007:
--------------------------------------

- A compatible explicit renaming library has been included.
  See documentation in standard-libraries.scm and tutorial
  in examples.scm.
- Now completely r5rs-portable.
- Greatly expanded the porting notes.
- The few bindings necessary at runtime have been separated
  into the small runtime include file runtime.scm.
  As a result, since object programs do not depend on the full
  expander, their runtime images can be much smaller than before.
- All runtime uses of eval, and a number of expand-time uses of eval,
  have been removed.
- Added ex:expand-r5rs-file for expanding r5rs-like toplevel files.
- We now check that libraries imported at runtime are the
  same build that the client library was expanded against.
- Uses better environment compression policy to make library
  object code smaller.
- Changed some uses of unspecified value to undefined value
  denoting the letrec(*) black hole.
- Replaced explicit deconstruction with pattern matching for
  better readability and maintenance.
- The code has been modularized with let and letrec-syntax to
  minimize the number of toplevel exports.  It has been reorganized
  so that internal definitions will work with both r5rs letrec
  and r6rs letrec* semantics.
- The expander now relies on r5rs syntax-rules and letrec-syntax
  and should run in a correct r5rs system.  It is also metacircular.
  In other words, if you don't have r5rs macros, you may use the
  provided fully expanded version to bootstrap modifications by
  feeding the expander to its expanded self.
- Removed uses of srfi-9.
- Library bindings are now properly removed before reinvoking
  a previously invoked library.
- Changed representation of libraries.
- Added missing export char? to (rnrs base) and (rnrs).
- Expansion of lambda has been optimized to eliminate a closure
  in the no-internal-definition case.
- Added 'only' annotations to standard libraries to make their
  environments, and therefore object code, much smaller.
- Removed uses of let-values.
- Removed uses of nonstandard parameterize.
- Reorganized code so it should run on r5rs systems with letrec
  semantics for internal definitions.
- Cleaned up initialization code.
- Fixed a bug where a new color was not created when invoking
  identifier-macro targets of set!
- Improved performance of eval by preparing environment fully
  during evaluation of (environment ---), not (eval ---) as before.
- Converted #'. #`, #, and #,@ to (quasi-)syntax and unsyntax(-splicing)
  so we are independent of this reader extension.

CHANGES SINCE VERSION OF JULY 12, 2007:
--------------------------------------

- Immutability checks:
  It is now a syntax violation if an explicitly exported variable appears on the 
  left-hand side of a set! expression, either in the exporting or importing libraries.
  It is also a syntax violation if a variable appears on the left-hand side of a set!
  expression in any code produced by an exported macro outside of the library in which 
  the variable is defined. 
  It is also a syntax violation if a reference to an assigned variable appears in any 
  code produced by an exported macro outside of the library in which the variable is 
  defined.
- Added library (rnrs eval reflection (6)) for reflection facilities:
  (environment-bindings <environment>) lists the bindings in an environment
  Very useful for development and dsbugging of libraries
  See the examples in examples.scm file
- replaced implementation of QUASIQUOTE with optimised Dybvig portable syntax-case version
- an unbound identifier at toplevel will now match unbound literal in library
- improved some error messages to state source libraries of bindings
- Larceny compat updated to v0.94
- corrected free occurrence of FIRST in ex:unify-imports that prevented an error message
- removed caching of imported macros
- added missing version (6) to library (rnrs mutable-strings)
- only, except, prefix, and rename now enforce the constraints stated in the r6rs draft

CHANGES SINCE VERSION OF JULY 10, 2007:
--------------------------------------

 - Fixed bug causing internal error when a literal appeared followed by
   ellipses, as in (syntax-rules (x) ((m x ...) #t))

CHANGES SINCE VERSION OF JUNE 22, 2007:
--------------------------------------

 - simplified expansion of bodies - "wraps"
 - changed "mark" to "color" everywhere to avoid confusion with Dybvig algorithm
 - changed unify-imports so it runs much faster
 - reorganized standard libraries so more common names near front of envs
 - simplified environment model significantly
 - fixed bug preventing nonpair, nonvector, nonsymbol values in syntax
   objects and patterns
 - removed UNSPECIFIED
 - we now detect and raise syntax error for pattern variables outside templates
 - corrected bug where pattern variable occurred outside a template in standard-libraries.scm
 - corrected bug preventing FOR clause from having no import levels
 - as required by r6rs, free-identifier=? now treats unbound identifiers specially
 - as required by r6rs, modified free-identifier=? behaviour w.r.t phases.
   Previously, free-identifier=? could return #f in cases where both arguments referred to
   the same binding but either argument was out of phase.  Now an out of phase error is
   raised in this case, so that a #f result can only happen when the arguments refer to
   different bindings, and a #t result when they refer to the same binding.
 - as allowed by R6RS, included refined phase checking of arguments to free-identifier=?
   An out of phase error is raised if the comparison succeeds but either argument is
   out of phase.  This is sufficient to ensure that literals such as ... in syntax-case
   are used in the correct phase.  See the examples file for more discussion.  
 - out of phase uses of internal DEFINE, DEFINE-SYNTAX, LET[REC]-SYNTAX, BEGIN will now give a phase error
 - for portability, out of context references to let[rec]-syntax bindings now give a syntax error
 - refined algorithm for better detection of definitions that affect previously expanded
   undeferred portions of body.  Should now catch all cases.
 - changed syntax of (primitives (id ...)) to (primitives id ...) 
 - implemented r6rs library version reference syntax
 - implemented (library ---) import clause
 - EVAL now raises a syntax error for definition or sequence containing definition
 - added (rnrs r5rs) including 
      (null-environment 5) 
      (scheme-report-environment 5)
      delay and force
 - Added stubs for (rnrs uncode), (rnrs io simple), (rnrs mutable-strings)
 - cleaned up standard library organization.

CHANGES SINCE VERSION OF MARCH 1, 2007:
---------------------------------------

 - added Larceny, MzScheme and Chez compatibilty files.
 - procedure ex:expand-file provided, that can be used as an r6rs front end
   to an existing r5rs compiler.  Libraries and toplevel programs may be in 
   separate files or in the same file, and may be separately or jointly 
   expanded or compiled.  Interface is 
   (ex:expand-file source-filename target-filename dependency ...)
   The dependencies must list the already expanded files
   containing libraries to be imported.  
 - added examples and scripts for separate and joint compilation in examples.scm
 - code no longer depends on the non-r5rs assumption of letrec* semantics for 
   internal definitions.
 - replaced uses of define-struct by uses of srfi-9.
 - export-levels of identifiers in standard libraries made consistent with 5.95,
   although I think 5.95 is still wrong on this.  
 - ... and _ disallowed as literals, although I think 5.95 is wrong on this.   
 - removed requirement that toplevel program should end with expression.
 - removed unused "level" argument in expanded library visiting and instantiation code. 
 - fixed bug that prevented certain redefinitions in toplevel repl (begin ...) sequences.  
 - changed contract-violation -> assertion-violation.
 - added (r6rs control) to standard libraries.
 - fixed bug that prevented shadowing of macro keywords by later variable definitions
   ar repl toplevel.

CHANGES SINCE VERSION OF JANUARY 15, 2007:
------------------------------------------

 - updated README to reflect compatibility with R(5.92)RS.

CHANGES SINCE VERSION OF DECEMBER 25, 2006:
-------------------------------------------

 - corrected bug in phase shift enforcement while visiting and
   invoking, which affected behaviour of syntax forms evaluated 
   in a library's top-level expand-time or runtime sequence.  
 - (r6rs base) now also exports ... and _ for use with syntax-rules.

CHANGES SINCE VERSION OF DECEMBER 17, 2006:
-------------------------------------------

 - dropped unshared instantiation option, keeping only shared semantics.
 - corrected export phase of syntax-rules and identifier-syntax
   in (r6rs base).  These are now exported for expand instead of 
   for run.
 - simplified documentation in readme.    

CHANGES SINCE VERSION OF DECEMBER 14, 2006:
-------------------------------------------

 - removed support for declarations
 - changed script -> program everywhere
 - changed add-prefix -> prefix everywhere
 - dropped the unparenthesized shorthand for library names
 - changed forall -> for-all everywhere
 - fenders have been removed from syntax-rules
 - added "identifier-syntax" to (r6rs base)

CHANGES SINCE VERSION OF NOVEMBER 27, 2006:
-------------------------------------------
 
 - corrected implementation and description of invocation semantics  
   and its interaction with negative import levels.    
   
CHANGES SINCE VERSION OF NOVEMBER 5, 2006:
------------------------------------------
 
 - added UNSHARED semantics (see above) as a configurable option.  
 
CHANGES SINCE VERSION OF SEPTEMBER 13, 2006:
--------------------------------------------
 
 - added support for scripts.
 - quasisyntax now works with unsyntax(-splicing) instead of unquote(-splicing). 
 - many bug fixes and improvements in code.
 - reorganized standard libraries. 
 - properly avoids defect in R(5.91)RS assigning lexical scope violating 
   semantics to certain expressions.
 - implemented library reference syntax.