Whalesong: a Racket to JavaScript compiler

Version: 5.1.3

Whalesong: a Racket to JavaScript compiler

Source code can be found at: https://github.com/dyoo/whalesong. The latest version of this document lives in http://hashcollision.org/whalesong.

Current commit head is 139b31d706023d6ec93b4712877c0367c0afc40f .

Warning: this is work in progress!

1 Introduction

Whalesong is a compiler from Racket to JavaScript; it takes Racket programs and translates them so that they can run stand-alone on a user’s web browser. It should allow Racket programs to run with (hopefully!) little modification, and provide access through the foreign-function interface to native JavaScript APIs. The included runtime library supports the numeric tower, an image library, and a framework to program the web in functional event-driven style.

The GitHub source repository to Whalesong can be found at https://github.com/dyoo/whalesong.

Prerequisites: at least Racket 5.1.1. If you wish to use the JavaScript compression option, you will need Java 1.6 SDK.

1.1 Examples

Here are a collection of programs that use the web-world library described later in this document:

attr-animation.html [src] Uses update-view-attr and on-tick to perform a simple color animation.
boid.html [src] Uses update-view-css and on-tick to perform an animation of a flock of boids.
dwarves.html [src] Uses view-show and view-hide to manipulate a view. Click on a dwarf to make them hide.
dwarves-with-remove.html [src] Uses view-focus? and view-remove to see if a dwarf should be removed from the view.
field.html [src] Uses view-bind to read a text field, and update-view-text to change the text content of an element.
phases.html [src] Switches out one view entirely in place of another. Different views can correspond to phases in a program.
tick-tock.html [src] Uses on-tick to show a timer counting up.
redirected.html [src] Uses on-tick to show a timer counting up, and also uses open-output-element to pipe side-effecting printfs to a hidden div.
todo.html [src] A simple TODO list manager.
where-am-i.html [src] Uses on-location-change and on-mock-location-change to demonstrate location services.

I also gave a presentation of Whalesong at RacketCon 2011, including examples like:

rain.html [src] Uses the image libraries to show droplets of water falling down.
pacman.html [src] Pacman.

2 Getting started

2.1 Installing Whalesong

At the time of this writing, although Whalesong has been deployed to PLaneT, the version on PLaneT is probably a little out of date.

If you want to use Whalesong off of PLaneT, run the following to create the "whalesong" launcher:

#lang racket/base
(require (planet dyoo/whalesong:1:3/make-launcher))

This will create a "whalesong" launcher in the current directory.

2.2 Installing Whalesong from github

Otherwise, you can download the sources from the github repository. Doing so requires doing a little bit of manual work. The steps are:

Check Whalesong out of Github.
Set up the PLaneT development link to your local Whalesong instance.
Run raco setup over Whalesong to finish the installation

We can check it out of the source repository in GitHub; the repository can be checked out by using git clone. At the command-line, clone the tree with:

$ git clone git://github.com/dyoo/whalesong.git

This should check the repository in the current directory.

Next, let’s set up a PLaneT development link. Make sure you are in the parent directory that contains the "whalesong" repository, and then run this on your command line:

$ planet link dyoo whalesong.plt 1 4 whalesong

(You may need to adjust the 1 and 4 major/minor numbers a bit to be larger than the latest version that’s on PLaneT at the time.)

Let’s make the "whalesong" launcher somewhere appropriate. Run Racket with the following require:

(require (planet dyoo/whalesong/make-launcher))

This will create a "whalesong" executable in the current working directory.

Finally, we need to set up Whalesong with raco setup. Here’s how to do this at the command line:

$ raco setup -P dyoo whalesong.plt 1 4

This should compile Whalesong. Any time the source code in "whalesong" changes, we should repeat this raco setup step again.

At this point, you should be able to run the "whalesong" executable from the command line.

$ ./whalesong

Usage: whalesong <subcommand> [option ...] <arg ...>

where any unambiguous prefix can be used for a subcommand

The Whalesong command-line tool for compiling Racket to JavaScript

For help on a particular subcommand, use 'whalesong <subcommand> --help'

whalesong build build a standalone html and javascript package

whalesong get-runtime print the runtime library to standard output

whalesong get-javascript Gets just the JavaScript code and prints it to standard output

and if this does appear, then Whalesong should be installed successfully.

To repeat: whenever Whalesong’s source code is updated from Github, please re-run the raco setup step. Otherwise, Racket will try to recompile Whalesong on every single use, which can be very expensive.

2.3 Making .html files with Whalesong

Let’s try making a simple, standalone executable. At the moment, the program must be written in the base language of (planet dyoo/whalesong). This restriction unfortunately prevents arbitrary racket/base programs from compiling at the moment; the developers (namely, dyoo) will be working to remove this restriction as quickly as possible.

Write a "hello.rkt" with the following content

"hello.rkt"
#lang planet dyoo/whalesong
(display "hello world")
(newline)

This program is a regular Racket program, and can be executed normally,

$ racket hello.rkt

hello world

However, it can also be packaged with "whalesong".

$ whalesong build hello.rkt

Writing program #<path:/home/dyoo/work/whalesong/examples/hello.js>

Writing html #<path:/home/dyoo/work/whalesong/examples/hello.html>

$ ls -l hello.html

-rw-r--r-- 1 dyoo dyoo 3817 2011-09-10 15:02 hello.html

$ ls -l hello.js

-rw-r--r-- 1 dyoo dyoo 2129028 2011-09-10 15:02 hello.js

Visit hello.html to execute this program.

Running whalesong build on a Racket program will produce a ".html" and ".js" file. If you open the ".html" in your favorite web browser, you should see a triumphant message show on screen.

We can do something slightly more interesting. Let’s write a Whalesong program that accesses the JavaScript DOM. Call this file "dom-play.rkt".

Visit dom-play.html to execute this program.

"dom-play.rkt"
#lang planet dyoo/whalesong

;; Uses the JavaScript FFI, which provides bindings for:
;; $ and call
(require (planet dyoo/whalesong/js))

;; insert-break: -> void
(define (insert-break)
  (call-method ($ "<br/>") "appendTo" body)
  (void))

;; write-message: any -> void
(define (write-message msg)
  (void (call-method (call-method (call-method ($ "<span/>") "text" msg)
                    "css" "white-space" "pre")
              "appendTo"
              body)))

;; Set the background green, and show some content
;; on the browser.
(void (call-method body "css" "background-color" "lightgreen"))
(void (call-method ($ "<h1>Hello World</h1>") "appendTo" body))
(write-message "Hello, this is a test!")
(insert-break)
(let loop ([i 0])
  (cond
   [(= i 10)
    (void)]
   [else
    (write-message "iteration ") (write-message i)
    (insert-break)
    (loop (add1 i))]))

This program uses the JQuery API provided by (planet dyoo/whalesong/js), as well as the native JavaScript FFI to produce output on the browser. If w run Whalesong on this program, and view the resulting "dom-play.html" in your web browser, we should see a pale, green page with some output.

2.4 Using Whalesong functions from JavaScript

Whalesong also allows functions defined from Racket to be used from JavaScript. As an example, we can take the boring factorial function and define it in a module called "fact.rkt":

The files can also be downloaded here:
fact.rkt
index.html
with generated JavaScript binaries here:
fact.js
runtime.js

"fact.rkt"
#lang planet dyoo/whalesong
(provide fact)
(define (fact x)
  (cond
   [(= x 0)
    1]
   [else
    (* x (fact (sub1 x)))]))

Instead of creating a standalone .html, we can use whalesong to get us the module’s code. From the command-line:

$ whalesong get-javascript fact.rkt > fact.js

$ ls -l fact.js

-rw-r--r-- 1 dyoo dyoo 27421 2011-07-11 22:02 fact.js

This file does require some runtime support not included in "fact.js"; let’s generate the runtime.js and save it as well. At the command-line:

$ whalesong get-runtime > runtime.js

$ ls -l runtime.js

-rw-r--r-- 1 dyoo dyoo 544322 2011-07-11 22:12 runtime.js

Now that we have these, let’s write an "index.html" that uses the fact function that we provideed from "fact.rkt".

"index.html"
<!DOCTYPE html>
<html>
<head>
<script src="runtime.js"></script>
<script src="fact.js"></script>

<script>
// Each module compiled with 'whalesong get-runtime' is treated as a
// main module.  invokeMains() will invoke them.
plt.runtime.invokeMains();

plt.runtime.ready(function() {

   // Grab the definition of 'fact'...
   var myFactClosure = plt.runtime.lookupInMains('fact');

   // Make it available as a JavaScript function...
   var myFact = plt.baselib.functions.asJavaScriptFunction(
        myFactClosure);

   // And call it!
   myFact(function(v) {
              $('#answer').text(v.toString());
          },
          function(err) {
              $('#answer').text(err.message).css("color", "red");
          },
          10000
          // "one-billion-dollars"
          );
});
</script>
</head>

<body>
The factorial of 10000 is <span id="answer">being computed</span>.
</body>
</html>

See: bad-index.html.

Replacing the 10000 with "one-billion-dollars" should reliably produce a proper error message.

3 Using whalesong

Whalesong provides a command-line utility called whalesong for translating Racket to JavaScript. It can be run in several modes:

To create HTML + js documents
To output the compiled JavaScript as a single ".js" file

Using whalesong to generate HTML+js documents is relatively straightforward with the build command. To use it, pass the name of the file to it:

$ whalesong build [name-of-racket-file]

A ".html" and ".js" will be written to the current directory, as will any external resources that the program uses.

The whalesong commands support these command line options:

--compress-javascript
Use Google Closure’s JavaScript compiler to significantly compress the JavaScript. Using this currently requires a Java 1.6 JDK.
--verbose
Write verbose debugging information to standard error.
--dest-dir
Write files to a separate directory, rather than the current directory.
--split-modules
Write each dependent module as a separate file, rather than in one large ".js". This may be necessary if your browser environment prohibits large ".js" files. The files will be numbered starting from 1.

For more advanced users, whalesong can be used to generate JavaScript in non-standalone mode. This gives the web developer more fine-grained control over how to control and deploy the outputted program.

3.1 build

Given the name of a program, this builds ".html" and ".js" files into the current working directory.

The ".html" and ".js" should be self-contained, with an exception: if the file uses any external resources by using define-resource, those resources are written into the current working directory, if they do not already exist there.

3.2 get-javascript

Given the name of a program, writes the JavaScript to standard output, as well as its dependent modules. The outputted file is meant to be used as a SCRIPT source.

By default, the given program will be treated as a main module. All main modules will be executed when the JavaScript function plt.runtime.invokeMains() is called.

3.3 get-runtime

Prints out the core runtime library that the files generated by get-javascript depend on.

4 Including external resources

(require (planet dyoo/whalesong:1:3/resource))

Programs may need to use external file resources that aren’t themselves Racket programs, but instead some other kind of data. Graphical programs will often use ".png"s, and web-related programs ".html"s, for example. Whalesong provides the (planet dyoo/whalesong:1:3/resource) library to refer and use these external resources. When Whalesong compiles a program into a package, these resources will be bundled alongside the JavaScript-compiled output.

(define-resource id [path-string])

Defines a resource with the given path name.

For example,

#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/resource))
(define-resource my-whale-image-resource "humpback.png")

As a convenience, you can also write

#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/resource))
(define-resource humpback.png)

which defines a variable named humpback.png whose resource is "humpback.png".

If the resource given has an extension one of the following:

".png"
".gif"
".jpg"
".jpeg"

then it can be treated as an image for which image? will be true.

If the resource has the extension ".html", then it will be run through an HTML purifying process to make sure the HTML is well-formed.

(resource? x) → boolean

x : any

Returns #t if x is a resource.

(resource->url a-resource) → string?

a-resource : resource?

Given a resource, gets a URL.

For example,

#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/resource)
(planet dyoo/whalesong/image))

(define-resource my-whale-image-resource "humpback.png")

(define WHALE-IMAGE
(bitmap/url (resource->url my-whale-image-resource)))

5 The web-world API

(require (planet dyoo/whalesong:1:3/web-world))

The web-world library allows you to write functional event-driven World programs for the web; the user defines functional callbacks to handle events, and receive and consume a world argument.

One difference introduced by the web is the web page itself: because the page itself is a source of state, it too will be passed to callbacks. This library presents a functional version of the DOM in the form of a view.

Visit tick-tock.html to execute this program.

Let’s demonstrate this by creating a basic ticker that counts on the screen every second.

The first thing we can do is mock up a web page with a user interface, like this.

"index.html"
<html>
  <head><title>My simple program</title></head>
  <body>
    <p>The current counter is: <span id="counter">fill-me-in</span></p>
  </body>
</html>

We can even look at this in a standard web browser.

Once we’re happy with the statics of our program, we can inject dynamic behavior. Write a file called "tick-tock.rkt" with the following content.

"tick-tock.rkt"
#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/web-world)
         (planet dyoo/whalesong/resource))

(define-resource index.html)

;; draw: world view -> view
(define (draw world dom)
  (update-view-text (view-focus dom "counter") world))

;; tick: world view -> world
(define (tick world dom)
  (add1 world))

;; stop?: world view -> boolean
(define (stop? world dom)
  (> world 10))

(big-bang 0
          (initial-view index.html)
          (to-draw draw)
          (on-tick tick 1)
          (stop-when stop?))

Several things are happening here.

We require a few libraries to get us some additional behavior; in particular, (planet dyoo/whalesong:1:3/web-world) to let us write event-driven web-based programs, and (planet dyoo/whalesong:1:3/resource) to give us access to external resources.
We use define-resource to refer to external files, like "index.html" that we’d like to include in our program.
We use big-bang to start up a computation that responses to events. In this example, that’s clock ticks introduced by on-tick, though because we’re on the web, we can bind to many other kinds of web events (by using view-bind).

5.1 big-bang and its options

(big-bang w h ...) → world

w : world

h : big-bang-handler

Start a big bang computation. The big-bang consumes an initial world, as well as several handlers to configure it, described next:

(initial-view x) → big-bang-handler

x : any

Provide an initial view for the big-bang. Normally, x will be a resource to a web page.

...
(define-resource page1.html)
...
(big-bang ...
(initial-view page1.html))

(stop-when stop?) → big-bang-handler

stop? : ([w world] [dom view] -> boolean)

Tells big-bang when to stop.

...
(define-struct world (given expected))
...

;; stop?: world view -> boolean
(define (stop? world dom)
(string=? (world-given world) (world-expected world)))

(big-bang ...
(stop-when stop?))

(on-tick tick-f delay) → big-bang-handler

tick-f : ([w world] [v view] [e event]? -> world)

delay : real

(on-tick tick-f) → big-bang-handler

tick-f : ([w world] [v view] [e event]? -> world)

Tells big-bang to update the world during clock ticks.

By default, this will send a clock tick 28 times a second, but if given delay, it will use that instead.

...
;; tick: world dom -> world
(define (tick world view)
(add1 world))

(big-bang ...
(on-tick tick 5)) ;; tick every five seconds

(on-mock-location-change location-f) → big-bang-handler

location-f : ([w world] [v view] [e event]? -> world)

Tells big-bang to update the world during simulated movement.

During the extent of a big-bang, a form widget will appear in the document.body to allow us to manually send location-changing events.

The optional event argument will contain numbers for "latitude" and "longitude".

...
;; move: world view event -> world
(define (move world dom event)
  (list (event-ref event "latitude")
        (event-ref event "longitude")))
...
(big-bang ...
          (on-mock-location-change move))

(on-location-change location-f) → big-bang-handler

location-f : ([w world] [v view] [e event]? -> world)

Tells big-bang to update when the location changes, as received by the Geolocation API.

The optional event argument will contain numbers for "latitude" and "longitude".

...
;; move: world view event -> world
(define (move world dom event)
  (list (event-ref event "latitude")
        (event-ref event "longitude")))
...
(big-bang ...
          (on-location-change move))

(to-draw draw-f) → big-bang-handler

draw-f : ([w world] [v view] -> view)

Tells big-bang how to update the rendering of the world. The draw function will be called every time an event occurs.

...
(define-struct world (name age))

;; draw: world view -> view
(define (draw world dom)
  (update-view-text (view-focus dom "name-span")
                    (world-name world)))
...
(big-bang ...
          (to-draw draw))

5.2 Views

A view is a functional representation of the browser DOM tree. A view is always focused on an element, and the functions in this subsection show how to traverse and manipulate the view.

(->view x) → view

x : any

Coerse a value into a view whose focus is on the topmost element. Common values for x include resources.

(view-focus? v id) → boolean

v : view

id : String

Return true if the view can be focused using the given id.

(view-focus v id) → view

v : view

id : String

Focuses the view on an element, given the id. The view will be searched starting from the toplevelmost node.

(view-left? v) → boolean

v : view

See if the view can be moved to the previous sibling.

(view-left v) → view

v : view

Move the focus to the previous sibling.

(view-right? v) → boolean

v : view

See if the view can be moved to the next sibling.

(view-right v) → view

v : view

Move the focus to the next sibling.

(view-up? v) → boolean

v : view

See if the view can be moved to the parent.

(view-up v) → view

v : view

Move the focus to the parent.

(view-down? v) → boolean

v : view

See if the view can be moved to the first child.

(view-down v) → view

v : view

Move the view to the first child.

(view-forward? v) → boolean

v : view

See if the view can be moved forward.

(view-forward v) → view

v : view

Move the view forward, assuming a pre-order traversal.

(view-backward? v) → boolean

v : view

See if the view can be moved backward.

(view-backward v) → view

v : view

Move the view backward, assuming a pre-order traversal.

(view-text v) → string

v : view

Get the textual content at the focus.

(update-view-text v s) → view

v : view

s : string

Update the textual content at the focus.

(view-bind v type world-updater) → view

v : view

type : string

world-updater : ([w world] [v view] [e event]? -> world)

Attach a world-updating event to the focus.

Attach a world-updating event to the focus. When the world-updater is called, the view will be focused on the element that triggered the event.

Common event types include "click", "mouseenter", "change".

(view-show v) → view

v : view

Show the element at the focus.

(view-hide v) → view

v : view

Hide the element at the focus.

(view-attr v name) → view

v : view

name : String

Get the attribute name at the focus.

(update-view-attr v name value) → view

v : view

name : String

value : String

Update the attribute n with the value v at the focus.

(view-css v name) → view

v : view

name : String

Get the css value name at the focus.

(update-view-css v name value) → view

v : view

name : String

value : String

Update the css value n with the value v at the focus.

(view-id v) → world

v : view

Get the unique identifier of the node at the focus.

(view-form-value v) → view

v : view

Get the form value of the node at the focus.

(update-view-form-value v value) → view

v : view

value : String

Update the form value of the node at the focus.

Dom nodes can be created by using xexp->dom, which converts a xexp to a node, and attached to the view by using view-append-child, view-insert-left, and view-insert-right.

(view-append-child v d) → view

v : view

d : dom

Add the dom node d as the last child of the focused node. Focus moves to the inserted node.

(view-insert-left v d) → view

v : view

d : dom

Add the dom node d as the previous sibling of the focused node. Focus moves to the inserted node.

(view-insert-right v d) → view

v : view

d : dom

Add the dom node d as the next sibling of the focused node. Focus moves to the inserted node.

(view-remove v) → view

v : view

Remove the dom node at the focus from the view v. Focus tries to move to the right, if there’s a next sibling. If that fails, focus then moves to the left, if there’s a previous sibling. If that fails too, then focus moves to the parent.

5.3 Events

An event is a structure that holds name-value pairs. Whenever an event occurs in web-world, it may include some auxiliary information about the event. As a concrete example, location events from on-location-change and on-mock-location-change can send latitude and longitude values, as long as the world callback can accept the event as an argument.

(struct event (kvpairs)
#:extra-constructor-name make-event)
kvpairs : (listof (list symbol (or/c string number)))

(event-ref evt name) → value

evt : event?

name : (or/c symbol string)

Get an value from the event, given its name.

(event-keys evt) → (listof symbol)

evt : event?

Get an list of the event’s keys.

5.4 Dynamic DOM generation with xexps

We often need to dynamically inject new dom nodes into an existing view. As an example where the UI is entirely in code:

#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/web-world))

;; tick: world view -> world
(define (tick world view)
  (add1 world))

;; draw: world view -> view
(define (draw world view)
  (view-append-child view
                     (xexp->dom `(p "hello, can you see this? "
                                    ,(number->string world)))))

(big-bang 0 (initial-view
             (xexp->dom '(html (head) (body))))
            (on-tick tick 1)
            (to-draw draw))

Normally, we’ll want to do as much of the statics as possible with ".html" resources, but when nothing else will do, we can generate DOM nodes programmatically.

We can create new DOMs from an xexp, which is a s-expression representation for a DOM node. Here are examples of expressions that evaluate to xexps:

"hello world"

'(p "hello, this" "is an item")

(local [(define name "josh")]
`(p "hello" (i ,name)))

'(div (@ (id "my-div-0"))
(span "This is a span in a div"))

`(div (@ ,(fresh-id))
(span "This is another span in a div whose id is dynamically generated"))

More formally, a xexp is:

‹xexp›	::=	‹string›
	\|	‹symbol›
	\|	( ‹id› ‹xexp›* )
	\|	( ‹id› ( @ ‹key-value›* ) ‹xexp›* )
‹key-value›	::=	( ‹symbol› ‹string› )

To check to see if something is a xexp, use xexp?:

(xexp? x) → boolean

x : any

Return true if x is a xexp.

(xexp->dom an-xexp) → dom

an-xexp : xexp

Return a dom from the xexp.

When creating xexps, we may need to create unique ids for the nodes. The web-world library provides a fresh-id form to create these.

(fresh-id) → string

Return a string that can be used as a DOM node id.

We may also want to take a view and turn it back into an xexp.

(view->xexp a-view) → xexp

a-view : view

Coerses a view into a xexp.

5.5 Tips and tricks: Hiding standard output or directing it to an element

For a web-world program, output is normally done by using to-draw. However, side effecting functions, such as printf or display, are still available, and will append to document.body.

We may want to disable such printing or redirect it to a particular element on the page. For such purposes, use a combination of current-output-port and open-output-element to redirect the output of these side effect functions to somewhere else.

For example:

...
;; Redirect standard output to a div called "stdout-div".
(current-output-port (open-output-element "stdout-div"))
...
(big-bang ...
          (on-tick (lambda (world dom)
                     (printf "Tick!\n")
                     (add1 world)))
          ...)

All subsequent I/O side effects after the call to current-output-port will be written out to the stdout-div, which can be easily styled with display: none to hide it from normal browser display.

(open-output-element id) → output-port

id : string

Opens an output port that will be directed to write to the DOM element whose id is id. Note: writing to this port shouldn’t fail, even if the id does not currently exist on the page.

6 The JavaScript Foreign Function Interface

(require (planet dyoo/whalesong:1:3/js))

This needs to describe what hooks we’ve got from the JavaScript side of things.

In particular, we need to talk about the plt namespace constructed by the runtime, and the major, external bindings, like plt.runtime.invokeMains.

The contracts here are not quite right either. I want to use JQuery as the type in several of the bindings here, but don’t quite know how to teach Scribble about them yet.

(alert msg) → void

msg : string?

Displays an alert. Currently implemented using JavaScript’s alert function.

body : any/c

A JQuery-wrapped value representing the body of the DOM.

(call-method object method-name arg ...) → any/c

object : any/c

method-name : string?

arg : any/c

Calls the method of the given object, assuming object is a JavaScript value that supports that method call. The raw return value is passed back.

For example,

(call-method body "css" "background-color")

should return the css color of the body.

($ locator) → any/c

locator : any/c

Uses JQuery to construct or collect a set of DOM elements, as described in the JQuery documentation.

For example,

(call-method ($ "<h1>Hello World</h1>")
"appendTo"
body)

will construct a h1 header, and append it to the document body.

(in-javascript-context?) → boolean

Returns true if the running context supports JavaScript-specific functions.

(viewport-width) → number?

Can only be called in a JavaScript context.

Returns the width of the viewport.

(viewport-height) → number?

Can only be called in a JavaScript context.

Returns the height of the viewport.

7 Simple world programming

Whalesong provides a library to support writing functional I/O programs (A Functional I/O System). Here’s an example of such a world program:

[FIXME: embed a world program here.]

8 Internals

Please skip this section if you’re a regular user: this is really notes internal to Whalesong development, and is not relevant to most people.

These are notes that describe the internal details of the implementation, including the type map from Racket values to JavaScript values. It should also describe how to write FFI bindings, eventually.

8.1 Architecture

The basic idea is to reuse most of the Racket compiler infrastructure. We use the underlying Racket compiler to produce bytecode from Racket source; it also performs macro expansion and module-level optimizations for us. We parse that bytecode using the compiler/zo-parse collection to get an AST, compile that to an intermediate language, and finally assemble JavaScript.

AST IL
parse-bytecode.rkt -----> compiler.rkt ----> assembler.rkt

The IL is intended to be translated straightforwardly. We currently have an assembler to JavaScript "js-assembler/assemble.rkt", as well as a simulator in "simulator/simulator.rkt". The simulator allows us to test the compiler in a controlled environment.

8.2 parser/parse-bytecode.rkt

(We try to insulate against changes in the bytecode structure by using the version-case library to choose a bytecode parser based on the Racket version number. Add more content here as necessary...)

8.3 compiler/compiler.rkt

This translates the AST to the intermediate language. The compiler has its origins in the register compiler in Structure and Interpretation of Computer Programs with some significant modifications.

Since this is a stack machine, we don’t need any of the register-saving infrastructure in the original compiler. We also need to support slightly different linkage structures, since we want to support multiple value contexts. We’re trying to generate code that works effectively on a machine like the one described in http://plt.eecs.northwestern.edu/racket-machine/.

The intermediate language is defined in "il-structs.rkt", and a simulator for the IL in "simulator/simulator.rkt". See "tests/test-simulator.rkt" to see the simulator in action, and "tests/test-compiler.rkt" to see how the output of the compiler can be fed into the simulator.

The assumed machine is a stack machine with the following atomic registers:

val: value
proc: procedure
argcount: number of arguments

and two stack registers:

env: environment stack
control: control stack

8.4 js-assembler/assemble.rkt

The intent is to potentially support different back end generators for the IL. "js-assembler/assemble.rkt" provides a backend for JavaScript.

The JavaScript assembler plays a few tricks to make things like tail calls work:

Each basic block is translated to a function taking a MACHINE argument.
Every GOTO becomes a function call.
The head of each basic-blocked function checks to see if we should trampoline (http://en.wikipedia.org/wiki/Trampoline_(computers))
We support a limited form of computed jump by assigning an attribute to the function corresponding to a return point. See the code related to the LinkedLabel structure for details.

Otherwise, the assembler is fairly straightforward. It depends on library functions defined in "runtime-src/runtime.js". As soon as the compiler stabilizes, we will be pulling in the runtime library in Moby Scheme into this project. We are right in the middle of doing this, so expect a lot of flux here.

The assembled output distinguishes between Primitives and Closures. Primitives are only allowed to return single values back, and are not allowed to do any higher-order procedure calls. Closures, on the other hand, have full access to the machine, but they are responsible for calling the continuation and popping off their arguments when they’re finished.

8.5 Values

All values should support the following functions

plt.runtime.toDomNode(x, mode): produces a dom representation. mode can be either ’write’, ’display’, or ’print’
plt.runtime.equals(x, y): tests if two values are equal to each other

8.5.1 Numbers

Numbers are represented with the js-numbers JavaScript library. We re-exports it as a plt.baselib.numbers namespace which provides the numeric tower API.

Example uses of the plt.baselib.numbers library include:

Creating integers:
42
16
Creating big integers:
plt.baselib.numbers.makeBignum("29837419826")
Creating floats:
plt.baselib.numbers.makeFloat(3.1415)
Predicate for numbers:
plt.baselib.numbers.isSchemeNumber(42)
Adding two numbers together:
plt.baselib.numbers.add(42, plt.baselib.numbers.makeFloat(3.1415))
Converting a plt.baselib.numbers number back into native JavaScript floats:
plt.baselib.numbers.toFixnum(...)

Do all arithmetic using the functions in the plt.baselib.numbers namespace. One thing to also remember to do is apply plt.baselib.numbers.toFixnum to any native JavaScript function that expects numbers.

8.5.2 Pairs, NULL, and lists

Pairs can be constructed with plt.runtime.makePair(f, r). A pair is an object with first and rest attributes. They can be tested with plt.runtime.isPair();

The empty list value, plt.runtime.NULL, is a single, distinguished value, so compare it with ===.

Lists can be constructed with plt.runtime.makeList, which can take in multiple arguments. For example,

var aList = plt.runtime.makeList(3, 4, 5);

constructs a list of three numbers.

The predicate plt.runtime.isList(x) takes an argument x and reports if the value is chain of pairs, terminates by NULL. At the time of this writing, it does NOT check for cycles.

8.6 Vectors

Vectors can be constructed with plt.runtime.makeVector(x ...), which takes in any number of arguments. They can be tested with plt.runtime.isVector, and support the following methods and attributes:

ref(n): get the nth element
set(n, v): set the nth element with value v
length: the length of the vector

8.7 Strings

Immutable strings are represented as regular JavaScript strings.

Mutable strings haven’t been mapped yet.

8.8 VOID

The distinguished void value is plt.runtime.VOID; functions implemented in JavaScript that don’t have a useful return value should return plt.runtime.VOID.

8.9 Undefined

The undefined value is JavaScript’s undefined.

8.10 EOF

The eof object is plt.runtime.EOF

8.10.1 Boxes

Boxes can be constructed with plt.runtime.makeBox(x). They can be tested with plt.runtime.isBox(), and they support two methods:

box.get(): returns the value in the box

box.set(v): replaces the value in the box with v

8.10.2 Structures

structure types can be made with plt.runtime.makeStructureType. For example,

var Color = plt.runtime.makeStructureType(

'color', // name

false, // parent structure type

3, // required number of arguments

0, // number of automatically-filled fields

false, // OPTIONAL: the auto-v value

false // OPTIONAL: a guard procedure

);

makeStructuretype is meant to mimic the make-struct-type function in Racket. It produces a structure type value with the following methods:

constructor: create an instance of a structure type.
For example,
var aColor = Color.constructor(3, 4, 5);
creates an instance of the Color structure type.
predicate: test if a value is of the given structure type.
For example,
Color.predicate(aColor) --> true
Color.predicate("red") --> false
accessor: access a field of a structure.
For example,
var colorRed = function(x) { return Color.accessor(x, 0); };
var colorGreen = function(x) { return Color.accessor(x, 1); };
var colorBlue = function(x) { return Color.accessor(x, 2); };
mutator: mutate a field of a structure.
For example,
var setColorRed = function(x, v) { return Color.mutator(x, 0, v); };

In addition, it has a type whose prototype can be changed in order to add methods to an instance of a structure type. For example,

Color.type.prototype.toString = function() {

return "rgb(" + colorRed(this) + ", "

+ colorGreen(this) + ", "

+ colorBlue(this) + ")";

};

should add a toString method for instances of the Color structure.

8.11 Tests

The test suite in "tests/test-all.rkt" runs the test suite. You’ll need to run this on a system with a web browser, as the suite will evaluate JavaScript and make sure it is producing values. A bridge module in "tests/browser-evaluate.rkt" brings up a temporary web server that allows us to pass values between Racket and the JavaScript evaluator on the browser for testing output.

8.12 What’s in js-vm that’s missing from Whalesong?

(This section should describe what needs to get done next.)

The only types that are mapped so far are

immutable strings
numbers
pairs
null
void
vectors

We need to bring around the following types previously defined in js-vm: (This list will shrink as I get to the work!)

immutable vectors
regexp
byteRegexp
character
placeholder
path
bytes
immutable bytes
keywords
hash
hasheq
struct types
exceptions
thread cells
big bang info
worldConfig
effectType
renderEffectType
readerGraph

)

What are the list of primitives in "js-vm-primitives.js" that we haven’t yet exposed in whalesong? We’re missing 178:

abort-current-continuation
andmap
append
apply
argmax
argmin
arity-at-least-value
arity-at-least?
assoc
assq
assv
boolean=?
box-immutable
box?
build-list
build-string
build-vector
byte?
bytes
bytes->immutable-bytes
bytes->list
bytes-append
bytes-copy
bytes-fill!
bytes-length
bytes-ref
bytes-set!
bytes<?
bytes=?
bytes>?
bytes?
caaar
caadr
caar
cadar
cadddr
caddr
cadr
call-with-continuation-prompt
call-with-current-continuation
call-with-values
call/cc
cdaar
cdadr
cdar
cddar
cdddr
cddr
char->integer
char-alphabetic?
char-ci<=?
char-ci<?
char-ci=?
char-ci>=?
char-ci>?
char-lower-case?
char-numeric?
char-upper-case?
char-whitespace?
char<=?
char<?
char>=?
char>?
compose
cons?
continuation-mark-set->list
continuation-mark-set?
continuation-prompt-tag?
current-continuation-marks
current-inexact-milliseconds
current-seconds
default-continuation-prompt-tag
e
eighth
empty
empty?
eof
eof-object?
equal~?
exact->inexact
exn-continuation-marks
exn-message
exn:fail:contract:arity?
exn:fail:contract:divide-by-zero?
exn:fail:contract:variable?
exn:fail:contract?
exn:fail?
exn?
explode
false
false?
fifth
filter
first
foldl
foldr
for-each
fourth
hash-for-each
hash-map
hash-ref
hash-remove!
hash-set!
hash?
identity
immutable?
implode
inexact->exact
inexact?
int->string
integer->char
js-function?
js-object?
js-value?
length
list->bytes
list-tail
make-arity-at-least
make-bytes
make-continuation-prompt-tag
make-exn
make-exn:fail
make-exn:fail:contract
make-exn:fail:contract:arity
make-exn:fail:contract:divide-by-zero
make-exn:fail:contract:variable
make-hash
make-hasheq
make-struct-type
make-thread-cell
map
memf
memq
memv
null
ormap
pi
placeholder-get
posn?
print-values
quicksort
raise
remove
replicate
rest
second
seventh
sixth
sleep
sort
string->immutable-string
string->int
string-alphabetic?
string-copy
string-fill!
string-ith
string-lower-case?
string-numeric?
string-upper-case?
string-whitespace?
struct-accessor-procedure?
struct-constructor-procedure?
struct-mutator-procedure?
struct-predicate-procedure?
struct-type?
struct?
subbytes
symbol=?
third
throw-cond-exhausted-error
true
undefined?
values
vector?
verify-boolean-branch-value
void?
write
xml->s-exp

(I should catalog the bug list in GitHub, as well as the feature list, so I have a better idea of what’s needed to complete the project.)

(We also need a list of the primitives missing that prevent us from running racket/base; it’s actually a short list that I’ll be attacking once things stabilize.)

9 The Whalesong language

(require (planet dyoo/whalesong:1:3/lang/base))

This needs to at least show all the bindings available from the base language.

true : boolean

The boolean value #t.

false : boolean

The boolean value #f.

pi : number

The math constant pi.

e : number

The math constant pi.

null : null

The empty list value null.

(boolean? v) → boolean?

v : any/c

Returns true if v is #t or #f

(let/cc id body ...)

(null? ...)

(not ...)

(eq? ...)

(equal? ...)

(void ...)

(quote ...)

(quasiquote ...)

9.1 IO

(current-output-port ...)

(current-print ...)

(write ...)

(write-byte ...)

(display ...)

(newline ...)

(format ...)

(printf ...)

(fprintf ...)

(displayln ...)

9.2 Numeric operations

(number? ...)

(+ ...)

(- ...)

(* ...)

(/ ...)

(= ...)

(add1 ...)

(sub1 ...)

(< ...)

(<= ...)

(> ...)

(>= ...)

(abs ...)

(quotient ...)

(remainder ...)

(modulo ...)

(gcd ...)

(lcm ...)

(floor ...)

(ceiling ...)

(round ...)

(truncate ...)

(numerator ...)

(denominator ...)

(expt ...)

(exp ...)

(log ...)

(sin ...)

(sinh ...)

(cos ...)

(cosh ...)

(tan ...)

(asin ...)

(acos ...)

(atan ...)

(sqr ...)

(sqrt ...)

(integer-sqrt ...)

(sgn ...)

(make-rectangular ...)

(make-polar ...)

(real-part ...)

(imag-part ...)

(angle ...)

(magnitude ...)

(conjugate ...)

(string->number ...)

(number->string ...)

(random ...)

(exact? ...)

(integer? ...)

(zero? ...)

9.3 String operations

(string? s)

(string ...)

(string=? ...)

(string->symbol ...)

(string-length ...)

{}

(string-ref ...)

{}

(string-append ...)

{}

(string->list ...)

{}

(list->string ...)

{}

9.4 Character operations

(char? ch)

(char=? ...)

9.5 Symbol operations

(symbol? ...)

(symbol->string? ...)

9.6 List operations

(pair? ...)

(cons ...)

(car ...)

(cdr ...)

(list ...)

(length ...)

(append ...)

(reverse ...)

(map ...)

(for-each ...)

(member ...)

(list-ref ...)

(memq ...)

(assq ...)

9.7 Vector operations

(vector? ...)

(make-vector ...)

(vector ...)

(vector-length ...)

(vector-ref ...)

(vector-set! ...)

(vector->list ...)

(list->vector ...)

10 Acknowledgements

Whalesong uses code and utilities from the following external projects:

jshashtable (http://www.timdown.co.uk/jshashtable/)
js-numbers (http://github.com/dyoo/js-numbers/)
JSON (http://www.json.org/js.html)
jquery (http://jquery.com/)
Google Closure Compiler (http://code.google.com/p/closure-compiler/)

The following folks have helped tremendously in the implementation of Whalesong by implementing libraries, giving guidence, reporting bugs, and suggesting improvements.

Asumu Takikawa
Cristina Teodoropol
Doug Orleans
Emmanuel Schanzer
Eric Hanchrow
Ethan Cecchetti
Greg Hendershott
Gregor Kiczales
Jay McCarthy
Jens Axel Søgaard
Richard Cleis
Robby Findler
Sam Tobin-Hochstadt
Scott Newman
Shriram Krishnamurthi
Zhe Zhang

1	Introduction
2	Getting started
3	Using whalesong
4	Including external resources
5	The web-world API
6	The Java Script Foreign Function Interface
7	Simple world programming
8	Internals
9	The Whalesong language
10	Acknowledgements