1 Introduction
1.1 Demo
1.2 Simple Example
2 Terminal Diversity
2.1 Protocol
2.2 Key Encoding
2.2.1 Keylabel
2.2.2 Keycode
charterm-keycode?
2.2.3 Keyinfo
charterm-keyinfo?
charterm-keyinfo-keyset-id
charterm-keyinfo-bytelang
charterm-keyinfo-bytelist
charterm-keyinfo-keylabel
charterm-keyinfo-keycode
charterm-keyinfo-all-keycodes
2.2.4 Keyset
charterm-keyset?
charterm-keyset-id
charterm-ascii-keyset
charterm-dec-vt100-keyset
charterm-dec-vt220-keyset
charterm-screen-keyset
charterm-linux-keyset
charterm-xterm-x11r6-keyset
charterm-xterm-xfree86-keyset
charterm-xterm-new-keyset
charterm-rxvt-keyset
charterm-wyse-wy50-keyset
charterm-televideo-925-keyset
2.2.5 Keydec
charterm-keydec-id
charterm-vt100-keydec
charterm-vt220-keydec
charterm-screen-keydec
charterm-linux-keydec
charterm-xterm-new-keydec
charterm-xterm-keydec
charterm-rxvt-keydec
charterm-wy50-keydec
charterm-tvi925-keydec
charterm-ascii-keydec
charterm-ansi-keydec
charterm-insane-keydec
2.3 Termvar
3 charterm Object
charterm?
charterm-termvar
charterm-protocol
charterm-keydec
current-charterm
open-charterm
close-charterm
with-charterm
4 Terminal Information
charterm-screen-size
5 Display Control
5.1 Cursor
charterm-cursor
charterm-newline
5.2 Displaying
charterm-display
5.3 Video Attributes
charterm-normal
charterm-inverse
charterm-underline
charterm-blink
charterm-bold
5.4 Clearing
charterm-clear-screen
charterm-clear-line
charterm-clear-line-left
charterm-clear-line-right
5.5 Line Insert and Delete
charterm-insert-line
charterm-delete-line
5.5.1 Misc. Output
charterm-bell
6 Keyboard Input
charterm-byte-ready?
charterm-read-key
charterm-read-keyinfo
7 References
8 Known Issues
9 History
10 Legal
3:1

CharTerm: Character-cell Terminal Interface in Racket

Neil Van Dyke

 (require (planet neil/charterm:3:1))

1 Introduction

The CharTerm package provides a Racket interface for character-cell video display terminals on Unix-like systems – such as for GNU Screen and tmux sessions on cloud servers, XTerm windows on a workstation desktop, and some older hardware terminals (even the venerable DEC VT100). Currently, it implements a subset of features available on most terminals.
This package could be used to implement a status/management console for a Racket-based server process (perhaps run in GNU Screen or tmux on a server machine, to be detached and reattached from SSH sessions), a lightweight user interface for a systems tool, a command-line REPL, a text editor, creative retro uses of old equipment, and, perhaps most importantly, a Rogue-like application.
The CharTerm package does not include any native code (such as from terminfo, termcap, curses, or ncurses) in the Racket process, such as through the Racket FFI or C extensions, so there is less potential for a problem involving native code to threaten the reliability or security of a program. CharTerm is implemented in pure Racket code except for executing /bin/stty for some purposes. Specifically, /bin/stty at startup time and shutdown time, to set modes, and (for terminal types that don’t seem to support a screen size report control sequence) when getting screen size. Besides security and stability, lower dependence on native code might also simplify porting to host platforms that don’t have those native code facilities.

1.1 Demo

For a demonstration, the following command, run from a terminal, should install the CharTerm package (if not already installed), and run the demo:
  racket -pm neil/charterm/demo
This demo reports what keys you pressed, while letting you edit a text field, and while displaying a clock. The clock is updated roughly once per second, and is not updated during heavy keyboard input, such as when typing fast. The demo responds to changing terminal sizes, such as when an XTerm is window is resized. It also displays the determined terminal size, and some small tests of the #:width argument to charterm-display. Exit the demo by pressing the Esc key.
Note: Although this demo includes an editable text field, as proof of concept, the current version of CharTerm does not provide editable text fields as reusable functionality.

1.2 Simple Example

Here’s your first CharTerm program:
#lang racket/base
 
(require (planet neil/charterm))
 
(with-charterm
 (charterm-clear-screen)
 (charterm-cursor 10 5)
 (charterm-display "Hello, ")
 (charterm-bold)
 (charterm-display "you")
 (charterm-normal)
 (charterm-display ".")
 (charterm-cursor 1 1)
 (charterm-display "Press a key...")
 (let ((key (charterm-read-key)))
   (charterm-cursor 1 1)
   (charterm-clear-line)
   (printf "You pressed: ~S\r\n" key)))
Now you’re living the dream of the ’70s.

2 Terminal Diversity

Like people, few terminals are exactly the same.
Some key (ha) terms (ha) used by CharTerm are:
  • termvar a string value like from the Unix-like TERM environment variable, used to determine a default protocol and keydec.

  • protocol how to control the display, query for information, etc.

  • keydec how to decode key encodings of a particular terminal. A keydec is constructed from one or more keysets, can produce keycodes or keyinfos.

  • keyset a specification of encoding some of the keys in a particular terminal, including keylabels and keycodes.

  • keylabel a string for how a key is likely labeled on a keyboard, such as the DEC VT100 PF1 key would have a keylabel "PF1" for a keycode 'f1.

  • keycode a value produced by a decoded key, such as a character for normal printable keys, like #\a and #\space, a symbol for some recognized unprintable keys, like 'escape and 'f1, or possibly a number for unrecognized keys.

  • keyinfo an object that is used like a keycode, except bundles together a keycode and a keylabel, as well as alternatate keycodes and information about how the key was decoded (e.g., from which keyset).

These terms are discussed in the following subsections.
CharTerm is developed with help of original documentation such as that curated by Paul Williams at vt100.net, various commentary found on the Web, observed behavior with modern software terminals like XTerm, various emulators for hardware terminals, and sometimes original hardware terminals. Thanks to Mark Pearrow for contributing a TeleVideo 950, and Paul McCabe for a Wyse S50 WinTerm.
At time of this writing, the author is looking to acquire a DEC VT525, circa 1994, for ongoing testing.
The author welcomes feedback on useful improvements to CharTerm’s support for terminal diversity (no pun). If you have a terminal that is sending an escape sequence not recognized by the demo, you can run the demo with the -n (aka --no-escape) argument to see the exact byte sequence:
  racket -pm- neil/charterm/demo -n
When -n is used, this will be indicated by the bottom-most scrolling line, rather than saying “To quit, press Esc.” instead will say “There is no escape from this demo.” You will have to kill the process through some other means.

2.1 Protocol

The first concept CharTerm has for distinguishing how to communicate with a terminal is what is what is called here protocol, which concerns everything except how keyboard keys are decoded. The following protocols are currently implemented:
  • ansi protocol Terminals approximating [ANSI X3.64], which is most terminals in use today, including software ones like XTerm. This protocol is the emphasis of this package; the other protocols are for unusual situations.

  • wyse-wy50 protocol Terminals compatible with the Wyse WY-50. This support is based on [WY-50-QRG], [WY-60-UG], [wy60], and [PowerTerm]. Note that video attributes are not supported, due to the WY-50’s model of having video attribute changes occupy character cells; you may wish to run the Wyse terminal in an ANSI or VT100 mode.

  • televideo-925 protocol Terminals compatible with the TeleVideo 925. This support is based on [TVI-925-IUG] and behavior of [PowerTerm]. Note that video attributes are not supported, due to the 925’s model of having video attribute changes occupy character cells; you may wish to run your TeleVideo terminal in ANSI or VT100 mode, if it has one.

  • ascii protocol Terminals that support ASCII but not much else that we know about.

2.2 Key Encoding

While most video display control, they seem to vary more by key encoding.
The CharTerm author was motivated to increase the sophistication of its keyboard handling after a series of revelations on the Sunday of the long weekend in which CharTerm was initially written. The first was discovering that four of the function keys that had been working fine in rxvt did not work in XTerm. Dave Gilbert somewhat demystified this by pointing out that the original VT100 had only four function keys, which set into motion an unfortunate series of bad decisions by various developers of terminal software to be needlessly incompatible with each other. After Googling, a horrifying 2005 Web post by Phil Gregory [Gregory], which showed that key encoding among XTerm variants was even worse than one could ever fear. Even if one already knew how much subtleties of old terminals varied (e.g., auto-newline behavior, whether an attribute change consumed a space, etc.), this incompatibility in newer software was surprising. Then, on a hunch, I tried the Linux Console on a Debian Squeeze machine, which surely is ANSI, and found, however, that it generated yet different byte sequences, for the first five (not four) function keys. Then I compared all to the [ECMA-48] standard, which turns out to be nigh-inscrutable, so which might help explain why everyone became so anti-social.
CharTerm now provides the abstractions of keysets and keydecs to deal with this diversity in a maintainable way.

2.2.1 Keylabel

A keylabel is a Racket string for how a key is likely labeled on a particular terminal’s keyboard. Different keyboards may have different keylabels for the same keycode. For example, a VT100 has a PF1 key (keylabel "PF1", keycode 'f1), while many other keyboards would label the key F1 (keylabel "F1", keycode 'f1). The keylabel currently is most useful for documenting and debugging, although it could later be used when giving instructions to the user, such as knowing whether to tell the user the Return key or the Enter key; the Backspace or the Rubout key; etc.

2.2.2 Keycode

A keycode is a value representing a key read from a terminal, which can be a Racket character, symbol, or number. Keys corresponding to printable characters have keycodes as Racket characters. Some keys corresponding to special non-printable characters can have keycodes of Racket symbols, such as 'return, 'f1, 'up, etc.

procedure

(charterm-keycode? x)  boolean?

  x : any/c
Predicate for whether or not x is a valid keycode.

2.2.3 Keyinfo

A keyinfo represents a keycode for a key, a keylabel, and how it is encoded as bytes. It is represented in Racket as a charterm-keyinfo object.

procedure

(charterm-keyinfo? x)  boolean?

  x : any/c
Predicate for whether or not x is a charterm-keyinfo object.

procedure

(charterm-keyinfo-keyset-id ki)  symbol?

  ki : charterm-keyinfo?
(charterm-keyinfo-bytelang ki)  string?
  ki : charterm-keyinfo?
(charterm-keyinfo-bytelist ki)  (listof byte?)
  ki : charterm-keyinfo?
(charterm-keyinfo-keylabel ki)  string?
  ki : charterm-keyinfo?
(charterm-keyinfo-keycode ki)  charterm-keycode?
  ki : charterm-keyinfo?
(charterm-keyinfo-all-keycodes ki)  (listof charterm-keycode?)
  ki : charterm-keyinfo?
Get information from a charterm-keyinfo object.

2.2.4 Keyset

A keyset is a specification of keys on a particular keyboard, including their keylabel, encoding as bytes, and primary and alternate keycodes.
The means of constructing a keyset is currently internal to this package.

procedure

(charterm-keyset? x)  boolean?

  x : any/c
Predicate for whether or not x is a keyset.

procedure

(charterm-keyset-id ks)  symbol?

  ks : charterm-keyset?
Get a symbol identifying the keyset.

value

charterm-ascii-keyset : charterm-keyset?

From the old [ASCII] standard. When defining a keydec, this is good to have as a final keyset, after the others.

value

charterm-dec-vt100-keyset : charterm-keyset?

From the DEC VT100. This currently defines the four function keys (labeled on the keyboard, PF1 through PF4) as 'f1 through 'f4, and the arrow keys. [VT100-UG] and [PowerTerm] were used as references.

value

charterm-dec-vt220-keyset : charterm-keyset?

From the DEC VT220. This currently defines function keys F1 through F20.

value

charterm-screen-keyset : charterm-keyset?

From the GNU Screen terminal multiplexer, according to [Gregory]. Also used by tmux.

value

charterm-linux-keyset : charterm-keyset?

From the Linux console. Currently defines function keys F1 through F5 only, since the rest will be inherited from other keysets.

value

charterm-xterm-x11r6-keyset : charterm-keyset?

From the XTerm in X11R6, according to [Gregory].

value

charterm-xterm-xfree86-keyset : charterm-keyset?

From the XFree86 XTerm, according to [Gregory].

value

charterm-xterm-new-keyset : charterm-keyset?

From the current xterm-new, often called simply xterm, as developed by Thomas E. Dickey, and documented in [XTerm-ctlseqs]. Several also came from decompiling a terminfo entry. Thanks to Dickey for his emailed help.

value

charterm-rxvt-keyset : charterm-keyset?

From the rxvt terminal emulator. These come from [Gregory], and currently define function keys 'f1 through 'f44.

value

charterm-wyse-wy50-keyset : charterm-keyset?

From the Wyse WY-50, based on [WY-50-QRG] and looking at photos of WY-50 keyboard, and tested in [wy60] and [PowerTerm]. The shifted function keys are provided as both 'shift-f1 through 'shift-16, and 'f17 through 'f31.

value

charterm-televideo-925-keyset : charterm-keyset?

From the TeleVideo 925, based on [TVI-925-IUG], [PowerTerm], and from looking at a TeleVideo 950 keyboard.

2.2.5 Keydec

A keydec object is a key decoder for a specific variety of terminal, such as for a specific termvar. A keydec is used to turn received key encodings from a terminal into keycode or keyinfo values. A keydec is constructed from a prioritized list of keyset objects, with earlier-listed keysets taking priority of later-listed keysets when there is conflict between them as to how to decode a particular byte sequence.

procedure

(charterm-keydec-id kd)  symbol?

  kd : charterm-keydec?
Gets the ID symbol of the keydec being used.
ANSI Keydecs

value

charterm-vt100-keydec : charterm-keydec?

Keydec for termvar "vt100".

value

charterm-vt220-keydec : charterm-keydec?

Keydec for termvar "vt220".

value

charterm-screen-keydec : charterm-keydec?

Keydec for termvar "screen".

value

charterm-linux-keydec : charterm-keydec?

Keydec for termvar "linux".

value

charterm-xterm-new-keydec : charterm-keydec?

Keydec for termvar "xterm-new".

value

charterm-xterm-keydec : charterm-keydec?

Keydec for termvar "xterm". Currently same as the keydec for xterm, except for a different ID.

value

charterm-rxvt-keydec : charterm-keydec?

Keydec for termvar "rxvt".
Wyse Keydecs

value

charterm-wy50-keydec : charterm-keydec?

Keydec for termvar "wy50".
TeleVideo Keydecs

value

charterm-tvi925-keydec : charterm-keydec?

Keydec for termvar "tvi925".
ASCII Keydecs

value

charterm-ascii-keydec : charterm-keydec?

Keydec for termvar "ascii".
Default Keydecs

value

charterm-ansi-keydec : charterm-keydec?

Keydec for any presumed ANSI-ish terminal, combining many ANSI-ish keysets.

value

charterm-insane-keydec : charterm-keydec?

Keydec for the uniquely desperate situation of wanting to possibly have extensive key decoding for a terminal that might not even be ansi, but be Wyse, TeleVideo, or some other ASCII.

2.3 Termvar

A termvar is what the charterm package calls the value of the Unix-like TERM environment variable. Each termvar has a default protocol and keydec. Note, however, that TERM is not always a precise indicator of the best protocol and keydec, but by default we work with what we have.

3 charterm Object

The charterm object captures the state of a session with a particular terminal.
A charterm object is also a synchronizable event, so it can be used with procedures such as sync. As an event, it becomes ready when there is at least one byte available for reading from the terminal, and its synchronization result is itself.

procedure

(charterm? x)  boolean?

  x : any/c
Predicate for whether or not x is a charterm.

procedure

(charterm-termvar ct)  (or/c #f string?)

  ct : charterm?
Gets the termvar.

procedure

(charterm-protocol ct)  symbol?

  ct : charterm?
Gets the protocol.

procedure

(charterm-keydec ct)  symbol?

  ct : charterm?
Gets the keydec.

parameter

(current-charterm)  (or/c #f charterm?)

(current-charterm ct)  void?
  ct : (or/c #f charterm?)
This parameter provides the default charterm for most of the other procedures. It is usually set automatically by call-with-charterm, with-charterm, open-charterm, and close-charterm.

procedure

(open-charterm [#:tty tty    
  #:current? current?])  charterm?
  tty : (or/c #f path-string?) = #f
  current? : boolean? = #t
Returns an open charterm object, by opening I/O ports on the terminal device at tty (or, if #f, file "/dev/tty"), and setting raw mode and disabling echo (via "/bin/stty"). If current? is true, the current-charterm parameter is also set to this object.

procedure

(close-charterm [#:charterm ct])  void?

  ct : charterm? = (current-charterm)
Closes ct by closing the I/O ports, and undoing open-charterm’s changes via "/bin/stty". If current-charterm is set to ct, then that parameter will be changed to #f for good measure. You might wish to use with-charterm instead of worrying about calling close-charterm directly.
Note: If you exit your Racket process without properly closing the charterm, your terminal may be left in a crazy state. You can fix it with the command:
  stty sane

syntax

(with-charterm expr? ...)

Opens a charterm and evaluates the body expressions in sequence with current-charterm set appropriately. When control jumps out of the body, in a manner of speaking, the charterm is closed.

4 Terminal Information

procedure

(charterm-screen-size [#:charterm ct])

  
(or/c #f exact-nonnegative-integer?)
(or/c #f exact-nonnegative-integer?)
  ct : charterm? = (current-charterm)
Attempts to get the screen size, in character columns and rows. It may do this through a control sequence or through /bin/stty. If unable to get a value, then default of (80,24) is used.
The current behavior in this version of charterm is to adaptively try different methods of getting screen size, and to remember what worked for the next time this procedure is called for ct. For terminals that are identified as screen by the TERM environment variable (e.g., terminal emulators like GNU Screen and tmux), the current behavior is to not try the control sequence (which causes a 1-second delay waiting for a terminal response that never arrives), and to just use stty. For all other terminals, the control sequence is tried first, before trying stty. If neither the control sequence nor stty work, then neither method is tried again for ct, and instead the procedure always returns (#f, #f). This behavior very well might change in future versions of charterm, and the author welcomes feedback on which methods work with which terminals.

5 Display Control

5.1 Cursor

procedure

(charterm-cursor x y [#:charterm ct])  void?

  x : exact-positive-integer?
  y : exact-positive-integer?
  ct : charterm? = (current-charterm)
Positions the cursor at column x, row y, with the upper-left character cell being (1, 1).

procedure

(charterm-newline [#:charterm ct])  void?

  ct : charterm? = (current-charterm)
Sends a newline to the terminal. This is typically a CR-LF sequence.

5.2 Displaying

procedure

(charterm-display [#:charterm ct    
  #:width width    
  #:pad pad    
  #:truncate truncate]    
  arg ...)  void?
  ct : charterm? = (current-charterm)
  width : (or/c #f exact-positive-integer?) = #f
  pad : (or/c 'width boolean?) = 'width
  truncate : (or/c 'width boolean?) = 'width
  arg : any/c
Displays each arg on the terminal, as if formatted by display, with the exception that unprintable or non-ASCII characters might not be displayed. (The exact behavior of what is permitted is expected to change in a later version of CharTerm, so avoid trying to send your own control sequences or using newlines, making assumptions about non-ASCII characters, etc.)
If width is a number, then pad and truncate specify whether or not to pad with spaces or truncate the output, respectively, to width characters. When pad or width is 'width, that is a convenience meaning “true if, and only if, width is not #f.”

5.3 Video Attributes

procedure

(charterm-normal [#:charterm ct])  void?

  ct : charterm? = (current-charterm)
(charterm-inverse [#:charterm ct])  void?
  ct : charterm? = (current-charterm)
(charterm-underline [#:charterm ct])  void?
  ct : charterm? = (current-charterm)
(charterm-blink [#:charterm ct])  void?
  ct : charterm? = (current-charterm)
(charterm-bold [#:charterm ct])  void?
  ct : charterm? = (current-charterm)
Sets the video attributes for subsequent writes to the terminal. In this version of charterm, each is mutually-exclusive, so, for example, setting bold clears inverse. Note that that video attributes are currently supported only for protocol 'ansi, due to limitations of the TeleVideo and Wyse models for video attributes.

5.4 Clearing

procedure

(charterm-clear-screen [#:charterm ct])  void?

  ct : charterm? = (current-charterm)
Clears the screen, including first setting the video attributes to normal, and positioning the cursor at (1, 1).

procedure

(charterm-clear-line [#:charterm ct])  void?

  ct : charterm? = (current-charterm)
(charterm-clear-line-left [#:charterm ct])  void?
  ct : charterm? = (current-charterm)
(charterm-clear-line-right [#:charterm ct])  void?
  ct : charterm? = (current-charterm)
Clears text from the line with the cursor, or part of the line with the cursor.

5.5 Line Insert and Delete

procedure

(charterm-insert-line [count #:charterm ct])  void?

  count : exact-positive-integer? = 1
  ct : charterm? = (current-charterm)
Inserts count blank lines at cursor. Note that not all terminals support this.

procedure

(charterm-delete-line [count #:charterm ct])  void?

  count : exact-positive-integer? = 1
  ct : charterm? = (current-charterm)
Deletes count blank lines at cursor. Note that not all terminals support this.

5.5.1 Misc. Output

procedure

(charterm-bell [#:charterm ct])  void?

  ct : charterm? = (current-charterm)
Rings the terminal bell. This bell ringing might manifest as a beep, a flash of the screen, or nothing.

6 Keyboard Input

Normally you will get keyboard input using the charterm-read-key procedure.

procedure

(charterm-byte-ready? [#:charterm ct])  boolean?

  ct : charterm? = (current-charterm)
Returns true/false for whether at least one byte is ready for reading (either in a buffer or on the port) from ct. Note that, since some keys are encoded as multiple bytes, just because this procedure returns true doesn’t mean that charterm-read-key won’t block temporarily because it sees part of a potential multiple-byte key encoding.

procedure

(charterm-read-key [#:charterm ct    
  #:timeout timeout])  (or #f char? symbol?)
  ct : charterm? = (current-charterm)
  timeout : (or/c #f positive?) = #f
Reads a key from ct, blocking indefinitely or until sometime after timeout seconds has been reached, if timeout is non-#f. If timeout is reached, #f is returned.
Many keys are returned as characters, especially ones that correspond to printable characters. For example, the unshifted Q key is returned as character #\q. Some other keys are returned as symbols, such as 'return, 'escape, 'f1, 'shift-f12, 'right, and many others.
Since some keys are sent as ambiguous sequences, charterm-read-key employs separate timeouts internally, such as to disambuate the Esc key (byte sequence 27) from what on some terminals would be the F10 key (bytes sequence 27, 91, 50, 49, 126).

procedure

(charterm-read-keyinfo [#:charterm ct    
  #:timeout timeout])  charterm-keyinfo?
  ct : charterm? = (current-charterm)
  timeout : (or/c #f positive?) = #f
Like charterm-read-keyinfo except instead of returning a keycode, it returns a keyinfo.

7 References

[Gregory] Phil Gregory, “Terminal Function Key Escape Codes,” 2005-12-13 Web post, as viewed on 2012-06
[PowerTerm] Ericom PowerTerm InterConnect 8.2.0.1000 terminal emulator, as run on Wyse S50 WinTerm
[VT100-TM] Digital Equipment Corp., VT100 Series Technical Manual, 2nd Ed., 1980-09
[VT100-UG] Digital Equipment Corp., VT100 User Guide, 3rd Ed., 1981-06
[VT100-WP] Wikipedia, VT100
[XTerm-ctlseqs] Edward Moy, Stephen Gildea, Thomas Dickey, “Xterm Control Sequences,” 2012
[XTerm-FAQ] Thomas E. Dickey, “XTerm FAQ,” dated 2012
[XTerm-WP] Wikipedia, xterm

8 Known Issues

9 History

10 Legal

Copyright 2012 – 2013 Neil Van Dyke. This program is Free Software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See http://www.gnu.org/licenses/ for details. For other licenses and consulting, please contact the author.