Götterdämmerung

Pocket Scheme is hibernating

Last summer's projects never materialized, so nothing at all took place in this codebase. Witness the silence herein.

Realistically, I don't foresee anything happening in this codebase any time soon, which is a pity, as the guts of pscheme are still pretty embarrassing in many places. (Sure, it works. You just don't want to know how it works.) The problem is that I no longer use the CE platform, either personally or professionally.

Of my projects that are currently under active development^*, one of them-- yes, yet another Scheme implementation, on another platform-- could conceivably donate some R5RS- and R6RS-enabling technology to Pocket Scheme, were I ever to resume developing this. So I'm hesitant to come out and declare Pocket Scheme as completely dead. Too, one of last summer's projects might possibly return to life, in which case I'd find myself again hacking CE. Otherwise, though, I should publicly declare this project as frozen. If some other crazy person wanted to take it over, they'd have my blessing as well as all of the help that I could offer.

^*(Not being coy. I just don't care to hype things that don't yet work. At an appropriate time I'll post something here.)

I come to Squeak

In order to bootstrap a new project on the OLPC XO-1, I find myself learning a new environment: Squeak, a descendant of Smalltalk-80. On the XO-1, Squeak (in its Etoys guise) seems a nice fit. On my development system, however, I find working within Squeak very frustrating, with its idiosyncratic and uneven keyboard support, dependency on pointing devices, and complete lack of integration with the native desktop. Gradually, I am locating tools such as Win32NativeFonts (map) to stave off blindness, and SVI (map) (aka Emin) for Emacs-style keyboard bindings. And gradually, my frustration level is diminishing....

Note that SVI/Emin contains a conflict with squeak-dev (a popular Squeak image, preconfigured for development) as released. To run Emin in this image, disable the duplicateAllControlAndAltKeys preference (category keyboard).

der Schilderwald

Two days ago, an article on the front page of the Wall Street Journal captivated me with an arresting image:

My highways' signs warn against the road hazards of deer and cattle, but neglect the plight of the local toads and rattlesnakes, both of which lie too low to the ground to cause serious vehicle damage when striking them. The Germans, however, apparently cherish their toads, and for that I salute them.

Too, Krötenwanderung is a perfectly marvelous word. It brings to my mind what Chaucer might have written:

Whan that Aprille, with hise shoures soote,
The droghte of March hath perced to the roote
[...]
Thanne longen todes to goon on pilgrimages

at the opening of his A Tode's Book of the Tales of Caunterbury. (But no! Instead he slandered the lowly yet noble toad, in the hectoring voice of his Parson's "myrie tale in prose": Afterward speke we of scornynge, which is a wikked synne, and namely whan he scorneth a man for his goode werkes. For certes, swiche scorneres faren lyk the foule tode, that may nat endure to smelle the soote savour of the vyne whanne it florissheth. Boo, Chaucer!)

In the wintertime, a ski trail passes through a corner of my property, just behind my shop. Clearly, any skiers traversing my back yard must be warned against the perils of midwinter toad pilgrimages, lest warts foul their wax. However, the best sign I've found for sale lacks the charm of the model featured in the WSJ article. There is just something so charming about those orderly Teutonic toads proceeding in single file (im Gänsemarsch, sozusagen, od. Krötenmarsch? which would be nothing at all like a frog-march), and the verrie rustick hyphen, and the exclamation point! denoting the excitement of toadular promenades.

There's probably a special place in Hell being kept warm for tourists who steal essential public safety signs such as this. I must keep that in mind as I pursue the right sign.

Hygienic Cat Macros

(From the never-to-be-created icanhascheezmeburger.com.)

Smartphone support

I've received enough reports of Pocket Scheme working on various WM-powered smartphones to decide to claim Smartphone support. I do this with some trepidation, since I have no Smartphone test coverage other than these anecdotal reports. If a class of Smartphone-only bugs emerges, it's going to be just as much fun as debugging Japanese-only memory manager flaws was in 1999....

Planning a 1.4 release

Despite the "2.0: currently under development" claim on the front page, no development has taken place on Pocket Scheme since November 2006, the date of the last posting here. (Note to self: update front page.) And now I find myself needing some new features in the summer timeframe for a new project. So the 1.x codebase will receive yet another facelift.

In for certain:

Fix notification and SIP management bugs. In 1.3 I added a semi-nifty feature whereby if Pocket Scheme finishes a lengthy computation while in the background, it notifies the user with a foreground popup. Immediately afterwards in 1.3.1 I apparently broke this feature while trying to tame the unruly SIP with some entirely too clever (read: stupid) logic. There are plenty of other ways to confuse my overly clever code, too. I think I can back off on the SIP management and recover my notification feature without having to rewrite too much window management logic.
Add simple dialog input interface, supplanting the stub dialog creation procedures in w32.dll. Currently I have both child-pane and fullscreen modal dialog support planned. Modeless dialogs (e.g. the floating "Find" command window as seen in many text editors) are not supported.
Add simple graphical output interface, supplanting the stub window creation procedures in w32.dll. Nothing fancy-- no child windows, no user input-- just enough to draw a couple of graphs on the screen.

Unlike kludgy windows created with the current FFI, which operate in the Scheme execution thread, these dialog and canvas operations work within the main UI thread. Hence they don't confuse window activation or invite busy-locking the PPC.

The new project needs other long overdue Pocket Scheme features, but I'm not confident that I can get enough test coverage on them to put them into a public release. We'll see.

Direct-threaded Interpreter

I've completed a first cut at a proof of concept for a direct-threaded interpreter, implementing a amazing language capable of generating any list of the digits 1, 2, and 3. (Just look out, world. Next week: the digit 4!) One version of the interpreter runs on x86, where I could use inline assembly in my C compiler to manipulate the callstack while leaving the majority of the implementation in C: call this version the proof of concept of a proof of concept. The other version runs on my target processor, the ARM, where my compiler doesn't support any sort of inline assembly; there, the entire interpreter core is in assembly, calling back into the C-language pscheme kernel for consing and so forth. This offers a strong motivation to keep the VM small.

In the VM I described a couple of weeks ago, the FRAME and CONT registers can be unified, CONT being just a couple more fields in the frame. I was led astray by not originally realizing that even the topmost computation has a continuation, and that my VM should make that outermost continuation explicit. Oddly, I realized this immediately upon waking one morning.

The 2.0 VM

I've spent the last couple of weeks hand-compiling Scheme programs, working out the details of the VM for the next release. I've ended up with a VM strongly reminiscent of the CEK abstract machine.

While this is a work in progress, it'll do me good to write it up here.

Registers

CV: The current code vector. All branch operations (JUMP, JUMPFALSE) are relative within this.
IP: The instruction pointer, an offset within CV denoting the next instruction to decode and execute. Think of this as R15.
DV: The current data vector, a pool of read-only values. All literal data references (LITERAL) are relative within this.
VAL: The evaluated value of the last expression (in Scheme terms). This is used as a kind of general accumulator, a R0 type of register.
ENV: The top of a chain of frames implementing the current lexical environment.
FRAME: One or more partial calculations in progress within the current environment, implemented as a chain of partially defined frames. Taken together with ENV, this is R13 (though we're not a stack machine).
CONT: The top of a chain of records tracking the continuations of pending calls. Each record comprises an IP, a saved ENV and FRAME, and a link to the next record. Think of this as R14. This would be the callstack, were it not for call-with-current-continuation.
One advantage of keeping the activation chain separate from the process callstack is having Windows system calls interoperate nicely with reentrant continuations. In Pocket Scheme 1.x, FFI wndprocs and the interpreter share a common stack, which means that any windows running in the Scheme thread (i.e. any windows with a wndproc written in Scheme) must take care never to reenter themselves accidentally.

Instructions

FRAME size

Creates a new frame capable of storing size values, linking it to the existing value of FRAME, then setting FRAME to this new frame.

A frame is a multipurpose structure, responsible for saving the values of local variables, saving temporary computation results, and finally passing a set of parameters to a procedure. A frame starts life with a FRAME instruction, and has its elements set with FRAMESET instructions. It may stay in FRAME for a while, possibly as the tail of a chain of frames as created by subsequent FRAME operations, but eventually it will move into the ENV chain via PUSHENV, CALL, or TAILCALL. A frame ends its life via RETURN or POPENV, both of which immediately recycle the frame for subsequent FRAME calls, or a CALL or TAILCALL that makes it unreachable.

A frame is recycled by returning it to a pool from which subsequent FRAME instructions will reallocate it. Operations that break the stack access pattern of a frame will mark it as nonrecyclable, in which case the frame does not return to the pool, but instead is left for eventual GC. The CLOSURE instruction will mark every frame in the current ENV as nonrecyclable. Likewise, the procedure call-with-current-continuation will mark every frame in both ENV and CONT as nonrecyclable, so that those frames last for the life of the reified continuation.

For this reason, the compiler attempts to elide CLOSURE operations wherever possible (e.g. tail-calls to procedures defined via let and letrec).

FRAMESET index

Sets an element of the topmost frame in FRAME to the current contents of VAL. Used for apply and let, but not letrec.

CALL

Applies the closure in VAL to the arguments in the topmost frame. The current ENV, the remainder of the partial FRAME chain, and IP+1 are all saved in a new record linked to the top of CONT, from where they will be restored by the next RETURN operation. ENV is set to the environment saved in the closure, then extended by the topmost frame. FRAME is cleared. IP is set to the first instruction in the closure's code.

RETURN

Sets IP, FRAME, and ENV to the last values saved by CALL, popping the topmost element of CONT. The topmost frame of the previous ENV is immediately recycled, if possible.

Issue: if VAR-ARITY has extended the environment, RETURN should recycle two frames.

TAILCALL

Applies the closure in VAL to the arguments in the topmost frame, using tail-position semantics. ENV is set to the environment saved in the closure, then extended by the frame in FRAME, which should be the only frame in the chain. IP is set to the first instruction in the closure's code. FRAME is cleared. No new record is added to CONT.

PUSHENV

Transfers the topmost frame from FRAMES to ENV, extending the lexical environment with the values stored therein. If every element in the frame has been set, this is a Scheme let; otherwise, it is a Scheme letrec.

POPENV

Removes the topmost frame from ENV, immediately recycling it if possible.

JUMP offset

Adjusts the IP by the specified offset.

I always write JUMP with a symbolic offset, just as I do JUMPFALSE and CLOSURE. A LABEL pseudo-instruction gives a name to the target.

JUMPFALSE offset

If the contents of VAL are false, adjust the IP by the specified offset. Otherwise, execution will continue at IP+1.

CLOSURE offset

Creates a new closure object, saving the current ENV, the current CS, and the current IP, adjusted by the specified offset. VAL is set to the new closure.

Closing over the environment marks every frame in the current environment as nonrecyclable.

LITERAL offset

Loads VAL with the value found in DV at the specified offset.

I always write LITERAL with the explicit datum, just as I do GLOBDEF, GLOBSET, and GLOBREF.

GLOBDEF offset

Defines a new entity in the global namespace, named by the symbol found in DV at the specified offset, and taking the current VAL.

GLOBSET offset

Sets the global element named by the symbol found in DV at the specified offset to the current VAL.

GLOBREF offset

Sets VAL to the value of the global element named by the symbol found in DV at the specified offset.

LEXSET depth offset

Sets the element in the current environment specified by the depth and offset (de Bruijn indices) to the current VAL.

LEXREF depth offset

Sets VAL to the element in the current environment (ENV) specified by the depth and offset.

FIXED-ARITY arity

Saves meta-information about a procedure. Currently, tests topmost frame in ENV (as applied by CALL or TAILCALL) to compare its length to the specified arity. Raises a runtime error if the two quantities differ.

The intent is for all -ARITY instructions eventually to emit procedure metadata that is interpreted by CALL instead of performing the check inline. This would allow a better error message in the case of a mismatch in a tail call, allow a direct jump to the callsite for a case-lambda, and for calls to statically determined sites, would allow arity and type checks to be hoisted higher in the calltree.

MIN-ARITY arity

Saves meta-information about a procedure. Currently, tests topmost frame in ENV (as applied by CALL or TAILCALL) to compare its length to the specified minimum arity. Raises a runtime error if the specified arity is greater than the number of parameters passed.

VAR-ARITY offset

Skips the first offset parameters, copying the remaining passed parameters from the topmost frame in ENV into a fresh list. Extends ENV with a new frame containing this new list value. Raises a runtime error if the offset is greater than the number of parameters passed.

1.3.2 released

Happy autumn equinox! I'm back.

1.3.2 publishes the registry access routines that I described last May, and corrects the bugs reported against 1.3.1, including a couple of problems in writing non-Unicode text output that have slept undetected since 1.2's release last year. (Thank you, George.)

1.3.x still doesn't support first-generation Pocket PC devices, which I suspect will now join Handheld PCs as an abandoned platform. This means that I no longer have test coverage for grayscale displays and non-ARM processors. (Actually, the Scheme engine itself works fine on old devices, but it never displays any output. Fixing this would involve experimenting with the HTML viewer on the PPC 2000 platform until it does the right thing there— and yet doesn't break every other platform's output.)

Summer break

Absolutely nothing has happened on the pscheme front for the last two months, and I foresee nothing else happening for quite a little while yet. The days are very long, and I am very busy.

We will return to life in the autumn, when I should have a chance to investigate how I broke PPC2000 support in 1.3, continue work on replacing my lame little Editor, and recover the progress I had made on a FFI to Microsoft's COM.

Have a good summer.

New registry glue

New in the forthcoming 1.3.2, scheduled for release in late June: registry access.

w32:reg-key root path → key
Opens a key in the registry. The root can be an object returned by another call to w32:reg-key, or one of the symbols local-machine, current-user, or classes, which correspond respectively to HKEY_LOCAL_MACHINE (HKLM), HKEY_CURRENT_USER (HKCU), and HKEY_CLASSES_ROOT (HKCR). The path is a string that uses backslashes to separate parent and child keys (e.g. "Software\\Goetter\\Pocket Scheme" in Scheme syntax). Returns a key object that can be passed to other registry routines.

w32:reg-key-add! key name
Adds a new subkey with the given name.

w32:reg-key-delete! key name
Removes the subkey with the given name.

w32:reg-key-names key → list of strings
Returns a list of the names of the subkeys of the given key.

w32:reg-value key name → value
Returns the value stored under the name in the key, or #f if no such value exists in the key. The type of the value will be a number, a string, a list of strings, or a raw vector, depending on whether the registry value contains type REG_DWORD, REG_SZ, REG_MULTI_SZ, or REG_BINARY.

w32:reg-value-names key → list of strings
Returns a list of the names of the values stored in the key.

w32:reg-value-alist key → association list
Returns an alist of every name and value stored in the key.

w32:reg-value-set! key name value
Saves the value under the given name in the key. Works like the converse of w32:reg-value, with the exception of REG_MULTI_SZ, which I lazily didn't implement.

w32:reg-value-delete! key name
Removes the value with the given name.

Monday

Run to gym: good! (It had better be, I hadn't run since the last Wednesday.) One hour lower body and core strength: mostly range of motion exercises for the hips, those having tightened from a weekend of burning slash piles and my continued absence from the kungfu club.

Curiously, I'm in less discomfort from the epic hauling and burning of Friday and Saturday than the Mazama to Seattle drive on Sunday evening. I was completely cooked on Sunday, but it was a comfortable cooked. Only on Monday was I in discomfort, and that in the usual long-drive mid-back band. There is a lesson in this, I am sure... if only I could discern it.

Tweaking Thunderbird

Dissatisfied with Outlook's IMAP handling, I switched my email client from Outlook to Thunderbird a year ago. Thunderbird had some warts on its UI, generally lacking the polish and user-experience attention that the Microsoft product had and still has, but I endured those warts in exchange for its better behavior over slow connections. Last December's 1.5 released addressed most of those warts, leaving me a fairly satisfied Thunderbird user.

I still correspond with a majority of Outlook users, however, and missed the ability to reply to a message in Outlook style. Configuring Thunderbird to top-post its replies (a configuration hidden in Account Settings instead of Options.Composition, where I would at least expect a default) wasn't enough: I wanted the full Microsoft Mail-style header, and didn't want to mangle the attached quoted message with quote marks. Just this morning, however, I discovered an extension that inserts the "correctly" formatted header, and another extension (by the same author) that I can configure to remove the quoting. Amusingly, this is almost exactly the opposite transformation that my old extension did for Exchange, long, long ago. My email client hacking days being long past, I am happy not to have to learn how to extend Thunderbird myself!

My other Thunderbird wishlist item is Emacs-compatible key bindings. Switching keyboard focus from an XEmacs window to a Thunderbird composition window and back is a jarring experience, with two sets of incompatible key bindings for accomplishing similar tasks. Real Emacs zombies would instead run a mail program written in Emacs, but that is both clearly insane (in terms of architecture) and ass-ugly (in terms of the affront to my eyeballs). For now, I am trying to use XKeymacs, a brute-force solution that translates Emacs keystroke sequences into their Windows/CUA equivalents. XKeymacs doesn't limit its scope to within editor windows, a feature that I may come to appreciate someday but for right now is driving a lot of tweaking, as I teach it that I want C-r to mean "Reply to selected message" instead of "Search backwards." (I am not so deeply brainwashed by Emacs that I expect list controls to work like Dired. Yet.) It's a very pretty utility, still.

My other, other Thunderbird wishlist item is address book sync support with the Pocket PC, but that one I may have to write myself. The good news is that Thunderbird already has Palm sync support, though it looks a bit fragile. At least it'd give me a roadmap of where to go and what to do, a PalmOS conduit being not too far architecturally from an ActiveSync desktop service provider. The bad news is that Mozilla in its great wisdom uses an incompatible clone of COM, XPCOM (as opposed to "MSCOM" in their parlance), which the service provider will have to use to access the Thunderbird address book even as it uses Microsoft COM to talk to ActiveSync. That wouldn't be so bad, except that writing to XPCOM entails getting into bed with the entire Mozilla build environment. Ugh. (Does XPCOM standalone still work? Five-year-old link. See also this skeleton app.) Also, XPCOM only works in-process. PalmSync uses an extension running within the Thunderbird image to access AB data, which it then sends cross-process to the conduit (running in a sync manager's image) via COM. Double ugh. See previous note about my email client hacking days being over.

Wednesday

Ran to the accountant's office to deliver a check for the Feds. Ran the hills back, walked the rest. Right plantar very tender on the trip out. Further hillwork indicated. Start intervals!

Tuesday

Run to gym. One hour upper-body and core strength. Shocking shoulder weakness. Walk home.

I'm writing this entry on Wednesday morning, and my abs are sore,

Monday

Run to gym. One hour lower-body and core strength. Am making progress with oblique and rotation strength! Walk home.

Other than the jog to the gym this morning, I haven't run since last Friday. And I haven't practiced the full form since Saturday a week ago. Bad.

Progress to 1.3.1

I had hoped to release 1.3.1 today, but I'm tired and out of time. Instead I'll offer a brief status report of what has turned out to be the memory manager sub-release.

The D-S-W bug fell easily. Pocket Scheme has enormous conses (hmmm... that sounds a bit rude), a misfeature which I look forward to correcting in 2.0. I've been using what otherwise would be wasted padding space to track various attributes of different types: closed state of ports, read-only state of strings, various object ownership flags, even variable lookup address memoization. Since vectors and continuations didn't yet use that space, I now use it to track the additional state needed for marking a variable-sized node (per the Thorelli variant of D-S-W, q.v. Jones and Lins Garbage Collection, p. 84) With this change, the mark phase no longer recurses, and hence no longer blows the Scheme thread's runtime stack.

The objects that we're marking occupy three separate heaps: the cons heap, of fixed-sized cons cells; a vector heap, for storing strings and vectors; and a continuation heap, for saving pieces of the runtime stack that embody the current continuation. Prior to 1.3.1, only the cons heap adhered to the configured (Tools - Configure - Memory) size limit, with the other two heaps being allowed to grow without bound. Furthermore, on hibernation we would release memory from the cons heap, but would never do anything with the vector or continuation heaps. Hence a deeply recursing program could hog all of the device's memory, then would refuse to release it gracefully, eventually necessitating that the system kill pscheme. Like the D-S-W recursion bug, this is a problem that only became manifest in 1.2, when such deep recursion became possible through runtime stack juggling.

Anyway, Pocket Scheme now abides fully by the configured heap size limit, which specifies the sum of the limits of the three types of heap. Furthermore, it releases memory from the continuation heap much more aggressively than before. There is some fun new UI with slider bars to configure the three heaps' share, too. (Ideally, this would auto-configure as the program runs, but at present I need to know at startup the limit of each particular heap. In 2.0 we'll make these heaps two or three subsections of a single size-constrained area.) Since we're now trying to run three heaps out of the space formerly hosting just one, an advertised-as-1-megabyte heap offers less capacity than it once did. I've bumped the minimum default heap to 2 megs for now, and have made a note to look into our stack consumption at some point... the last time I looked at it carefully was in 1999, and that was on a MIPS device.

Two work items remain before I can release 1.3.1. Presently I accumulate data for the transcript window in a temporary file \Temp\Scheme Transcript.htm. On WM5, unlike previous releases, the entire filesystem is hosted in flash NVRAM, which is both relatively slow to write and has a limited number of writes in its lifetime. So in the interest of minimizing wear and tear to the PDA, I'm moving the transcript into a true RAM buffer (which is where the temporary file would have been hosted, pre WM5). Also, while in the garbage collector I noticed some dodgy assumptions about how we mark "extension" objects (e.g. the windows from w32, or the connections from tcp), which I need to review while GC is still paged into my mind, so to speak.

There's a lot of talk about the "2.0" vaporscheme in this entry. Partly driven by distaste of pscheme's current performance, partly as a procrastination tactic to avoid actually grovelling the pointer-reversing graph traversal code, I sat down and hacked out much of the object layout for the 2.0 release, reducing conses to 8 bytes apiece. I can't implement any of this until 2.0, however, because it depends on my reworking the evaluator extensively. Oh, speed the day.

Friday

Run up the county road to the Sitts place just beyond Gate Creek and back. Felt sluggish at the start and didn't really want to run, so I took with me an Axim loaded with a CF of some Metallica - an old motivational tool of mine from a few years back that I stopped using after one surprise encounter with a bear on a trail encouraged me to pay more attention. I had forgotten how good "Am I Evil" (and its tempo changes) is for running.

Wednesday

Ran along county road to the Fawn Creek trailhead and back. Still plenty of snow on the trail: it might yet be skiiable.

Tuesday

Run along county road and highway to Freestone junction and back. Nice not to have any traffic on the highway.

1.3 released

There is so much left to do, but this must suffice. It works. I have defeated the wily code generator of my C compiler, which cheerfully reorders elements on the runtime stack so as to frustrate my garbage collector. Yea, verily, it works.

By dint of immense strength of will I left the achingly lame Pocket Scheme Editor alone. I'll fix that when I rewrite it in the next release. Similar strength of will kept my code generic to WM2003. There is nothing in 1.3 that expects even WM2003SE, let alone WM5. Again, for the next release. And I left out the debugging support that I've been fussing with over the winter, and I didn't dive into the D-S-W bug, and, and, and ....

Smart Mode's pretty nice. A few more ideas for it

differentiate quote and backquote based on the presence of a comma (backquote being difficult to enter in HR mode)
replace curly brackets with < > instead - another char difficult to enter in HR mode
hack the WM5 "shell input context" mechanism to get better token separation

Now to catch up on other parts of my life.

Saturday

Run to Chinatown. Two hours taiji: good! Watch how far the right foot turns out while cat-walking. Walk home.

Thursday

Five laps with the dead. Barely escaped being locked in for the night.

The soreness from my preceding days of training and gluttony is abating.

Wednesday

Four laps with the dead (cut short by the gaster).

Tuesday

Bike to gym. One hour upper-body strength, hindered by lower-abdominal bloating from the previous evening's repast. The return bike is challenging due to sore hamstrings from yesterday's workout.

Drive to kung-fu, thanks to the same leg soreness. Two hours taiji.

Monday

Jog to gym. One hour lower-body strength. I have got to do something about the music in the gym— had to walk down the block and punch out a telephone pole (which must have looked pretty stupid, if anybody saw that display) to discharge my annoyance. Gene handled my tantrum with poise and quite tolerantly.

Later, four laps with the dead in the afternoon spring sunshine. After the midday strength work, I felt like just another fulltime resident.

Biceps still store from Saturday's (unlogged) work there.

Ate a two-pound beefsteak at dinner. Tuesday's workouts should be interesting, assuming that I can move at all.

The Iota discovery

It took me a day to locate a suitably burly debugger, install it, and build pscheme with it. This entailed installing an almost-expired beta of VS 2005 (which, incidentally, looks like a very nice product—I'll save my pennies for a copy), then installing a matching WM5 SDK, and then endless fussing: first to get pscheme to build; then to get the debugger talking to the device, working around bugs in the beta; then to make the iota bug repro; and finally to capture the bug. A productive 24 hours, really, in the same sense than running ten miles on a stationary treadmill is running a great distance.

Turns out that I didn't break anything: the bug's been there for the last year, since I released 1.2. (Longer, really. I wrote the code in 2003.) And the underlying flaw in the garbage collector that creates the bug has been there since the last millenium.

The pscheme garbage collector uses the pointer-reversing D-S-W (Deutsch-Schorr-Waite) algorithm to perform the mark phase of a mark-sweep in a constrained amout of stack. (This design originated from running on a very old version of Windows CE that limited its applications to 58Kb of staack.) However, I was lazy on that day in 1998 and didn't D-S-W when marking vectors (and continuations, which in pscheme terms are implemented as a sort of degenerate vector), simply recursing over each element instead. Also, since 1.2 pscheme has used chained continuations to allow it to evaluate arbitrarily complex expressions in a constrained runtime stack: whenever it exhausts its runtime stack, it saves the current continuation, then resets the stack, restoring the previous continuation as necessary to resume computation from that point. Finally, debug builds drastically reduce the size of these stack segments in order to exercise the chaining code.

Taken all together, a call to Iota on a debug build generates a large number of stack-segment continuations. And those continuations contain references to other continuations in the stack, which in turn refer to others, and so on, and so on. The garbage collector calls itself recursively when marking this sequence. And the garbage collector is the only part of pscheme that recurses without chaining continuations. So eventually a few thousand stack segments build up, and pscheme simple-mindedly tries to mark them recursively. Boom, stack fault.

The right fix for this is to finish the job that I started in 1998 and do the D-S-W pointer reversing trick within vectors and continuations. That, however, may have to wait until after 1.3. Short term, I'll turn off the debug-mode stack-swap exercising code that was creating a new continuation with every Kb of runtime stack, and I'll add some extra stack discipline to the mark phase so that it fails more gracefully in the face of (iota 0 1000000) or whatever.

Speaking of stack discipline, here's a happy discovery: WM5 lets exception handlers capture stack faults! In previous releases of Windows CE, any stack fault in a thread would terminate the entire application, exception handling notwithstanding.

Saturday

Eight laps with the dead, plus a side excursion to play tour guide at Bruce Lee's gravesite.

It never fails. When I see an automobile with out-of-state plates slowing to meet me, and the passenger window lowering on the approach, I know who I'll see within, and what they'll ask of me. The occupants: inevitably, young men of some Chinese heritage. Typically the car is full of young men, but sometimes less so and with girlfriends or young wives instead. The question: "Can you show us where Bruce Lee is buried?" I enjoy these interactions. I feel like one of the gnarled trees in the cemetery. And the tourists, they are always so young! Or—am I old?

Oh, yeah, the run. Energy level good throughout, despite almost no breakfast. Mild discomfort in medial right knee, tempered by leveraging the glutes properly. Glutes didn't start to ache until the seventh lap. Incredibly, too, there is currently (six hours later) only minor heel discomfort. Am I finally healing that heel?

Featurizing 1.3

I can feel the undertow sucking me back to the Real World even as I type, so I'd better write this up before I drown. [Ed.: I fell asleep last night immediately after typing that.]

1.3 basically works. I've spent the last couple of days fiddling with the interface, trying to polish what passes for the Pocket Scheme user experience, as well as how we manage multiple script-type operations on a platform that does not provide a good generic way to do so. Those two goals have at times worked at cross purposes. For example, I revoked some fairly involved soft input panel manipulation when I found that it confused CE's cross-application SIP manipulation, especially when another instance of pscheme was running a background script.

Presently I'm looking at the following:

Tracking running scripts, and notifying the user when they complete. Recent versions of Windows Mobile offer what appear through the scant documentation to be some nice notification mechanisms, intended for "You have mail!" blurbs, that might assist here.
Fixing SIP manipulation again - in unrolling the last delta, it went from overly clever to obviously stupid.
And at some point in 1.3 I apparently managed to booger the pscheme core's stack juggler. Evaluating (iota 0 5000) now crashes the interpreter thread. This one's going to be a lot of fun to chase, since I don't have a working CE debugger right now, and the bug doesn't manifest itself on my console development platform.

After that, I have to finish coding DWIM, aka Input→Smart Mode, and add a couple more paren-balancing options to the context menu. My current notion of DWIM entails:

Replace two successive single quotes with double quotes. This common character recognition error is almost impossible to see in a proportional font.
Replace a b ... with (a b ...).
Replace (a b ...) ... with (begin (a b ...) ...), which will make the case where the second or later forms raise an error easier to resume. In fact, I should do this one even if the user disables Smart Mode.
Outside of string and character constants, replace the alternate left brackets [ and { with (, and likewise for their right-bracket complements. This is another common character recognition error.
Complete all pending right parens in the expression.
Add a space after every sequence of right parens. This will make the expression wordwrap prettily when echoed in the transcript.

And I want an explicit "Balance this S-expression" command, and for successive "Select S-expression" commands to increase their scope beyond the current selection. And I want a pet elephant, too.

Singing yourself to sleep

Or, how I exercised the hibernation code in the version of pscheme under development:

(load "w32.dll")
(define p (w32:foreign-procedure "coredll.dll" "PostMessageW" '(w32api bool handle dword dword)))
(p w32:*host-hwnd* #x03ff 0 0)

That magic number in the last line is the value of the (poorly named) WM_HIBERNATE message, broadcast by the system in low memory scenarios. I built a thunk to call the PostMessage API, used that thunk to simulate the broadcast of WM_HIBERNATE, then sat back and watched the fireworks.

Zzzzz....

A preliminary 1.3 release

For the brave and the bold, I have finished a preliminary cut at 1.3. To install it, start with an installation of 1.2.2 on a WM 2003, 2003SE, or WM5 device, then unzip this archive into your installation directory thereon.

Again, this version only supports WM 2003, WM 2003 SE, and WM 5.0.

The user interface has changed a fair bit, a result of both my accommodating limitations of the HTML viewer control and my attempting to improve the overall user experience. The changes are mostly obvious upon inspection.

Separate output and transcript windows.
Property sheet configuration UI.
Simpler memory configuration.
New View→Input, View→Output menus with new commands
History for the program input window (should help JACAL)

Not yet implemented:

View→Input→Smart Mode
Part of Edit→Select S-Expr (presently it works like the awkwardly located 1.2 Edit→Select→Parenthesis)
Edit→Balance S-Expr
Context menus
Script fixes
Notifications for long-running background tasks
Legacy hardware (the early-model Pocket PCs)

Not yet exercised:

Hibernation

Known bugs:

The transcript of (display 'something)(newline)(read-char) doesn't put echoed input on its own line.
View→Match Parenthesis doesn't really belong on the View menu. Edit, however, is getting crowded, the toolbar is full, and I don't want to stuff this command on a second level menu.

Known annoyances that 1.3 will have to endure:

The transcript window quickly flashes the first few items at its top with every update. Correcting this will require either replacing the html viewer control or else logging the transcript in a very different manner.
The transcript windows slows as the transcript lengthens.
The edit window moves a little bit when the SIP appears and disappears.

One solution to both the flashing transcript window and the moving edit window involves abandoning the TTY metaphor, i.e., the conceit that your entered expression and the system's response to it appear at the bottom of the screen, scrolling up to make room for new data as necessary. If instead new items appear at the top, with older data scrolling down, then the edit control could be at the top of the screen, where it wouldn't move on SIP state change, and the transcript window below it, not flashing on update because it now starts with the most recent entries. Two problems immediately occur to me, however. First, separating the editable input pane and the command buttons and keys forces the user's eyes to scan greater distances while editing, passing the noise in the transcript pane (though View→Input→Full Screen could help). Second, scrolling down doesn't work so well if the standard program output is interlaced with the transcript. That program output must appear top to bottom and left to right, in the order of the English language.

Tuesday

Bike to gym, 1 hr upper body and core strength. Bike home. Some left shoulder weakness. Midline of abs sore from yesterday.

Weakest move: the front-delt isolating lift. Shoulders overall weak.

Monday

1.5 mile run, 1 hr lower body strength. Gentle walk home in the sunshine.

Need to work more Bulgarian squats into my week. Currently hip extension limits me more than glute strength.

A training log

If I'm going to keep a [b]log, why not log training here, too? I haven't logged my training since 2003, which was the year that I ran the Cutthroat Classic. Now that I've healed sufficiently to resume running in earnest (he typed, hoping it's true), let's get serious (if not S.E.R.I.O.U.S.) and resume logging.

(In 2001, I logged all Mazama-side road- and trailwork on a truly excellent calendar from the Canadian women's Olympic cross-country ski team, which they issued only in that single year, alas. That calendar motivated my training in so many wonderful ways. It was a pleasure to log each day's entry, and a particular pleasure to move to each new month. Oh, yeah....)

Anyway, let this beautiful Sunday afternoon start my new countdown to the Cutthroat!

Seven laps with the dead. Achilles comfortable throughout. Mild right plantar tenderness post-run. Discomfort from fifth lap in medial mid and lower right knee. Energy high throughout.

1.3 works, sort of

Happy times! My initial cut at 1.3 is now REPLing merrily on WM 5.0.

Unfortunately, the control that I'm now using for per-REPL-pass output (output displayed in real time, as opposed to the static transcript of the entire session) doesn't have the same concurrency rigor as the control that I used in the last version. Cross-thread activations are confusing it something fierce. I have a number of possible solutions that I cooked up during yesterday's long commute, but they'll have to wait for tonight. Right now the sun is shining.

The new runtime

In the same sense that 1.2 was about changing the I/O model, 1.3 is about changing the runtime window hosting model. The existing hosting model has been in place since 1998, when it seemed simply the easiest way to get something running at the time; since then, it's been extended, patched, and twisted into something that I no longer wish to maintain. Fortunately, I now have the opportunity to replace that model: partly because the current model uses htmlctrl in a way incompatible with that control on WM 5.0, and partly because I'm dropping support for the Handheld PC platform, thus sparing me a lot of testing and a blizzard of #ifdef statements.

1.2, and indeed every version since 0.1.0, uses two threads of operation. The primary thread creates the user interface windows and receives notifications from the operating system of system memory exhaustion, application shutdown, and the like. A secondary thread interprets all Scheme code, leaving the primary thread free to respond to user and system actions in a timely manner even as an evaluation (let loop (forever) (loop)) runs. This dual-thread design makes Pocket Scheme a well-behaved application on the device.

To evaluate an expression, the user enters it into the lower pane of the application, then taps the Eval button. In 1.2, the UI thread takes the expression, echoes it to its output window, copies it to a common memory buffer, sets some flags to indicate that it's busy in an evaluation, then signals the Scheme thread, which has been asleep awaiting that signal. The Scheme thread makes a private copy of that data, then signals back to the UI thread that it may resume. The UI thread now runs in an "eval busy" mode, hiding the old eval input window, as the Scheme thread evaluates the submitted data as an s-expression. The Scheme thread emits output by sending text data to the appropriate output window. Should the Scheme thread need input from the user, it sends a notification to the UI thread, then goes to sleep awaiting a response; the UI thread receives the notification and changes its mode from "eval busy" to "input visible," displaying a new pane for user input. After the user enters data as requested, the UI thread signals the Scheme thread again, which in turn copies the data from the input window, then notifies the UI thread that it has finished with that data. Once Scheme has finished evaluating its expression, it prints the evaluated value to the output window, then goes back to sleep, notifying the UI thread that it has finished evaluation. The UI thread then moves from "eval busy" back into "eval ready" mode, displaying once again the eval input pane and awaiting a new expression from the user. What a modal mess.

Features of the 1.2 design (perhaps "design" is overstating it):

The output window is constantly updated with a transcript of activity, tty-style.
The UI thread drives the REPL.
The Scheme thread is idle between passes of the REPL.
There are multiple data paths between the threads.
There are multiple synchronization methods between the threads.

As I've said before, 1.2 does not work on WM 5.0 because of its reliance on the htmlctrl as an output window that it continually updates with new output. 1.3 fixes this by splitting this output function into two windows: a per-evaluation output window that displays any current output in real time, and a per-session transcript window that displays the results of all prior computations, updated with every pass through the REPL. To respond in real time, the new output window cannot use htmlctrl. The new transcript window, however, can continue to leverage htmlctrl for that control's text formatting benefits.

The 1.3 release unifies the data paths and synchronization methods between the UI and Scheme threads by making the Scheme thread drive the REPL, with the UI thread in every case responding to a notification from the Scheme thread. The UI thread creates some windows, or promises of windows, then hands them to the Scheme thread. The Scheme thread then runs something similar to the following loop:

        (with-input-from-window THE-INPUT-WINDOW
         (lambda ()
          (with-output-to-window THE-OUTPUT-WINDOW
           (lambda ()
            (let loop
             (let ((value (eval (read (THE-EVAL-INPUT-WINDOW)))))
              (if (not (void? value))
               (display value (THE-TRANSCRIPT-WINDOW))))
             (loop))))))

The UI thread satisfies any read request from an input window in the same manner. The transcript window only updates in response to a direct write request, at which time it displays all of the previous REPL pass, using data that the other three window I/O channels have logged to it.

Contrasting the 1.3 redesign with 1.2:

The output window is constantly updated with a transcript of activity, tty-style.
The styled-text transcript window is separate from the output window, and is updated only episodically with each pass through the REPL.
The Scheme thread drives the REPL.
The Scheme thread is idle whenever it awaits data.
The UI thread responds to user activity, system events, and Scheme thread requests.
There is a single data path between the threads, and a single synchronization design.

An additional asset of the new design is the ease with which I can either reduce or extend the number of windows. All UI modality now operates as a function of the current active windows, which in turn is a function of the pending requests from the Scheme thread. A lightweight program runtime will lack the eval input and transcript windows, while a better debugger would have at least one additional input window along with multiple output windows.

As of this writing, the new REPL is in place, along with the new windows for input, output, and transcript, and all of the management logic coordinating those windows. With the Scheme thread now always active, I've lost the old means (an error code, returned from an evaluation pass) by which the REPL UI knew to retain syntactically invalid expressions in the input pane for editing and possible resubmission, so I'm experimenting with some new ideas— perhaps making errors on input ports call an optional function on that port, or perhaps generalizing the current GC announcement mechanism into a general hosting metainformation announcement mechanism. I also need to make the styled transcript window work on old-model Pocket PCs, which had a more primitive HTML output control. Finally, I need to think more about displaying error text, which presently takes place via another text stream interleaved into the output text (and displayed in a nice error-flag red in the transcript window).

A taste of last summer

Every summer I subscribe to a CSA run by a couple of upper Methow farmers. Once a week I trundle down to the weekly Mazama market, pick up my bag of goodies, eat everything that Kathryn won't touch, and haul the rest back to Seattle on the weekend to feed to Kathryn. The bag usually includes fruit, either local or imported from the next valley over (Okanogan) or downvalley (near the Methow-Columbia confluence). For the last couple of years the summer's final bag (usually arriving shortly after the equinox, making it rather the autumn's final bag) has included a considerable supply of d'Anjou pears, which require cold storage to finish ripening. Definitely a piece of deferred gratification.

In 2004 the pear payload was enormous: at least twenty pounds of pears, packed in a plastic foam crate. I took the crate and stashed it high in my shop, high enough to be safe from mice, figuring that the winter cold in my unheated shop would take care of the cold storage requirement. In the middle of January 2005 I checked my pears. The mice had left them alone, but they had ripened well beyond what I find appetizing, being brown and nearly liquid. I took each pear and hurled it into the field behind my shop, where it exploded into a brown pulp on the snow. Perhaps a family of voles got tight on fermented pear pulp that night.

In 2005's final CSA package I received not quite so many pears, yet still enough to fill a grocery-sized paper bag. This time I stashed the bag in my little refrigerator, rolling the top of the bag to close it. All autumn and winter I faced the bag whenever I opened the fridge to make dinner; all autumn and winter I remembered the previous winter's deliquescence. All winter I have been postponing opening the bag to reveal the state of its contents. (Schrödinger's pears.)

Yesterday I gathered my courage and opened the bag.

After six months, the pears had skins crinkly with many fine wrinkles, like the face of somebody's beautiful great-grandmother. Tentatively I bit into one. It was soft, but not too soft, with the grainy slightly-sandy texture of a d'Anjou, and a taste of honey and spice. It was perfect. I gobbled it and ate another immediately.

Today, immoderately, I ate three after returning from an afternoon run. It was 6C outside, warm enough to run without a hat or jacket, and the sun was beating on the mountains of plowed snow to either side of the road. Until I empty that bag of pears, I can still taste a little bit of last summer. That won't last long at the rate that they're disappearing.

My freezer holds three gallons of blueberries, another relic of last summer. It's hard for me to work up enthusiasm for frozen berries, however, when so much snow remains on the ground. I can mix the berries with yogurt, then stir until the yogurt freezes. When the yogurt berries enter my mouth, my head freezes. When I look out the window at the meter-plus of snow remaining, the rest of my body freezes.

The sun has a lot of work ahead of it. I'm almost out of pears.

(begin (display "Hello, world") (newline))

This entry serves to break a bottle of champagne over the bow of the new Pocket Scheme development log.

The preceding entries come from the main Pocket Scheme page, which will become more static as this blog thingy develops further.

German Word of the Week

The German Word of the Week is, of course, not updated weekly.

OMGWTFBBQ!!!!1!1!!eleven!!

Golly! This crazy gizmo really works!

I may not have anything to say, but by God I'll say it anyway.