Listen Reader: An experiment in multi-modal reading
The Listen Reader is a personal interactive reading experience that combines the look and feel of a real book - a beautiful cover, paper pages and printed images and text - with the rich, evocative quality of a movie soundtrack. It is designed to preserve and even heighten the experience of immersive reading. It explores the idea of multi-modal reading: the use of background sound to provide a sense of place, and to add affect to the experience of reading a book without interrupting the flow of the story.
The Listen Reader also preserves the beauty of the book as token object. The design uses no LCD or pixeled screen, but traditional, richly colored printed pages and a classic immersive reading environment: a comfortable chair, a polished hardwood swing-arm reading stand, a pool of light in an otherwise shadowed corner. The soundtrack is controlled by the motion of the reader's hands above the page; any motion within eight inches of the sensors is read as a volume, pitch, or pan control. Thus the natural gestures of the reader turning the pages provide a simple soundtrack; more complex, deliberate gesturing allows the reader to create richer soundscapes.
The technology is designed to be unobtrusively embedded in the pages and binding of the book. RFID (radio frequency identification) tags embedded in each page sense what page the book is open to; capacitive field sensors measure human proximity to the pages. Proximity measurements (hand gestures) control volume and other expressive parameters of the sounds associated with each page. In this instance, the page-ID data controls which set of sounds is presently being heard. The sound is multi-track and includes music and sound effects (but *not* a spoken version of the words on the page; that would interfere with the reader's own ability to absorb the text.)
A technical paper on the Listen Reader.
Listen Reader credits:
Maribeth Back and Jonathan Cohen
Thanks to: Rich Gold, Chronicle Books, Atlantic Records, Roy Want, Sharon
Masnaghetti, Toby Sticpewich, and most particularly to Neal Stephenson for writing The Diamond Age which features the book we really want to build...
The three Listen Readers created social immersive reading environments.
Copyright The Reading Lab 2005. Last updated March 20, 2005. Comments welcome.