KENNETH LIM @ ART PAGES
SOUNDWAVES
Soundwaves, is an interactive computer system created to explore new applications for existing technology. No attempt was made to break new technical ground with this system, as it is a work of art - developed primarily for performance.
Soundwaves follows commercial trends by attempting to meet the demands of the entertainment industry for a dynamic real time graphic system. The use of multi-screen projection in music concerts calls for more sophisticated image systems during performance. Current image processing systems are limited to the scripting of pre-developed graphics and image components. To gain the maximum audience, this installation must have entertainment value. A decision was made to develop Soundwaves for public arenas, such as shopping centres and libraries, hence, the need to be portable . Interactive systems create unique moments for the user within a particular context. Soundwaves share stolen moments individually with its user that is unique to the participant. Such moments will be dynamic, constantly evolving and mutating in real time, driven by the participant. It uses genetic1 algorithms with sufficiently complex behaviours to ensure that no two moments will ever be the same. So far, human-computer interaction is uneven with humans holding the upper hand and making decisions for the interaction. Things will change in the future, when computer technology become more autonomous. Bearing in mind that special effects, such as animation and other transformational processes found in the motion pictures involve extensive capital investment and production time, this installation must have an alternative aesthetic system that has the flow and continuity of the motion pictures, without its heavy processing workload. The computer used should not distract the user with its presence, yet the visuals must seduce the users into maximum interaction, whilst entertaining him/her in his/her participation. The most common leisure activity among humans is singing. Singing has a continuity and flow that is perfect as the system can sample the activity continuously in real time to drive the computer graphics. The equipment needed to link human to machine is also minimal - only a microphone is needed.
To form a symbiosis linking Cyberspace with the external physical world, Soundwaves binds the energies that feed both the human sensory system and the computer processing system. Computer graphics are generated in real time according to the quality of the voice inputs. Adopting Krueger's position for developing an interactive human-computer installation, Soundwaves is a dynamic computer graphic system that serves as an extension to the human performer.
Funded by the New Technology Research Grant of the Australian Film Commission, Soundwaves is made by an aggregate of artists and technologists with a common aim of developing an innovative interactive computer system for artistic expression:
Producer/Direction & Concept: Kenneth Lim
Engineer: Malcolm May
Music: Scott Ludlam, Jose Garcia, Derek Rucki
Graphics: Kenneth Lim
Script: Kenneth Lim
Voice: Geraldine Williams
The equipment used for this installation are minimal:
486DX33, 16 Mb RAM, 210 Mb Hard disk
Sound Analog to Digital converter (built for this function)
Tempra Software & Corel Draw for graphics
Turbo C for Soundwaves system software
Dynamic microphone
SVGA to Pal scan converter
Video projectors
Music sound system, including a CD player
Though co-dependent, the participant and cellular automata in Soundwaves are bodies of dynamic processes that continually regenerate, decay and mutate to survive. The dynamics of one environment affects not only its inhabitants, but have a parallel and causal effect on others related to it in a dynamic loop system where relations are negotiated and established according to survival needs within the dynamics of multiples, achieving a form of "Structural Coupling" (1980) described by Maturana and Varela. "Structural Coupling" describes the ability of an organism to be flexible enough to adapt itself internally to the dynamics of the environment in which it exits, that is, changes within the environment will cause a parallel change, or reorganization within the organisms that exist within it. The plasticity of organisms are crucial to their survival, yet, their organisms must also be rigid enough to maintain their fundamental characteristics. The artificial organisms, Cellular Automata or Artificial Life as they are often called, are patterns developed mathematically, possessing a set of DNA-like genetic codes that make up their behaviours and livelihood within an environment. Within this Cyberspace or Artificial World, there exists an artificial food chain in which organisms take on predator and prey roles; adopted by survival instincts in response to the demands of the environment they live in. Artificial organisms mutate by varying reproduction. The most influential work on lifeforms has been the work of Maturana and Varela (1980) which looked at lifeforms as "autopoietic machines1", and the environment by the computer system to generate a fluid graphic representation of the activity. Briefly, Soundwaves aims to provide an environment where users can interact with the simulated environment. surrounding them.
Soundwaves involves a primitive creation of the Gibsonian "simultaneous experience of time, space, and the flow of multi-dimensional, pan-sensory data" (Theall, 1992, [2]). The multiple experiences of the performer during performance will negotiate with his/her environment through sensory data input with his/her voice output, which is interpreted as the environment in which the artificial organisms live within cyberspace, which make up the evolving graphics of the installation.
Soundwaves explores such an integration, considering the continued coupling between humans and digital technology, primarily for developing an interactive computer system that will co-evolve with the activities of a biological (human) body. Such co-evolution is possible only when all information is reduced to the 1/0 binary system, and reinterpreted in a multitude of ways that will stimulate human audio-visual senses. The installation will have, as its basis, an interpretation procedure, whereby a human activity will be interpreted in computer generated graphics.
Figure 2: Proposed Interactive system for Soundwaves
Having identified the various strata of human-computer interaction, and considered the potential of Cyberspace, the abstract environment within the computer, and the material world of the user, Soundwaves will attempt to have an interactive system whereby Cyberspace and the material world are linked by the human user. The artificial reality (Krueger, 1993) in Soundwaves will be computer generated graphics in real time, upon human input. This activity is projected to the external environment of the participant, enveloping him/her and creating a closed loop system, where the activity of the human user will directly affect activities within Cyberspace, which affects the artificial environment of the user, which will motivate him/her to respond to the environmental changes. Important explorations will be the type of activity the human user should perform during the installation, and the type of aesthetic Cyberspatial activity that will influence the user. Soundwaves picks up where Japanese Karaoke machines left off. The popular activity of singing to pre-recorded music and images whilst accompanied by a moving text in Karaoke sessions, relies on the participant to follow a pre-programmed output of the digital system, without any level of negotiation or interaction, just a functional interface between user and the Karaoke system. It takes this human-computer relationship a little further, by making the computer respond in real time to the participant's voice output graphically; by dancing, negotiating and coupling (Sofia, 1993, Sofia 1992, Winograd and Flores, 1986, and Maturana and Varela, 1982) with the human activity.
Soundwaves ensures that human activity is not intruded by any technological gadgetry other than a microphone that can either be held by the participant or planted somewhere within the participation space. The synchronicity of the graphics with the voice output of the computer should seem to come from thin air.
The energy of the voice output of the participant in Soundwaves will be sampled to give a numerical value whereby the environment of the computer graphics can be varied accordingly. The aesthetic meaning of the computer graphics generated, such as colour and form, will be coded loosely according to conventional aesthetic interpretation. Intensity of voice signals may also be reflected in corresponding intensities of colour in the computer driven graphics. Information is attained when sufficient energy is spent organizing the bits into an array, making up a higher level sign system. The signs are then organized to form syntactic and semantic structures of a language, hence in the context of Soundwaves, the information is found in the interpretation of the artificial reality by the participant as it is projected onto the walls of the physical space that he/she is in.
Soundwaves explores multiple experiences linking human voice with computer graphics to create a loop system of interpretation within both the computer system and the participant, but its potential for multiple experience is even greater because the multiple stimuli occurs externally within the physical environment of the human participant, immersing the human body fully within the artificial reality, within the multiplicities and fluidity of experience. The potential for rhizomic "assemblage" of multiple connections of the computer, with other machines, users, software, promotes a multiplicity of interactions, processes and actions at the same time, dissolving the holistic Self into an aggregate body of multiple experiences, into a symphony of information processes, has enormous potential for Soundwaves.
Interpretation of sound input
Sound input is measured in two ways. Firstly, they are sampled (the interval between sampling is adjustable) for digital values. To map out the fluidity of the signal, the difference between two sample values are taken as the variable unit;
A1-A2 = Av (Amplitude variable)
F1-F2=Fv (Frequency variable)
System overview
There are three parts to this system, as I specified to my software engineer.
1. Computer graphics must be allowed to be scripted when required, to accompany the genetic algorithms. Graphics utility must utilize standard format such as bitmaps.
2. Random text function, where ASCII files (American Standard Characters) are recalled randomly during the operation, to create a non-linear narrative function.
3. Genetic algorithms must possess predator and prey entities. This is the tricky part of the project because of the autonomous nature of the algorithms. We can only specify the behavioural characteristics, whilst have no control over the evolution of the graphics resulting from the nature of the voice. Utilizing chaos and entropy theories, graphic entities are created, mutated and interact with each other according to the variation in the environment caused by the variation of the voice input. The only real input we have was to specify the shape of the entities, their colour characteristics, and the scope of their behavioural patterns under particular threshold conditions, according to our aesthetic demands and then let evolution take its course.
Random Text Files:
Random Text is to be superimposed onto bitmap graphics files to form a non-linear random narrative. They are single-line texts randomly selected from a 64 line (max) ASCII text file. Different bitmap graphics can have different text files to select random texts. Here is a sample of text to be generated for the installation:
File 1:
reason
rationality
science
truth
science = truth
science = untruth
paradigm
paradigm shifts
body
mind
technology + body = cyborg
dialectics
logic
nothing
nothingness
0 = nothing
0 = something
transform
transformation
translate
translation
sign
signifier
signification
sign + context = meaning
inference
deduction
induction
analytic
logical truths
synthetic
intuition
instinct
automatic
autonomous
association
knowledge
cognition
intellect
intelligent
information
meaning
semantic
context
content
pragmatic
interpretation
metaphor
collision
convergence
circulate
divergence
disintegration
dissipation
diffusion
decentralize
disembody
structural coupling
randomness
continuity
transform
craft
virtual
File 2:
real
reality
social
socialize
socialization
evolution
selection
Bionomics
network
Global Village
emerge
emergent
emergence
abstract
organ
organize
organization
system
systematize
material
economy
biological
artificial
commodification
data
natural
unnatural
knowledge
representation
attention-seeking
capital=technology
capital
$
dynamic
innovate
technology=capital
innovative
innovation
imitate
imitation
mimic
simulate
simulation
evolve
mutate
continuity
genetic
behaviour
function
behaviourism
functionalism
relate
relation
relational
relative
relativism
realism
idealism
synonym
synonymous
dual
dualistic
dualism
discrete
File 3:
real
reality
social
socialize
socialization
evolution
selection
Bionomics
network
Global Village
emerge
emergent
emergence
abstract
organ
organize
organization
system
systematize
material
economy
biological
artificial
commodification
data
natural
unnatural
knowledge
representation
attention-seeking
capital=technology
capital
$
dynamic
innovate
technology=capital
innovative
innovation
imitate
imitation
mimic
simulate
simulation
evolve
mutate
continuity
genetic
behaviour
function
behaviourism
functionalism
relate
relation
relational
relative
relativism
realism
idealism
synonym
synonymous
dual
dualistic
dualism
discrete
Graphics input:
The graphics for Soundwaves for this current application are created using Tempra and Corel Draw in the .PCX graphic file formats. Initially they are stored in the hard disk of the computer in Windows 16 colour Bitmaps as the system can presently work in that utility. The system is set up to display the bitmaps at the minimal rate of 4 frames per second. The other option, if the bitmaps slows the processing time of the genetic algorithms, is to let the animation run on Tempra Show or Animator on a separate machine and mixed into the performance by a vision mixer. Ideally, the installation should have live video images of the performer captured in real time during performance, or be attached to a videodisc player to display prerecorded images.
An aesthetic decision was made to generally split the graphics presentation into two. The first part which reflects Western philosophies are developed to give a digital feel. Colours chosen are opposing such as black and white, with simple patterns, such as stripes and squares. The place Japanese philosophies in a background that reflect their analog concerns, such as their Mandalas. Below are samples of graphics for this installation:
The screen below is taken from blocks of bitmaps developed for the Japanese section of the installation. The definition of the pattern is lost from the reduction of colours from 256 to 16 in the Windows Bitmap conversion.
Genetic algorithms:
Genetic algorithms are created in this instance to depict a situation. The scenario will depend on the sort of graphic entities we decide should coexist within a particular environment. A dynamic situation occurs when the entities actively negotiate their place within the environment, either by repelling their neighbours, or preying on them. These entities are stored independently to facilitate recall when required. They have different behavioural and graphical characteristics, providing different responses to the same environmental variable. They are created solely for their independent aesthetics, and the overall situational aesthetics created by their interaction. The response time of the algorithm will depend on the hardware. If the computer has sufficient processing power, then the evolutionary movements will be smooth and in real time, if not, the movements may seem a little stunted.
Malcolm May, the software engineer and the Director worked around existing fractal or cellular automata graphics (See Below) to provide a working base, which evolved to incorporate behavioural patterns.
Figures 3 & 4: Samples of Cellular Automata as featured in CA LAB (Rucker, 1989, pp. 32,33)
Figure 3: The RC Rug rule with character "R"
Figure 4: The RC Rug rule run with Character hex DC.
Ideally, the algorithms should look like the works of Yoshiyuki Abe.
Figure 5: Legend I by Yoshiyuki Abe (1992)
Figure 6: Legend II by Yoshiyuki Abe (1992)
Figure 7: Legend III by Yoshiyuki Abe (1992)
Figure 8: Communication - Dawn by Yoshiyuki Abe (1990)
Figure 9: D-Minor by Yoshiyuki Abe (1991)
Figure 10: Vibrant Drive by Yoshiyuki Abe (1991)
Figure 11: Memorable Time by Yoshiyuki Abe (1991)
Figure 12: Communication - Day by Yoshiyuki Abe (1990)
Music
A strict combination of Eastern and Western musical instruments are used for Soundwaves. Scott Ludlam, the composer was only allowed to begin with only one instrument which evolves fluidly into a rich symphony towards the end. The reference music for him was the theme from the Stanley Kwan movie "Centrestage" (1991), entitled "Buried Heart" written by Taiwanese songwriter Xiao Cong, released by Rock Records.
A decision was made to use an Asian voice for the singing, hence Geraldine Williams.
Projection and Performance space:
The ideal projection for the output is a direct digital projection system, but all it needs is a minimal of 10 ft3 of white performance space, with a microphone in the middle. The projected graphics should be as big as possible, even utilizing 4 projectors whenever possible - 3 for the walls, and one covering the floor space. Apart from the microphone (preferably wireless), the performance space is free.
What you will have seen in this accompanying video copy of Soundwaves is a prototype of the system. There is much more to be done, such as developing more graphic file capability, such as .PCX, .TIF, developing a much larger base of genetic algorithms and finding new ways of processing the algorithms to allow for more than then 16 colour, 640x480 resolution that it currently operates under. One of the biggest problem so far, has been processing speed. The hardware as handled by an existing Turbo C software has limited memory allocation for a real time processing function demanded by Soundwaves. Further developments in memory allocation are needed to produce higher resolution algorithms. Another associated problem is the speed, or lack of speed of accessing bitmap files. The desired speed of dumping 4 bitmaps per second onto the screen has yet to be realized.
The graphics, video images and text you see in the video are edited onto the recorded video of the genetic algorithm. This is due to problems Malcolm had with getting the computer to read between different resolutions in all components. Anyway, this prototype should be effective as a stand alone installation even as it is, if the performance space is enclosed as prescribed above, because the link between voice and graphics, which is the most important component, has been achieved.
NOTES
1. A contribution by Tom Holroyd, (from the Centre for Complex Systems and Brain Sciences, Florida Atlantic University, tomh@bambi.ccs.fau.edu, dated Tue, 28 Sep 93 16:39:46 EDT) in a series of ongoing discussions on the autopoiesis network in the INTERNET (autopoiesis@world.std.com), referred to Robert Rosen's definition of ""living organisms as being "closed to effective causation. This means, roughly, that living organisms are fully autopoietic - they produce the means of their own production. This is also, I believe, what is meant by 'organizationally closed.'" Holroyd reads Rosen to classify natural systems under "two broad classes of models, analytic and synthetic. Synthetic models are constructed of parts, these parts representing the end products of reductionism. Analytic models are made of observables and their interactions (behavioural, holistic)..." and concluded that ""mechanisms" are not capable of the organizational closure needed in order to be autopoietic. ..that a purely formal description of life or any autopoietic system can never be complete." What is interesting about Holroyd's position is his interest in "the ties between Buddhism and complex systems theory.", an ex-Strong AI practitioner, Halroyd drew an interesting dichotomy between:
East West
wave particle
mind brain
semantics syntax
content form
Holroyd also defines perception as "...not just to process sensory data, but to apprehend meaning through interaction." The activity required to perceive, either within biological or mechanical systems, is, according to Holroyd, pluralistic, where the processing of sensory data has to take place within a series of interactions. The dualistic East/West distinction is of primary interest to this thesis as they explore the differing perceptions involved in the appropriation and development of technology, particularly with regards to Artificial Intelligence, and the need to resolve this dichotomy by developing a system versatile enough to accommodate both systems, a position well argued by McLuhan and Powers (1989). For more information on Rosen, read Robert Rosen's Life Itself , Complexity in Ecological Systems Series, Columbia University Press, NY, 1992.
Chapter 1: Attention for Sale: Capitalism and Interactive Computers Chapter 2: Defining Human-Computer Interaction Chapter 3: Representing our Worlds: Digital Translation Chapter 4: Digital Intelligence: Parallel flow of Multiples? Chapter 5: Japanese Philosophy & Artificial Intelligence Research Soundwaves Conclusion Appendix A, B, C, D, E, F, G Bibliography
This page hosted by 
Get your own Free Home Page