A Robust Distributed Intelligent System for Telematic Music Applications

…is a project funded by the National Science Foundation within the CreativeIT program. Within this project, we are interested to develop intelligent agents for improvisational music collaborations over the internet.

Project Investigators:

Jonas Braasch, Pauline Oliveros, Doug Van Nort


Jonas Braasch, Pauline Oliveros, Doug Van Nort
(photo: Jonathan Chen)

Overview

Complex communication for co-located performers within telepresence applications across networks is still impaired compared performers sharing one physical location. This impairment must be significantly reduced to allow the broader community to participate in complex communication scenarios. To achieve this goal, an avatar in the form of a musical conductor with forms of artificial intelligence will coordinate between co-located musicians. Improvised Contemporary Live Music of a larger ensemble, serving as a test bed, is arguably one of the most complex scenarios one could think of, because it requires engaged communication between individuals within a multiple-source sound field that also has to be considered as a whole. The results are expected to inspire solutions for other communication tasks.

The avatar system will actively coordinate co-located improvisation ensembles in a creative way. To achieve this goal, Computational Auditory Scene Analysis (CASA) systems, to allow robust feature recognition, and Evolutionary algorithms, for the creative component, will be combined, to form the first model of its kind. The research results are expected to be significant by themselves and are not bound to telematic applications. With regard to the latter, the proposed system will have a clear advantage over a human musician/conductor, while intelligent algorithms are clearly lacking behind human performance in most other applications, especially when it comes to creativity.

Triple Point

Triple Point (Pauline Oliveros, Jonas Braasch and Doug van Nort) is a 2009 trio with a post-genre approach well inclined to the Deep Listening practice. The trio work is based on the development of the new IT tools which are developed within their CreativeIT grant from the National Science Foundation. The band derives its name from the thermodynamical point in the phase diagram where all three phases of water exist. Figuratively, this is where the trio operates exploring musical spaces and boundary conditions where contrasting ideas and streams can co-exist, while expanding the vocabulary of musical instruments acoustically (Braasch on soprano saxophone) and electronically (Oliveros, digital accordion and Expanded Instrument System, EIS, Doug van Nort on laptop and GREIS).

For many decades Pauline Oliveros has been actively expanding the voice of her main instrument, the accordion. Given the limited natural possibilities of this instrument with respect to sound (fixed tuning, no pitch bends, narrow variety in overtone spectrum), Oliveros has begun half a century ago to alter the sound of her instrument using tape delays and other electronic devices. Van Nort’s work is based in digital signal processing, transforming Oliveros’, Braasch’s and his own sounds using Granular Synthesis, psychoacoustically-motivated sound analysis tools and Genetic Algorithms to explore new musical textures and timbres.

Publications

J. Braasch (2009) The Telematic Music System: Affordances for a New Instrument to Shape the Music of Tomorrow, Contemporary Music Review, 28(4): 421-432

J. Braasch, C. Chafe, P. Oliveros, D. Van Nort (2009) Mixing Console Design Considerations for Telematic Music Applications, Proc. 127th Audio Engineering
Society Convention, Preprint 7942

J. Braasch (2009) Importance of visual cues in networked music performances, J. Acoust. Soc. Am. (conference abstract), 125, pp. 2516

D. Van Nort, J. Braasch, P. Oliveros (2009) A system for musical improvisation combining sonic gesture recognition and genetic algorithms, in: Proceedings of the SMC 2009 – 6th Sound and Music Computing Conference, 23-25 July 2009, Porto, Portugal, 131-136.

Music Releases

Triple Point (2009) Sound Shadows (full digital album, six tracks), Jonas Braasch, soprano saxophone; Stuart Dempster, trombone, didjeridu, little instruments; Shane Myrbeck, mixing and mastering; Pauline Oliveros, digital accordion, Expanded Instrument System (EIS, Track 3); Doug Van Nort, laptop, GREIS, additional mixing and spectral processing on Tracks 4 and 6. Deep Listening Records, DL-DD-1

Concerts

March 12, 2009: Triple Point (Jonas Braasch, saxophone, Pauline Oliveros, Accordion, Doug Van Nort, live electronics) at the Emily Harvey Foundation 537 Broadway NYC at 8pm sponsored by Deep Listening Institute, Ltd.

April 3, 2009: Triple Point (Jonas Braasch, saxophone, Pauline Oliveros, Accordion, Doug Van Nort, live electronics), NYCEMF festival, New York City.

September 14, 2009: Triple Point (Jonas Braasch, saxophone, Pauline Oliveros, Accordion, Doug Van Nort, live electronics) at Harvestworks, New York City.

October 18, 2009: Triple Point (Jonas Braasch, saxophone, Pauline Oliveros, Accordion, Doug Van Nort, live electronics), Roulette, Part of New York Electronic Art Festival, New York City.

Listen to an excerpt from their concert at the New York Electronic Art Festival in New York City (October 18, 2009). Two other works by Moritz Wettstein and Lucky Dragons are also shown in this clip:

Here is another excerpt from our concert at the Emily Harvey Foundation (March 12, 2009, New York City), in which a number of new developments from the CreativeIT project were presented:

Here is a short excerpt from our telematic work:

Tintinnabulate & related Courses

Students are able to participate in the project through seminars that are linked to the project. The following excerpt shows a concert at EMPAC and Second Life with the RPI ensemble Tintinnabulate. Tintinnabulate and our CreativeIT project play an important role in our ongoing seminar on Music Composition, Improvisation and Telematic Music. Past seminars include:

“Experimental Telepresence”
(ARTS 4962/6962, instructor: Pauline Oliveros, co-instructors: Jonas Braasch, Doug Van Nort) Spring, 2010.

“Composition, Improvisation & Performance” (ARTS 4964/6964, instructor: Pauline Oliveros, co-instructors: Jonas Braasch, Doug Van Nort), Fall 2009/10.

“Experimental Telepresence”
(ARTS-4962/6962, 4 credits, instructor: Pauline Oliveros, co-instructor: Jonas Braasch), Spring 2009.

“Mixed Reality Seminar”
(ARTS-4961/6961, instructor: Pauline Oliveros, co-instructor: Jonas Braasch), Fall 2008/09.

Here is a short clip of Tintinnabulate in a Concert at EMPAC and Second Life:

Useful Links