Ever had any interest in hearing a birdsong, or perhaps the summer rain, or how about the ocean waves? Users can type in anything that comes to mind, and Audio Metaphor will generate a soundscape based on that query.

The soundscape generation system, developed by Philippe Pasquier and Miles Thorogood, is currently on exhibit as an interactive artwork at the online Sight + Sound International Digital Art Festival until the end of August.

Both creators are interdisciplinary artists and scholars who are working at the intersection of art, science and technology. Pasquier is an associate professor at the School of Interactive Arts & Technology at SFU and the director of the Metacreation Lab. Thorogood is an assistant professor at the Creative Studies Department at UBC’s Okanagan campus.

A soundscape search engine

The inception of the project began nearly a decade ago, inspired by the ideas of memory, location and emotion according to Thorogood.

“Listening to the soundscape of a location brings on an emotive effect in me,’’ he says. “I had the opportunity of traveling to southern Mexico once, but I didn’t have my sound recorder. Then I thought there are enough bits of sounds on the internet that can be brought together to have some sort of a representation of what that experience was like, so it became a sound design challenge.”

With a background in sound design for theatre, he also recognizes that there are a lot of repetitive tasks in finding and matching sounds.

“What is the cognitive process whilst creating sound design?” Thorogood asks. “How can we model those cognitive processes by machines so they can simulate and carry out those tasks, leaving us more time for the kinds of things that we enjoy, such as coming up with ideas?”

Thorogood worked with Pasquier at the Metacreation lab, which specifically focuses on creative AI and generative systems, and the project came into being and is still being refined.

“We are working on boosting the database. There are 40,000 sounds in the database right now and we are multiplying it by 10. If you try the system in one or two weeks, there will be more sounds, and you can make crazier soundscapes,” says Pasquier.

Thorogood adds that they have also linked it up with Twitter stream to access community-contributed audio database that is online and real-time.

Behind the simple appearance of a search bar, Audio Metaphor actually encompasses a pipeline of sophisticated computational tools for generating artificial soundscapes.

First, the text analysis algorithm identifies key semantic indicators to retrieve related sounds and optimize the results based on the combination of keywords. The results are then segmented into background and foreground sounds. Each segment is then run through an emotion predictive model that applies mood labels to the sounds. Finally, a mixing algorithm takes labelled sound segments and selects and arranges them into the final soundscape based on the overall mood of a mix.

Pasquier explains that there are also different versions of the model, one that is used as an interactive art project for the general audience and another one that is intended for sound designers to explore the creative process.

“Often in the lab, as in most of our projects, there is a continuum of pure scientific research and artistic creation. For some of the algorithms, we also work with companies that are making audio software, to develop some of our solutions,” he adds.

A promising future for creative AI

Both are excited about their research and the future potential of creative AI.

Leading the Metacreation Lab on multiple projects, Pasquier recently had a generative art exhibition in Istanbul that explores automatic video generation. He is also working on a house music album made with AI.

“Creative AI is booming. There will be a number of big changes in the industry because of algorithms. Sound and video editing are becoming the main types of repetitive creative tasks. We can certainly improve on the process,” says Pasquier.

Thorogood says they have just built the Sonic Production, Intelligence, Research and Applications Lab at the Okanagan campus of UBC to further explore other aspects of sound design to enrich the space that AI can search and create within.

“Creative AI provides a fantastic opportunity to explore what it means to be human as a creative sentient being. We can explore more deeply, relate better to other people and build new tools to further the creative process and the cultural forwarding of our society,” he says.

For more information, please visit: https://www.easternbloc.ca/en/exhibits-events/sight-and-sound-festival-online