[CLIP: Hubble Cantata]

Jason Drakeford: It’s 2016, and we’re in Prospect Park, Brooklyn. A live orchestra and choir perform opera music while thousands of people press tiny cardboard boxes to their faces. This is the Hubble Cantata.

Timmy Broderick: Yeah, so these virtual reality headsets are pretty jank, but the scenes the audience is looking at through them are majestic. It’s a 360-degree view of some of the most stunning galaxies that astronomers have ever captured. Meanwhile sweeping classical music that matches the images just envelops the audience.

[CLIP: show theme music]

Broderick: Alright, well. Whoosh whoosh whoosh—Hubble Cantata—whoosh whoosh whoosh. Jason, like, bring us in. 

Jason: [laughs] Okay, let’s do it. Are you ready? 

Drakeford: You are listening to Scientific American’s Science, Quickly. I’m Jason Drakeford. I’m an animator and video journalist.

Broderick: And I’m Timmy Broderick. I’m a freelance journalist who covers disability and a recent intern at SciAm.

[CLIP: “5K Exoplanets” by Matt Russo]

Broderick: This week we’re taking over the feed and blasting off into space. We’re going to take you on a journey through the stars.

Drakeford: Welcome to part one of a three-part Fascination on how scientists and artists are turning space into sound. You’ve seen photographs from the Hubble and James Webb Space Telescope. Now, get ready to hear them, including one brand-new sonification that has never been publicly released.

Broderick: But to get to the stars, we need to start somewhere a little bit closer to home.

Paola Prestini: What we did in the park was essentially create, like, an eight-point loudspeaker system so that people in the park could feel like they were traveling through space in this kind of completely enveloped sound. To essentially take you, as a viewer, were suspended in space, with Earth at your toes, going through this incredible journey, finally, through the Orion Nebula. 

Drakeford: This is Paola Prestini, she’s an award-winning composer who has collaborated with poets, and artists, and scientists. She also likes to say she “paints with data.”

Prestini: My training is obviously in composition. I like to say that I’m essentially an explorer, and I do that mainly through sound.

Broderick: Prestini isn’t the only one turning astronomical data into sound. Other artists and scientists are using markers such as orbital paths and star brightness to craft their own symphonies. It’s led to the birth of a new field: astronomical sonification.

Prestini: There’s this idea that there’s no sound in space because we can’t hear them. In fact, there’s been incredible explorations, most recently with black holes and pitching the [gravitational] waves up so that we can actually hear them with human ears.

Drakeford: Astronomy has always been a visual science, but data doesn’t care how it’s presented. Scientists have just been defaulting to images. And that’s changing now.

Broderick: Yeah. Turning astronomical data into sound is not just cool, these soundscapes can also inspire folks who are blind and visually impaired, in the same way that you or I, Jason, you know, are inspired by those images from Hubble and the JWST. And also, these sounds might lead to some discoveries, too.

To get there, the field is going to need to be more formalized. Should a bright star lead to a low pitch or a high pitch? Is a cello or an oboe a better instrument to capture a comet’s path? Right now there’s no standardization.

Prestini: In terms of sonification, and as composers, we kind of make up these rules.

Drakeford: Timmy, what the hell even is a sonification?

Broderick: Right, so a sonification is exactly what it “sounds” like. It’s turning data into sound. You can sonify the stock market by making a piano’s note correspond to the nightly closing stock number.

[CLIP: Music]

Or if you want to be silly, you could sonify the annual number of Prussian cavalry members killed by horse kicks, starting in 1875.

[CLIP: Music]

Broderick: In both of these clips, you can hear the data pretty clearly. What Paola is doing is a little different. The notes, dynamics and instrumentation of the Hubble Cantata don’t map exactly to the stars or galaxies that people saw in the VR simulation. 

[CLIP: Hubble Cantata]

Paola was more inspired by the images. But other scientists are doing that one-to-one mapping.

[CLIP: “TRAPPIST sounds” by Matt Russo]

Drakeford: What you’re hearing right now is a sonification of planetary orbits from astrophysicist Matt Russo. 

Broderick: The piece captures data from 40 light-years away in the TRAPPIST-1 system.

Drakeford: Seven rocky planets orbit TRAPPIST-1. And each time a planet passes in front of the star, it blocks a little bit of light.

Broderick: The notes you’re hearing are planets completing their orbit. This is an “orbital frequency.” Matt multiplied each planet’s frequency and turned these stellar passages into specific musical notes. He then stacked the frequencies, added drums for when neighboring planets passed one another, and voilà.

Russo: When I saw the pattern in their orbits, I could quickly calculate which notes those correspond to, and I knew would be a really, like pleasing, beautiful chord altogether.

Drakeford: Matt is a physics professor at the University of Toronto, as well as a musician and a data sonification specialist. When he was growing up, he loved both music and astronomy but he couldn’t find a way to marry the two passions.

Russo: As I grew older, it became actually a source of conflict because everyone told me I had to pick one or the other. I kept these two worlds in parallel, knowing they would eventually crash at some point and explode, or I don’t know what was going to happen.

Broderick: This “crash” has produced some pretty spectacular work. Matt’s sonifications have been heard across the world. Probably his most famous sonification is the clip you just heard. 

Russo: We had to give the system a voice. It’s not actually making any sound, but we can still hear the rhythm and harmony of this faraway solar system.

Drakeford: Matt released this sonification in 2017 with astrophysicist Dan Tamayo and musician Andrew Santaguida. The response was overwhelming. The video got a ton of press and was written about in places such as the New York Times and Gizmodo.

Russo: It’s always just overwhelming how much people connect with the sounds that we’ve created [out] of these images. So it’s an art form, but in the process, they are learning something about the astronomical system and also just about information and how you can experience the same information using two different senses.

Broderick: In the years since, Matt has gone on to create more sonifications and work with NASA, like this one of a black hole.

[CLIP: “M87 Jets” by Matt Russo]

If you want to join the fun and see Matt’s very cute dog, he has a series explaining how to make your own sonifications.

[CLIP: Russo speaks in a video in which he is accompanied by a King Charles spaniel named Marty: “I’m Matt, and this is Marty, and we’re going to show you how to convert any data into music.”]

Drakeford: But these sonifications are more than just a musical exercise or vanity project for Russo. He started the SYSTEM Sounds outreach project with Dan and Andrew to make sonifications.

Russo: We wanted to be able to convert astronomical images into sound, partially because we thought it was interesting and fun but also to make those images accessible to people that don’t have sight, people who are blind or visually impaired.

Broderick: Russo’s not the only one pushing for this. Astronomical sonification was started by a blind astronomer. We’ll have more about that story in our next episode, but this was always the goal: expand our understanding of the stars—especially for people with disabilities.

Drakeford: Before we go, let’s listen to an excerpt from a never-before-heard sonification of gravitational waves that Matt and Andrew made. Gravitational waves ripple across the fabric of space time thanks to big energy events, like a colliding black hole. These waves clue astronomers in to the structure and composition of the universe. Matt wanted to capture this.

[CLIP: “Gravitational Waves” by Matt Russo and Andrew Santaguida]

[CLIP: Outro music]

Broderick: Next episode, we’ll be digging into the origins of astronomical sonification and why sound can be just as useful as sight to understand space.

Wanda Díaz-Merced: And I yes— and I heard it! Yes, yes!

Drakeford: Science, Quickly is produced by Jeff DelViscio, Tulika Bose, Kelso Harper and Carin Leong. Our theme music was composed by Dominic Smith.

Broderick: Matt Russo and the open-source website TwoTone provided the sonifications you heard in this episode. Don’t forget to subscribe to Science, Quickly wherever you get your podcasts. For more in-depth science news and features, go to ScientificAmerican.com. And if you liked the show, give us a rating or review.

Drakeford: For Scientific American’s Science, Quickly, I’m Jason Drakeford.

Broderick: And I’m Timmy Broderick. See you next time!