Chris Chafe
Director of CCRMA at Stanford

Chris Chafe: Decades of Sonification work

with Scot Gresham-Lancaster / 16 May 2017 / 53:09

Chris Chafe and host Scot Gresham-Lancaster dive into an in depth one hour conversation about many aspects of Chris's decades of in depth work in sonificaiton. Chris is easily one of the worlds leading experst in this field with prolific output of new material and many rich new ideas that have made him one of the most important figures in the field. Chris Chafe's web page

The part of his website dedicated to sonification work can be seen at
Pointer to Sonification Workse

Bio:Chris Chafe is a composer, improvisor, and cellist, developing much of his music alongside computer-based research. He is Director of Stanford University’s Center for Computer Research in Music and Acoustics (CCRMA). At IRCAM (Paris) and The Banff Centre (Alberta), he pursued methods for digital synthesis, music performance and real-time internet collaboration. CCRMA’s SoundWIRE project involves live concertizing with musicians the world over.
Online collaboration software including jacktrip and research into latency factors continue to evolve. An active performer either on the net or physically present, his music reaches audiences in dozens of countries and sometimes at novel venues. A simultaneous five-country concert was hosted at the United Nations in 2009. Chafe’s works are available from Centaur Records and various online media. Gallery and museum music installations are into their second decade with “musifications” resulting from collaborations with artists, scientists and MD’s. Recent work includes the Brain Stethoscope project, PolarTide for the 2013 Venice Biennale, Tomato Quintet for the transLife:media Festival at the National Art Museum of China and Sun Shot played by the horns of large ships in the port of St. Johns, Newfoundland.

more information here


Scot: Welcome to Sound and data. This is for the sound data channel on the creative disturbance podcast. I've got the opportunity to talk to my old friend Chris Chafe, who is the director of the Center for Computer Research and Music and Acoustics CCRMA at Stanford. But I think the reason why I mean he Well, we have a common back. Well, what would you say was the first time with with the Pauline Oliveros stuff? Right? That that? Yeah, yeah. It's a course called online jamming and concert technology and it's offered through cadenza. Yeah, telematic performances, which I would recommend any listener to check out on Kadenze,

Chris: 0:38 the course on how to go the on ramp to doing telematics?

Scot GL: 0:58 Yeah, yeah. And it's, it's really thorough, and it uses our mutual friend, Juan Pablo Caceres software Jacktrip, that Chris also fundamental development. But anyway, the but the real thing, this is the sound and data channel, and I mean, I, I hadn't really caught up with your I knew you were doing some indications from Ping. That's the first time way back. Yeah. Yeah. I mean, you know, because we were working with 2001 in parallel and talking to each other in that period. So ping was 2001. What What was that piece about?

Chris: 1:30 That was, you know, literally is the first time I guess I could be accused of doing sonification. Yeah, ping was a project for SF MoMA. And we had this notion of basically converting real time traffic flows on the network into something, you know, audible. So you know, basically, we were already doing a lot of telematics, we were spewing audio around cat five cables, and connecting computers to walls, and then walls to other walls and computers on the other side of the world. And the notion was just to make the traffic, something that we could perceive, you know, that was never this

Scot GL: 1:38 Ping is like a command for those who don't know what, on Linux, where you ping another computer, and you can get back how many milliseconds it takes, which can be thought of as almost a rhythm in a way. Yeah, I always thought this telematic thing was, there was never a one. Because you're, you're now Yeah, it was like at least 15 milliseconds.

Chris: 2:42 Yeah. But But literally, this was this was like listening to the almost to the infrastructure or the flows. And so I was working with Greg Niemeyer who's an artist in Berkeley, Greg and I, at the time, this is our first collaboration, we are still working. Yeah. And since then, you know, it's, that was 17 years ago. So we're, you know, it's a long, long standing very fruitful twosome here. And, and Greg, basically, after a couple years of this together, just realize that what we were doing was going to make sense if we issued a CD called "Extrasensory Perceptions", plural. And the you know, very, very simple concept of like, you know, if we had a, you know, internet traffic monitoring since built into us, we wouldn't have needed ping or you know, and yeah, other projects we did.

Scot GL: 3:39 And we mentioned before I started the recording, David Worrall has had a similar sort of idea about, yeah, sonifying the network. Yeah. And my feeling is, and we didn't even know we were doing it. But the hub, which is a group I've been involved with, we're basically

Chris: 3:53 You were doing it,

Scot GL: 3:54 We're doing that we didn't know it, you know, but we were basing the whole thing on the sort of, were you like me, I sort of backed into the sonification business. Yeah. I mean, you know, because it came out a procedure and a post- Cagean thinking about numbers as ways of being and all sudden night got called up to go to a conference on sonification. And he realized I had been doing it

Chris: 4:16 L iterally need to there was, I, you know, I was doing computer music, at some point where it just, you know, that was, like, where my, my craft was gonna go, I could tell and, you know, I didn't, I'm not worried about doing, you know, detailed work by hand, I'd rather program something and take advantage of the computer and build algorithms and so on. And a lot of the early work like that was was really a form of sonification. But I just didn't think of it that way. You know, so I, I create systems and then listen to them. Yeah. And then tweak them until they made my music

Scot GL: 4:54 I mean, I was working with Xenakis way back in the 80s. And he said this thing where, you know, it's making the marble but you're still the sculptor. Yeah, so Exactly, exactly.

Chris: 5:07 Yeah. And when I say it's my music, yeah. And in the same sense , it's really a lot about discovery. It's not, it's not like, I know it's going to come out ahead of time. Yeah, I'll condition it and, you know, mess around and make a lot of mistakes. And sometimes just say, hey, well, that mistake was perfect. Yeah.

Scot GL: 5:23 Well, I mean, listening back, I went through I, you can find Chris's page, a whole breakdown of sonifications . So there's about let's see, what nine 910? Something like that. Yeah, it's at least a close to a dozen pieces that you've done over the years since ping. So let's see. Well, we might go through them chronologically. What happened after ping ? Oh, dear. Yes, we can look it up. Okay. So

Chris: 5:48 So Ping was, you know, real time flows in the internet. And then the next thing that happened was a piece called "oxygen flute" in 2002. So how that came about and and again, you know, this is this is Greg and I realizing we're enjoying working together and is very fruitful. Greg had a kind of a, you know, in a way this this homage or memorial in mind to a very tragic thing that had happened that year, it was a number of immigrants into Great Britain died in a container. Ship container. Yeah, they're kind of big container. Because they were squashed into half a container with another load, you know, the other half of the load being tomatoes. Well, they suffocated. And, you know, the, there are many aspects to that tragedy, you know, but the one that in a sense was was memorializing, you know, just how bad it was, was looking at carbon dioxide, okay, carbon dioxide, suffocated these people. On the other hand, carbon dioxide, you know, when you think of the gas exchange that keeps us alive on the planet, yeah, is one half of the equation with oxygen, right? So, again, you know, extrasensory perceptions, we aren't aware of the gas exchange that keeps us alive. And what we wanted to do was make that perceptible, ah, so, so we built a growth chamber full of bamboo that you could walk into, and this was in the museum. So yeah. You know, the sort of rubbery neoprene walls that that were translucent, contain, you know, large standard of bamboo and a gantry, that you'd walk up and stand in the middle of this, this forest, kind of clammy and you'd feel the planet , but it had a very, very accurate carbon dioxide gas sensor that was, you know, taking readings at some very rapid rate, I don't always millisecond or something like that. It was, you know, high quality, we could literally look at the data and know if someone was in there, or if there were two people or three people exhaling . And that that project. You know, again, was was kind of inspired by this tragedy, as I mentioned, but but not directly yet. You know, sort of making a kind of statement, you know, in terms of the this we all know about,

Scot GL: 8:51 Like your inspiration, but not actually?

Chris: 8:53 Well, you know, I mean, I think what it is was it captured, Greg, you know, and if I've got the Chronicle, right. I may, I may, I may be off by one project here. Yeah, we started working with co2, and then eventually with tomatoes, right,

Scot GL: 9:08 I saw the tomato pieces and these, they actually sound fantastic.

Chris: 9:13 Yeah, but, you know, this, this, this notion of individuals losing their lives, because we don't understand this stuff, and we don't understand how they became trapped by their circumstance. And we don't know. Yeah, things we don't understand how you know,

Scot GL: 9:31 Well co2 might become a thing.

Chris: 9:33 Well, here we go. Yeah, call it a growth industry. Yeah, b ad. Yeah, but, yeah, a couple of my daughters are actually scientists working in climate change, research. Also related to public health, and that is where I get, you know, you know, week to week information about just how dire the situation. We're in You know, I mean, I, I get it firsthand from some experts in the field who I'm related to. Yeah. But we're all we're all aware of it.

Scot GL: 10:07 I mean, what is weird anomaly in the North Pole? I mean, there's a temperature spike. That's just off the charts and unusual. And I guess from what I understand the North Pole itself is no longer. They're not mean it's just water. Yeah, it's a big lake.

Chris: 10:23 Right. Right. But But even in our immediate surroundings, I mean, everyone has stories, right?

Scot GL: 10:28 Yeah.

Chris: 10:29 And what's really interesting is you go back to your, say, your grandparents stories. My wife's father used to go out and drive on Long Island Sound in the winter,

Scot GL: 10:44 because it froze over.

Chris: 10:45 Can you imagine?

Scot GL: 10:45 Oh, yeah. You know he was nuts, anyway, but

Chris: 10:49 yeah, right.

Scot GL: 10:51 Yeah, he was a teenager, for sure. Yeah. But, you know, that's, that's an interesting, just to segue back into the sonification area, because I, when I talked, I had a discussion with Carla Scaletti. And she gave the keynote at the ICAD this last year, which was a fantastic talk. I wasn't at this conference, but but it's up on YouTube. And one thing she and she corrected me in the middle of the interview, I was talking about something like these pieces. He said, Well, what you're talking about, though, I would call that music and not sonification. And it was like this weird moment for me, like a light bulb went off. You know, she calls it like a three legged stool. There's like, vocal, and there's music, audio, and then there's sonification audio, which hasn't really been recognized as a separate thing. It's weird. I mean, because because I mean, just from what we were saying before about how we kind of backed into doing sonification just because of an interest in our own craft and working as a compositional determinant, that kind of stuff. But so I've been giving this a lot of thought it's like, what and talking to Mark Bollaro, Ballora sorry, who, who had sponsored this latest ICAD and done a lot of work in the area and stuff. And I thought for sure one of his cardiology things for the heart had actually scaled into being used as part of medical sonification thing. And he told me, no, that hadn't happened. And I haven't found anybody who's consciously tried to bridge the gap between the art practice of sonification and, and the actual functional thing. And so one of the things well, do you have any thoughts on that?

Chris: 12:36 Yeah, sure. Yeah. Having having bumped into that? Yeah.

Scot GL: 12:41 Yeah.

Chris: 12:41 Yeah, bye, bye, bye, good fortune. Fellow professor at Stanford, in the medical school Josef Parvizi is a neurologist and he's a, you know, with a clinical practice, but he's also a, you know, a scientist who's working on, you know, many aspects of, you know, neuroscience and cognition and so on. But on the clinical side, he works with epilepsy, and treats, folks with, you know, various forms of, you know, seemingly, you know, intractable problems and in, you know, seizure related disorder. Josef contacted me with the idea of listening to brainwaves that are registered from or recorded from patients, you know, who are in the clinic, well, in in making that sort of leap, and this is totally to his credit, he's he's someone who's very, very aware of, and appreciative of art, music. And promotes that in his own life and for other people and had been at a concert. And this was a concert of the coronas String Quartet playing a piece by Terry Riley called sun rings,

Scot GL: 14:11 Right. I know that piece. That's another sonification. I should talk to Terry actually.

Chris: 14:18 And, and literally, that's, that's, that's how this project started. That was his inspiration. Yeah, he was really enjoyed the concert.

Scot GL: 14:26 Now the piece you're talking about, it's called brain stethoscope. Right?

Chris: 14:28 Well, so so it turned into that. Yeah, that's, that's where it went. So he was, you know, he was kind of intrigued by the fact that, you know, this is data from outer space. Yeah. And what about, you know, maybe we could listen to data from inner space. Essentially, which the data is the kind of data that he works with all day long,

Scot GL: 14:48 Multi channel e.e.g.

Chris: 14:50 Exactly, giant strip charts. You know, many, many channels all you know, wiggling

Scot GL: 14:55 128 for some reason.

Chris: 15:00 So for Jopsef, if you know there was this curiosity, you know, what would it What would it sound like? Would would there actually be some sort of signature a way to recognize? when someone is seizing just by ear, you know, and he was absolutely right. So we were, again, a project. And now back to you know, what, what you were talking about with Carla

Scot GL: 15:24 Did it scale is the question? .

Chris: 15:25 Yeah. So, so the project, you know, this is now back several years. I mean, he contacted me in 2009. You know, after having heard "Sun Rings", and yeah, we got involved with some datasets from, you know, you know, some of the clinical work that he was doing. And immediately that fork in the road sort of present itself is like, Okay, are we doing this for, you know, just for, say, my purposes, which were like, I'm just totally getting off on the rhythms I'm hearing. I've never heard any rhythms like this in anything I've ever done it. And they're real. I mean, they're their music. And their This is a new frontier and music for me. And, you know, the more I'm delving into it, and kind of developing translations of data to music, the more intriguing I'm getting. I even sort of coined a word that doesn't exist in the English language. I think I've started talking about "musification" , because I was really using that term.

Scot GL: 16:32 We should talk more about well, this whole idea of standardization actually is where I was, where I'm heading. At least our research, okay. Because I think the only way it's going to, I mean, the techniques that you're using with the what became the Brain stethoscope From what, I understand that that's what you call that piece or right?

Chris: 16:51 Yeah, well, that was a device and yeah, I'll finish the story. Yeah. So anyway, yeah. I mean, literally, you know, I'm sort of exploiting this. For for my own devious musical purposes.

Scot GL: 17:05 Exactly. Yeah.

Chris: 17:06 And on the other hand, we're starting to realize that, indeed, you know, it's, it's something that's intelligible. You can actually know, when someone's seizing or not by Yeah. You know, once we've sort of kind of shaped the translation.

Scot GL: 17:13 Is there is there like a training learning period of listening? Well, for example, when someone does a stethoscope on your heart? Not everybody can put as doesn't go on and know what they're listening to?

Chris: 17:33 Yeah. Well, it turns out the just this sort of binary thing of like, is the patient seizing or not? Yeah. Is is we're hitting the 95% accuracy with almost no training. So it's just pretty clear. So I mean, if you look at the brainwave. Yeah, in this case, it's pretty clear to Yeah. Very rhythmic

Scot GL: 17:57 Like an alarm going off,

Chris: 17:58 but but hearing it actually hearing it directly in real time, gives you an instant readout on on that that,

Scot GL: 18:07 Are there any cursory sounds that lead you to believe that an epileptic fit is about to happen? Or anything like that?

Chris: 18:15 There's often a very high frequency tremolo for about 15 seconds ahead of time.But don't don't quote me on that. I'm not a neurologist. But yeah, I know, a few times, and it seems

Scot GL: 18:31 well, right. There is another big problem is all these things I've been calling them, like boutique projects, because you get to know the scientist, you kind of half ass get to know something about the science, but you can, you know, like, I'm not a, you know, a clinical neurologist or whatever, you know, there's just no way I could get that kind of understanding and still have the understanding of music and, and computer music techniques and all that kind of stuff. You need to do this stuff.

Chris: 18:56 So team effort.

Scot GL: 18:57 Yeah, it's a team effort that which is great. I think it's the new near New Era. I'm proud of the work with the hub, which is sort of a kind of a version of that. Yeah, they're all separate composers, but we were able to do a cooperative

Chris: 19:11 The only way you could get there, right?

Scot GL: 19:12 Yeah, right is the only way you can do computer network music anyway. And, you know, there's, there's ego things that happen and all that kind of stuff. That's true of any anything, you know, but

Chris: 19:12 uh, but but we're actually

Scot GL: 19:24 Yeah, you're actually making a thing.

Chris: 19:25 We have a device in a company manufacturing the device sale and we cleared FDA trials and we're, we're offering something that's proving out very nicely in practice.

Scot GL: 19:39 Oh, great.

Chris: 19:40 So we call it brain stethoscope. The company's called "Cerebell" which is C E R I B E L L. And I'm learning a lot about the medical device profession, you know by hanging in there with these, this team, which is now I mean, they've done all the heavy lifting Yeah, well concept,

Scot GL: 20:01 but I mean, it takes I, you know, this is my advocacy for the whole thing is one of the problems. The reason why this is not SIGGRAP, ICAD is not SIGGRAPH. It has to do with this integration of the arts you know, or at least a craft I mean, maybe not taken so far as being art, but, but there's a craftsmanship that needs to be present. I mean, when I was talking about Michael Winters thing with emotions, you need to know that the resolution of a German six chord is going to have a certain sort of emotional contour or something, you know, not necessarily but you know, you know, kind of things I'm talking about. That's certainly true. And Terry Riley's piece and in a lot of your pieces, you you hear the well, all of your, your this musicality that's coming through. So well, that's, that's, that's important. I mean, yeah.

Chris: 20:47 But I think that's all we can contribute. Yeah, as musicians.

Scot GL: 20:51 That's a part of the team that can play and but that stuff is getting back to my idea of standardization has not been standardized.

Chris: 20:58 Right? You know, and again, ever could be I mean, I'm really curious about

Scot GL: 21:02 We are working on these trainings. Right now we're working with the idea of ratio. Just how do you use sound to convey ratio. And I actually was working with a neuroscientist with his DARPA grant last year, and Gagan Wig, we did a bunch of versions. But one of the things that we did that, I'd love to hear your feedback on, what you think about this, is we had one set of data with one visual representation, which was a circus diagram. And then we used five different techniques of sonifying it. So the user could choose, you know, whether it was a chord, whether it was a melody, whether it was EQ of white noise, sure, yeah. EQ of Actually, I did the Ramones. I want to be sedated, just as kind of a joke. You could hear the you know, the different equalizer, you know.

Chris: 21:49 So, yeah, you make it a palette that people can

Scot GL: 21:53 Yeah, yeah. Just like, what's his Gardener had that thing about the different types of learners? There's probably different types of listeners, that would be attracted to different types of listening. Yeah, I don't know. Yeah. Yeah.

Chris: 22:06 Well, at one point, there was a variation of the old ping idea that they found out to NASA because they were very interested in sonifying in real time, a complex network situation and very heterogeneous network in what they do every summer, at least back in the 2000s. It was a summer project where they simulate a manned mission to Mars, you know, like, so this is, this is Yeah, this is people walking around and the great meteor crater in Arizona spacesuits on and there's a trailer nearby with one of the most complex networking things, you know, like, Wi Fi to the spacesuits up links to the satellites. landlines ...hardlines. Yeah. And they have their hands full. They said, Oh, well, what if we just, you know, play ping on one of the channels in the room, you know, so that we could know if things are stalled or not.

Scot GL: 23:04 And there's a change in the sound of the network. So

Chris: 23:07 Yeah, exactly. So so that was that was their kind of like, you know, auditory or conscious back channel thing? Oh, yeah, actually, they change. But then it was immediately a question like, do they want to hear my music? Or do they want to hear like Country and Western? Yeah, right. Well, I got into that space, I was influenced by reading the "User illusion" Tor Nørretrander Norwegian philosopher guy. I heard I heard about him from Will, Wright. But anyway, this got this idea of exformation is the idea that if we go "shave and haircut rif" we're not, you know, you know, maybe not transmitral. But at least in our culture, shave and a haircut two bits, you know, everybody's gonna do the two bits in our head, you know, so. So you could take normal music and just do something like this ping thing that you were doing on top of Country, Western, you know, or raagas or India or whatever, I know. But anyway, I .. these are but you know, the funny thing about it is and, and you are probably one of the most if not the most advanced Sonifier, and you know, that that was well I've got the you know, in terms of like getting something out as a product and like really working with it for now, decades. And yet, every time I talk about it just feels to me like it's just the beginning. Like if we knew what we were doing would we call it research? Yeah, but no, literally, it's just a garden of fun. That I just keep exploring new corners. And, like right now, I'm trying to get ready for performance in a couple weeks in a nuclear reactor in Stockholm, where Okay, That's already kind of peculiar, right? Okay, I'll say it's a decommissioned nuclear reactor with an audience inside, okay, safe, but to make it even sweeter, there's a Wurlitzer organ in there.

Scot GL: 24:43 Now,wait why is there Wurlizer organ in a ... ??? Wait a second, leave it to the Swedes, I guess

Chris: 25:19 This is a great project. This is a friend of mine, Leif Hanbury, who's, you know, had the insight to ask his university about that hole in the ground, you know, like, yeah, you know, a couple decades, whatever this was, you know, dozen years ago, but it was a, you know, it was a research reactor built in the 50s, operating in the 60s 70s 80s. That was doing research on you know, neutron physics and the, this is at KTH the Royal Institute, right. So, Leif asked if he could get involved with producing, you know, events down there, turn it into a venue after cleaned up, and he's been doing that ever since, and a few years ago. One of the original 1920s I think, or organs Yeah. had that had,

Scot GL: 26:18 Like the big theater organs?

Chris: 26:19 Yeah, removed from, you know, the fantastic theater, I'm sure. Well, you know, it was

Scot GL: 26:27 There is still one in the Fox theater. I mean, in

Chris: 26:29 the Stanford theater has 101. Downtown

Scot GL: 26:33 Does it? ... in Oakland, the Grand Lake. theaters has one. A Friday night at seven. You can watch them the guy plays and

Chris: 26:41 Does he come out of the ground?

Scot GL: 26:42 Oh, yeah. The whole nine yards. Yeah. It was one of those.

Chris: 26:45 Well, yeah, with all the pipes and stuff. It was in an attic somewhere. Well, actually in the town hall, I think. And then, you know, lovingly restored by some, you know, absolute dedicated, you know, work in the reactor itself, like, you know, and you can imagine the thickness of these walls. Yeah, seven storeys down. So what are the

Scot GL: 27:13 What's the acoustics like?

Chris: 27:14 Well, it's Cathedral. Oh, yeah.

Scot GL: 27:16 And I'm reminded of Pauline and Stuart Dempster. Exactly, yeah. Right there. Yeah. The the Cistern piece?

Chris: 27:24 Yeah. Yeah. What's

Scot GL: 27:25 the What was it? 43 seconds. You could go and hear something for 43 seconds.

Chris: 27:31 This one. This one's pretty long. Yeah, it's got really hard walls and a lot of granite, you know,

Scot GL: 27:38 particularly good for telematics stuff. Since there's no one.

Chris: 27:41 There's no one.

Scot GL: 27:42 Even even in this kind of acoustic space. There isn't I mean, you can hear events, but then they turn into a wash.

Chris: 27:48 So I digress. So what I'm working on

Scot GL: 27:50 Oh, yeah. Sorry,

Chris: 27:51 is trying to play this organ with with a simulation of nuclear fission. Right. So, I mean, you got an organ, it's in a reactor, you might as well be in the core. Right. So yeah, let's try to get the thing going in a chain reaction, right. Yeah. And I found him fantastic. This is this is all about, like where we are today. I mean, I, I was able to just say, you know, conceptually, I would like the organ to be the reactor core. And I told this to Leif. And he goes, Oh, well, that's funny. We, when we rebuilt the housing, we actually reproduced the color, the shape and the size of the reactor core, and put the organ inside. I have a picture of that now. So So then I'm going, Okay, where do I get the, you know, the code that will like sort of

Scot GL: 28:46 Play the Wulitzer

Chris: 28:47 , or at least Teach me Well, what a reaction dynamics looks like and I found a beautiful project in I'll get the name wrong probably but it's Alex Villar Stein. He's a professor at Stevens Institute of Technology. Oh, and online.

Scot GL: 29:08 I played a concert there.

Chris: 29:09 Have you? Yeah. Well, this guy put out for the public, a just absolutely beautiful, interactive application where in JavaScript actually written originally in processing. Ah, so I kind of reversed it back into processing because in processing, you can emit OSC. And eventually that OSC is going to get to Stockholm and play this organ, which I didn't mention had been converted to MIDI on the restoration process. So all that in one big sentence. I will be playing nuclear reactions on the organ. And the music is kind of in progress right now. Yeah, a couple more weeks.

Scot GL: 29:57 Well, and then there's the question of quantizing. I mean, what scales Do you choose and

Chris: 30:01 Exactly

Scot GL: 30:01 You know, is it all chromatic to be?

Chris: 30:03 What's the translation?

Scot GL: 30:04 Yeah, well, you know, not everybody wants to hear. Yeah. Although I've been going through Webern's piano pieces, just trying to wrap my head around what was going on with Stockhausen's Klavierstucke and all that stuff. I mean, this very chromatic, mid century 20th century music is kind of so enigmatic now, and not really something that caught on.

Chris: 30:25 Yeah, although my ears are still sort of there.

Scot GL: 30:28 Yeah. me too

Chris: 30:30 Yeah, I have in and out of tonality and not you know, and,

Scot GL: 30:35 but that really informs these choices. Because I mean, if you put it in like Gypsy, you know, like, C, D, E flat, F sharp, G, A, B flat, you know, you get this kind of like Django Reinhardt ish, instant thing out of any random, you know? That's a real cheat that I've heard a bunch of times. When sonification things were they Yeah, the node, the mode that's used. Right. But then there's that interesting thing. Have you heard of Alexandra Super by any chance? Alexandra Super had this idea of the the audio sublime, which is something that's holding sonification back compared to visualization. Her theoretical, I'll send you the paper, she tried to interview her son. You should. Yeah, there's no timescale. Yeah. But anyway, but she has this idea of even scientists that are working like like your friend with the neuros, or the neurologist that's working with that once he's he passed this, but a lot of scientists think it's a novelty. And then it's just gonna like kind of trivialize the seriousness with which their collegues will take their work. And that kind of stuff, because people are expecting something sublime out of the musical gestures that come from data. And this is like a really problematic thing. because not everything is going to sound beautiful, right? In the same way that not all bar charts look good or something. But we know what they are. We don't need to you don't need to have the standardization thing.

Chris: 32:14 Right, right. Yeah. And, you know, everyone has their own sort of, like, spread of what they call music or will be able to, you know, open filters and then start to become music.

Scot GL: 32:26 Yeah, right.

Chris: 32:28 I, you know, I find, not just among scientists, but you know, it's, it's a changing world that more and more, the ears in our ears are open more and more I can do just jam sessions with people that I've never met before. And we have almost a common language, which, which is a language that I thought was extremely esoteric back in the 70s. When I was starting when I was into free jazz. Yeah. Right. It's like, but but you're improvising on your cello. Yeah, I can play by Yeah, yeah. cello or bass? Yeah. Electric. Yeah. You know, but, but the fact that I can just start making music spontaneously with people around the globe, and we don't have to talk about style, you know, I mean, well, I mean, it's there.

Scot GL: 33:17 I'll refer back to Pauline Oliveros, who's that whole idea of deep listening if you get into that space, where you're really listening, but there is this kind of common language. I think you're right, that I I always think like when when, like Tom Duff, who works at Pixar, but is a big fan of and supporter of new music, and has these house concerts and, yeah, yeah, they're great. Especially cuz Susan, his wife puts up a jigsaw puzzle. There's nothing like listening to free improvised music while you're solving a jigsaw. I love it. But anyway, I digress. The main thing is that all my experience of the pieces there, this is just my own personal take on it, though. But there there is this moment when everyone knows when to stop when they're really good improvisers. And even everybody in the audience knows that, you know what, I've done the right thing. And it's like a really clearly momeny. Yeah. And, you know, oh, man, don't go. bad situation. There's the it goes back to what I would characterize as the beautiful death. Well, you know, we can go on and on I so we're all we're a lot of those based around the co2 of volatile organic compounds sort of sensing.

Chris: 34:37 So so there's a Yeah, there's a series of things. Yeah, that came out of actually, you know, have to think about the gas sensors themselves. Yes. After "Oxygen Flute", which used this co2 sensor that right there was a kind of, you know, like, you know, very scientific grade publication quality. Yeah, you know, several $1,000 instrument. The next thing that happened was we were going to work with tomatoes directly. And this is again, referencing what I mentioned.

Scot GL: 35:10 Yeah, that story.

Chris: 35:12 Yeah. And for that one, Greg and some students and colleagues built the sensors from the ground up, you know, and so these were these were, let's say artist, inspired, designed and, you know, pretty much executed instruments, they are affordable, replicable and worked really, really well once we, you know, started using them. And we did. Let's see, we projects. There was a, an installation of the Machine Project Gallery in LA, right.

Scot GL: 35:48 Which closes I heard the machine gallery

Chris: 35:51 No kidding.

Scot GL: 35:52 Yeah. Last year, sometime they blow I could not stop. Sorry. But yeah, be sad. Yeah. I did a couple projects. Yeah, it was very supportive of this kind of edgy. Yeah, yeah.

Chris: 36:04 We're just talking about them the other day. Yeah. Um, so yes. Anyway, this was sort of an artist's, you know, still life but written as a multimedia piece. Because there's a bunch of tomatoes. Yeah. That's it. Yeah. Right. Yeah. Right. You're listening to a bunch of tomatoes. Well, turned out there was a lot going on in those tomatoes.

Scot GL: 36:25 Well, that's, you know, I've just, this summer did a piece with my friend Bert Barton talking trees, which was using voc sensors and sap flow sensors, moisture and all that in a live concert setting with a big tree with a giant cedar, you know, just happened to be where about age, right? Bear? Another one in the sub near Eindhoven. Yeah, we're gonna do more of these. I'm going to bird is very active. Well, the cool part of that piece and it has to do with this improv thing was the voltages that ended up coming off of the conversion of the live sensors on the giant tree went directly into a big eurorack synthesizer that I was using. Yeah, so there was no computers at all right, just the voltages. And then I was sending those across the stage to this guy Tice, who had a giant Moog. You know, old style modular, three P, but more more than three, but then I got started that style of mode. It was a very interesting, and then Bert was playing data that he had a massaged using common. He was using Peter Stone's, symbolic composer to take the data from a tree for every month. And it was I was

Chris: 37:43 The role of the tree. And this is grand. Yeah, amazing.

Scot GL: 37:47 Well, it was the weird part was like the interaction of, you know, being there. You know, there's like, 1000 people out there. And you go, and you don't know if it's gonna go up or go down. I mean, I don't mind me playing with the hub.

Chris: 38:00 If you had this one member that would just never take a bow, right? Yeah, they're producing

Scot GL: 38:06 Doing a lot of stuff. See what the tomatoes I mean, I can hear it in a tomato music, which I found very compelling, the way you realize that, and then what to do with that, right? I mean, there's a million ways to turn these stuff into this garbage or something interesting or over music. You know, the musification thing is can be troubling. I mean, can you can over get over and over and over musical.

Chris: 38:28 Yeah, we all just follow our own ear. Yeah, let's see. So sensors, I guess, you know, I should I should mention that Greg's team's sensors actually started to become foundational to technology that got going outside and became a device, you know, oh,

Scot GL: 38:53 these are co2 sensors?

Chris: 38:54 Yes. You know, co2, voc, lights, sound they Oh, yeah. So

Scot GL: 38:58 yeah, the same Deaf stuff. This guy. Like, I forget his name. but he made these same things

Chris: 39:07 It's very, very cool. And as those progress let's see at some point, point I got deployed in various places. I haven't I have one use of them. There's still a live website, that Greg program the website, it's called "Seven Airs". Okay. S E V E N. A I R S .com.

Scot GL: 39:25 Okay."Seven Airs". Okay. Yeah, check it out.

Chris: 39:36 So it's it's that same category of sensor, which, which were then taken by a TV team, from Japan to seven different locations in Central California. Their their TV project had to do with a show that circumnavigating the globe at 35th parallel 35th. Okay, yeah. 35 into one. Yeah. So 35 gets you to about Pismo Beach like north of LA, right? If I've got the right number of degrees here.

Chris: 39:07 class. It's very, very cool. And as those progress let's see at some point, point I got deployed in various places. I haven't I have one use of them. There's still a live website, that Greg program the website, it's called seven errors. Okay. SEN. A ir Okay. Certain errors. Okay. Yeah, check it out. So it's it's that same category of sensor, which, which were then taken by a TV team, from Japan to seven different locations in Central California. Their their TV project had to do with a show that circumnavigating the globe at 35th parallel 35th. Okay, yeah. 35 into one. Yeah. So 35 gets you to about Pismo Beach like north of LA, right? If I've got the right number of degrees here.

Scot GL: 40:12 Well, I know 33 is Orange County. But that's only because I know this guy.

Chris: 40:18 And this is totally right on.

Scot GL: 40:19 Yeah. this other guy is doing the 33rd parallel. He's a wounded warrior. Vietnam guy. I mean, not the Iraqi guy. Well, anyway, that's the only reason why I know 33. Yeah, because he's doing the same thing with other at risk. Iraqi veterans going along the 33rd parallel and collecting the air in the style of Duchamp had that vile of air.

Chris: 40:46 Wow. Wow. Okay. Here's his production team . So they said, Okay, we're gonna do a, you know, a piece on where it comes on to the, you know, west coast of North America, California. What Well, let's see we need a theme for this. You know, when we did Malta we did you know, the the the Crusades was California will? Well, let's do pollution. You know, of course, that's everyone's travelogue for Yeah, California. Yeah. Ocean, right. Yeah. But then they kind of look looked us up. And because we have these sensors, and you know, sonifying the data. So they actually, this is a wonderful project, because they, on their own, they took, took our rig, went to LA went to a feedlot in the Central Valley went to an oil patch, where they're pumping stuff out on that went to the desert went to the beach, and and when you go to the website, you'll you'll be able to go to one of these sites, and then you will hear and see is, you know, a quote, "real time" now we're playing back a recording. Yeah, well, but but it's, you know, it's fake real time. Yeah. So you're hearing is a recording of, say, 15 minutes from, you know, the beach, and you'll you'll, you'll catch the change in carbon dioxide change in VOC and you'll hear I really market difference between, you know, those those sort of, you know, places that are open and you know, desert beach and feedlot probably, yes. versus a feedlot. Yes. LA Yeah, they got into trouble down at the oil patch when they were doing this. You know, he also has a great story. Yeah. make for good TV. Yeah. But in the end, it was also about like, again, that that question of like, you know, should it be my music? Or can we reach the audience better? If Yeah, if we turn it into something recognizably something so for that we actually play games and yeah, with the, with the LA translation, or maybe from the beach, also, on the TV show, we ended up doing something that you won't see on the website, but I took Hotel California, ah, because they really wanted it. And I use

Scot GL: 43:20 The most played song in the world, I think, honestly, perfect. It's one of those perfect, yeah, okay.

Chris: 43:25 Well, we hear this way. So we programmed a third harmonic distortion thing that was driven by Ah, the pollutants

Scot GL: 43:35 Was that Faust?

Chris: 43:37 Those days, probably Chuck. Yeah. But anyway, yeah, you know, so you hear this thing gets really, really distorted in LA, and it sounds kind of cool on the beach. You know, it's like a pedal.

Scot GL: 43:48 Yeah. Yeah. Right. Well, this is definitely in that what I call this exformation zone, you take something that's familiar, and you distort it with the data,

Chris: 43:57 If that makes sense. Yeah. Yeah.

Scot GL: 43:58 It's one of the one of the things I want to try Southern realizes. So his pitch follow a known melody, and harmonize it. Yeah. based on the data. No, no, with another voice. That's not in the hotel, California. That right. high treble sound. Whatever. Right. Yeah. But anyway,

Chris: 44:15 Yeah, but but we did learn that, you know, by by listening and seeing the graph, you know, this basically, you're following that? Yeah. The, you know, some indicators of where you are in the graph as you're listening. And that's super powerful.

Scot GL: 44:31 Okay. Okay. Now that there's a point and this is came up in an email this morning from Roger Malina, where he is boiling down. He's about to give some presentation at IBM, I think it's going into town and whatever. But he was saying, what we really want to do is figure out a way to make sounds that not only that do something that you can't do with a visual graph, and I think that's true. I mean, if you know where you are in the graph, and then you hear something, you get all the information about the graph. Then you get some kind of multi dimensional subtlety at that point of the visual graph. Maybe. I mean, there might be a way of proceeding.

Chris: 45:08 There's something really potent about the two together.

Scot GL: 45:11 Yeah. Okay. Well, I'm glad to hear you have the same experience.

Chris: 45:14 We're actually running an experiment and see if we can if we can actually measure the difference, this point, as well,

Scot GL: 45:25 Like a UX kind of study?

Chris: 45:27 Yeah, a little bit. Up at UBC, in Vancouver, there's a psychologist who I'm working with in her. Her, you know, her kind of area of psychology is basically climate change communication. And you know, how how we can better sort of, you know, message Oh,

Scot GL: 45:51 What's her name?

Chris: 45:53 Yeah, Jean Michel, and I can I can I can make a make a connection if you'd like,

Scot GL: 45:57 Yeah, that'd be great. I mean It sounds like so.

Chris: 45:59 So what she's done is take some techniques that she's already or methods she's already found to be pretty, pretty useful. Yeah. And what we were using those as kind of like, essays on surveys, which, okay, so I guess I'll explain that the data sets come from, you know, recent, kind of, you know, topical climate change data sets, right. Okay. You can see co2 in the past 100 years, you can see Arctic ice sheet coverage in the past 30 years, you can see, you know, so yeah, so several of these data sets. And we are presenting them first, just as graphs, you know, with a with a silent, you know, index. So like and then that coupled with a sound. And then we'll also do sound alone at some point, but we haven't yet. And we want to see to the extent that people can kind of like, predict where it's going to go and talk about the degree of severity of change. how memorable it is. And also, what she's very, very experienced in doing is finding out to what extent that is communicating some sense of urgency to the to the person taking the survey, right?

Scot GL: 47:33 So is there like eye tracking? And that kind of stuff

Chris: 47:34 There's eye tracking? Yeah, there's survey questions like, you know, the, you get a certain amount of funny money that you can grant to a cause, or you can use it to go, you know, buy your dinner or something. Oh, yeah. making choices. Yeah.

Scot GL: 47:51 And what's your motivation? That Was there some change in your belief system? Is that? Yeah, you got Oh, that's cool.

Chris: 47:57 So so we're doing in the that area,

Scot GL: 48:03 We are gonna have these training modules for listening that we're trying to get set up for UX. It's still in the early days, but what by but all on one other thing I'll just bring up is this idea of using Jupyter notebooks. which are these ipython notebooks of using for training initially, computer science for this kind of spread? There's like 6 million users now. Yeah. What do you pick the efficiency thing?

Chris: 48:27 Do you want to get out of the sun?

Scot GL: 48:28 Yeah, I can see. It's coming down. So great, being near Lake lagunitas. My childhood swimming hole. Anyway.

Chris: 48:42 Yeah, so the Python Yeah, that's just your notebooks.

Scot GL: 48:46 The nice thing about that is the thing that we were discussing earlier was the idea being not to become an expert in all the fields that you're interested in sonifying, but being an expert in the sonification part, you know, sort of the graphics department doesn't need to know what the Capital One finance, or Bank of America, whatever, they don't need to know all the ins and outs, they just know, they make a graph. It's gonna work for them, you know?

Chris: 49:09 Yeah. Yeah, that sounds right. You know, but it's always awkward when I make lots of comparison to, you know, the old days of the darkroom is, you know, maybe the digital equivalent, but you can, even when you take a picture, and you're going to share it with people, you know, or develop it as a as a work. You know, you put a frame around it, you heighten contrast, yeah, you know, this is all that, you know, stuff that goes on in our heads or whatever. Yeah, and it's very feels very analogous to what I do with data sets, you know, myself, you know, it's like, yeah, just the act of like, segmenting it and chopping it into a thing that we're going okay, here's a finite length.

Scot GL: 49:52 Well, something like the pulses, you're talking about the pulses of the ping, I why I almost so the time period Did you ever quantize that rhythm? So the know that a hit no 16th? No, no, I had one thing I was trying to do ratios. And so I had this idea of moving the boom chick, as the percentage " boom chink".

Chris: 50:19 No, no make sense.

Scot GL: 50:20 But what's weird is when you vary it like 52% or 60% suddenly just sounds like it's just off. It's because your mind is dividing it up into, into into 16. And when you lock it to 16, it's probably been 32nds, suddenly, if you're more comfortable with it, so in other words, the best resolution you can get is 16. divisions. And, of course, at the temp, what tempo to Yeah, it's just

Chris: 50:51 so there's just kind of perceptual quantizing that happen.

Scot GL: 50:54 Yeah. What is Oh, clearly all this cognitive stuff that needs to come into play in all this stuff, which is part of what we're trying to integrate in, in the training stuff as well.

Chris: 51:02 Right .. Yeah, fine. You know, I'm really fascinated by these rhythms that they kind of express their own sort of ratios, you know, and maybe they're pretty sloppy compared to what

Scot GL: 51:16 They are organic,you know, I got the chance of working with David tutors, neural net synthesizer, with Mark Holler. And that's here now right is Yeah, okay. So Mark is a good friend of mine. He said he gave to somebody.

Chris: 51:33 Tim O'Brien has it.

Scot GL: 51:35 Okay. Yeah. Well, I did a bunch of concerts. And actually, that concert ended Stevens. Nobody knew it. But at the time, I was doing IP packets to his garage in Palo Alto, playing the neural net synthesizer. And then he was sending it back to me over ice cast when I was playing it. I mean, there's too much to explain to the audience that we just did it. But the reason why that even came up was because of the organic quality of the rhythm and even the melodic structure of that neural net synthesis. You know, it's got this kind of real organic, almost expression,

Chris: 52:05 You wouldn't want to touch it, right?

Scot GL: 52:07 Yeah, no, contact. It's kind of like this problem with well, like, Ge did the I am Tpain app, you know? It harmonized. You know, they're tuned up your voice for shared that tune, I believe. And now it's on everything. And I just thought that'd be crazy, actually. Because this specivity gets squashed in. Hey, listen, I think we're I mean, we've done more than enough.

Chris: 52:32 I Oh, yeah. commercial break.

Scot GL: 52:34 Yeah. commercial break or something? Yes. Oh, almost an hour. Oh, my God. Okay, well, I probably try to go 20 minutes. This one's running a little long. Well, this is sound channel, flying the wall where we're gonna leave Chris Chafe's office now. And

Chris: 52:51 Scott, thanks so much. Hey, yeah, a really fantastic conversation.

Scot GL: 52:55 Yeah, I might have to break it into two parts. Okay, bye.
Outro tag 53:04 Sound and Data.
Transcribed by

Related Podcasts

Carla Scaletti Sonification Pioneer

with Scot Gresham-Lancaster/ 28 November 2017 / 1:30:20

Carla Scaletti is an experimental composer and designer of the Kyma sound design language and co-founder of Symbolic Sound Corporation. Her compositions always begin with a “what-if” hypothesis and involve live electronics interacting with acoustic sources and environments. The listener is encouraged to first watch Carla’s brilliant keynote at the 2017 International Conference of Audio Display if possible at This Link

Gregory Kramer on the beginnings of Sonification

with Scot Gresham-Lancaster / 16 September 2017 / 1:30:20

Gregory Kramer is a composer, scientific researcher, author, entrepreneur, and teacher. He is a founding figure in the emerging field of Sonification and published the first book in this area,”Auditory Display: Sonification, Audification and Auditory Interfaces” (Addison Wesley) The definition of sonification that everyone studying this field reads was written by Greg.

Featured Guests


Subscribe to our blog and get occasional updates on our ongoing work