Multimodal Dimensions of Sound Scholarship with Jacob Smith
Amplified is an audio blog series about the sounds of scholarship from our team here at the Amplify Podcast Network. This month, we’re joined by Dr. Jacob Smith, professor of Sound Arts at Northwestern University and author of Lightning Birds: An Aeroecology of the Airwaves (2021) and ESC: Sonic Adventure in the Anthropocene (2019). Together Jacob and I talk about tricky work of multimodal scholarship in practice from embracing the slowness of working with sound to thinking about audio right from the start of your project ideation.
-
TRANSCRIPT
Stacey Copeland: Welcome to Amplified, an audio blog about the sounds of scholarship from our team here at the Amplify Podcast Network, I'm your host, Stacey Copeland.
[theme music plays underneath]
[00:00:17] I first came across the term multimodal scholarship in a chapter by sound scholar, Jonathan Sterne, in the edited collection Sound Media and Ecology. Sterne writes, "while traditional scholarship privileges writing as its ideal mode of dissemination, multimodal scholarship might juxtapose sound and images with writing. It might produce non-linear forms of writing aided by digital navigation, or it might use media like sound recordings, videos, or games where writing takes on a secondary or supporting role" unquote.
[00:00:59] In reading this, I initially thought, isn't all scholarship in some way multimodal? What is the intent or purpose behind naming certain academic works under this multimodal banner? Maybe it was my media studies background or being a media producer, myself. I first came to this term with a bit of apprehension, but what I found over time was this distinction, this conscious effort to proclaim multi-modality as a scholarly endeavor carries with it a rich history of embracing the potential of sensory rich learning practices, and resisting traditional constraints over the form that our scholarly knowledge can take. In this episode of Amplified, I return to questions of multimodal scholarship alongside someone who certainly spends a lot of time these days thinking about and producing just these kinds of work. Jacob Smith's experimental audiobook, Lightning Birds: An Aeroecology of the Airwaves is a multimodal project that consists of five podcast style episodes, a curatorial essay, and a bibliography. It tells a story about radio and key scientific discoveries in bird migration, and it explores a mode of ecocriticism that combines traditional forms of text-based scholarship with sound art, music, and audio storytelling.
[00:02:26] The work is open access, which we love, and the winner of the 2022 Anne Friedberg Award for Innovative Scholarship. Lightning Birds is Jacob's second sound-first, multimodal work with Michigan University. Following ESC: Sonic Adventure in the Anthropocene, a work exploring CBS Radio Adventure Series "Escape" alongside the work of contemporary eco sound artists like Peter Cusack and Jana Winderen.
Coming up, Jacob and I dig into the tricky work of multimodal scholarship in practice from embracing the slowness of working with sound to thinking about audio right from the start of your multimodal project.
[music fades up and out]
Jacob Smith: [00:03:25] My name's Jacob Smith. I'm a professor and the director of the Sound Arts program at Northwestern University.
Stacey Copeland: I am so happy to finally have you here on Amplified Jacob. You've become quite an established figure really within the Sound Studies community. I, I'm interested to hear more about where your sound scholarship journey actually began. Where did that curiosity about sound come from in the first place?
Jacob Smith: [00:03:49] You know, as a young person, I became very obsessed with music and sound recording. This was the cassette era and there were some older kids in my neighborhood who had figured out that they had two cassette players and they could re-record something into one and then play it back and then overdub into another. And bounce back and bounce back. And that to me, was the most amazing thing I'd ever encountered. So from there on, you know, just very interested in music, I became a musician, kind of was in and out of college, dropping out of college to play music. These were the heady days of the nineties and indie rock and such things. So I was a musician and then went back to school as part of a Communication and Culture program [Stacey: Mm-hmm]. and you know, very film centric kind of curriculum. So I was always the kind of quirky figure in the program who was writing about records or radio or sound in the context of a kind of a media studies, film and television program. And it really wasn't until. A little bit later that sound studies kind of emerged as this field and I thought, oh, this is what I've been doing, this is what it's called.
Stacey Copeland: [00:05:00] Mm-hmm. Yeah. I think a lot of us folks in the sound studies world have had that aha moment. [Jacob: It's a great feeling]. You mentioned, yeah, you mentioned you were, are still a musician. I'm sure you still dabble. You can kind of hear that music influence in your ESC and Lightning Birds publications. There's a thoughtfulness in the soundtracks on both of those publications. I mean, could you talk a bit about how you got into publishing your work through sound? Cause I know you, you have more traditional quote unquote books as well, but these last two very thoughtfully engaged with sound, not just translating your work as an audiobook, but actually thinking about sound from the get-go in the publication process.
Jacob Smith: [00:05:45] Yeah, yeah. Well, I feel like I'd always heard, you know, in the world of film and television studies, I'd always heard publishers, senior, you know, colleagues, mentors, et cetera, talking about multimodal publishing. You know, I'd always heard people saying, we study film and television. You know, why can't we use clips from film and television in our publications? And so that had been something that people had been wondering about, thinking about talking about, for a long time. And so, you know, working in sound, you kind of wonder about the same things, you know, oh, here's a book that has an accompanying CD with examples, or here's a book that has a website with audio that you can listen to. So those kinds of multimodal ideas, I think had always been in the back of my mind.
[00:06:35] And as somebody who's, you know, before I became an academic, was mostly obsessively focused on making recordings of things. That was always, you know, a passion that I think maybe I want, you know, how can I bring that back into what I'm doing? I think another big factor for me was I'm very influenced by my colleague here at Northwestern Neil Vema, who you know from the world of radio studies and sound studies. Neil and I would talk a lot about using podcasts in teaching or just assigning more and more podcasts. And Neil and I both had the experience of when you're a teacher, you always have this, um, experience where you assign a reading and then you get the feeling that maybe not all the students did the reading, or maybe they just read the first few pages or something like that.
[00:07:21] But then when you assigned podcasts, they'd all listened to the whole thing, and in fact they'd asked for more. You know, so we were very excited about the possibility of using podcasts for teaching and was we're, we're folding that more and more into our syllabi. And so I think that was another thing that, got me thinking like, oh, this seems to really speak to students, listeners. Podcasting is booming. I'm very excited about podcasts. I'm learning a lot from podcasts, so it just seemed like a natural kind of next step. And then I think the third piece for me was just working in this sound arts MA program. Our curriculum is very much theory and practice.
These two things intersecting and just being very inspired by, students were doing with that and I, I think I got jealous. These students are having so much fun, you know, working on their theoretical and historical concepts in audio, I wanna do that too. So, I think all those things kind of conspired to wanting to take the big step and think about kind of major engagement with audio through my writing.
Stacey Copeland: [00:08:29] I wanna pull something out there. You were using the term multimodal scholarship, which gets thrown around a lot to kind of broadly encompass a lot of different kinds of publications that aren't the standard book manuscript, let's say. And you call ESC, a work of experimental audio-based scholarship. This is me just like reading, quoting off of the webpage, and Lightning Birds, a multimedia project and something I kind of pulled out in, in reading through the way that you set up both of these text is maybe attention around, how we actually describe this work and you know, you, you talk about podcasting as being one influence here, but when it comes to sound scholarship and putting our academic work into sound, it seems like it becomes a more complicated conversation.
I was wondering if you could talk more about your own thought process through how to describe these works. For example, were you drawing on other academic work or media work, and how did you come to define these two different books and in similar but different ways?
Jacob Smith: [00:09:35] Stacey, it's a great question. You completely nailed it. I was making it up as I went along, you know, it was hard to kind of pin down exactly what these things were or exactly how to describe them. And I think for me, one of the biggest tensions was, is this a podcast or an audiobook? You know, that was one way that I felt a tension, and looking at, you know, the work that other folks are doing, like you and Mack Hagood, you know, very similar kinds of engagements with scholarship audio. But what I kind of figured out over the course of working on ESC and Lightning Birds was I'm very, I was very influenced by a kind of a podcasting sound, you know, shows like Radiolab and the, kind of a pacing that I learned a lot from talking with, you know, podcasting professionals that taught me a lot about working in sound.
[00:10:29] So there's a big influence in kind of the audio and the structure of podcasting, but really, I think what ESC and lightning birds are, they're audiobooks because a podcast kind of suggests that this is an ongoing thing, you know, and you have seasons. You know, I have great respect for people that, that do that, but it kind of terrifies me because it's, it , it stretches out into the future in a way that I, I don't know if I could sustain, but also, um, I guess just the way I thought about these projects, it's one thing and it's got episodes. That are kind of like chapters, but they all drop it once and then it's done. You know? So, it felt more like an, in the end, I felt like I kept on telling people it's podcast. It's like a podcast, but it's more like an audiobook. So that was, um, one issue, I guess one tension that came up and that's one thing I'd still love to explore. I mean, one thing about multimodal online audio publishing is you could add to it, you know, you could, there could be a second season of ESC if, you know, if people wanted such a thing. You know, there's no reason, that website, the way it's held there, it couldn't get a new chapter or it could be added to in a certain, in a certain kind of podcast-y way.
Stacey Copeland: [00:11:48] That definitely hits at some of the questions I continue to linger with about the presumptions people have about certain formats. So podcasting being one, what are the presumptions that come with what a podcast is and how you're describing your work versus, say, an audio documentary versus an audiobook and I mean, how did that conversation go then when you went to Michigan University Press. And how did that opportunity come into play? This is one question, and then the second one is, how did you work with the press to decide how to describe and publish this work?
Jacob Smith: [00:12:21] I mean, the main thing to say is that I was very, very lucky to be working with Michigan and to be working with, at first my editor was Mary Francis, who I had been working with. [Stacey: Mm-hmm.] , you know, for previous book projects. And Mary Francis was just a great interlocutor. We had many conversations about what this might be, and I wanted to do something inspired again by my colleague, Neil Verma. [Stacey: Mm-hmm.] . I wanted to do something that was really about intensive listening. I wanted to listen to audio in a very close and detailed way, and so that's how I stumbled upon the radio series Escape.
[00:12:57] Those stories seemed to me to be so historically interesting and the genre of adventure was speaking to me. There seemed to be this interesting connection to contemporary wildlife field recording. I wanted to put those two genres in dialogue in some way or another, but I wasn't quite sure how, so Then Mary Francis and I just started talking about it, you know, and it's, it moved- Is this a book that we'll have some online clips that we can listen to now that seems kind of, you know, that doesn't seem bold enough. I wanted to go further with audio, you know, not just to have it be a little addition to it. I want it to be kind of centrally about audio. Maybe I'll re-stage these plays and record them and that's what it'll be. No, no, no, no. Maybe um, one model early on was like DVD commentary. Maybe I could have like one track that's the original radio show and then another track that's my commentary and you can, kind of control that you can toggle back and forth. That became tricky for a number of reasons, one of which was copyright.
[00:13:55] So these copyright ques like, oh no, you can't just, you know, repurpose the entire thing. You know, you, that might, you run, might run into trouble there. It went through many, many permutations before we settled on the format that we did, and Mary Francis was very patient and very open to anything. And then the folks at University of Michigan and their Fulcrum system, I was also very lucky because they, you know, once it was like, this is gonna be like a, you know, 90% audio, there's gonna be some accompanying text, but it's mostly gonna live as audio. I was lucky that they were like, yeah, sure, let's go for it.
[00:14:30] I mean, I think that they were very interested in the broad, multimodal, you know, what could multimodal publishing be? So I think for them it was, it was nice to, to, to have somebody like me who was, willing to give this thing a shot. I mean, the other thing about it was that it really relied on me doing all of that stuff. It wasn't like a commercial podcasting production company where they could give me an editor or you know, help me with the voiceover. It really depended on me being able to do all of that stuff. Kind of clumsily myself, but then they were fantastic about being able to support it all and think about how this would actually live online.
Stacey Copeland: [00:15:12] Yeah. Oh my gosh, yes. This plays into- I mean, I have a lot of questions, but I'll focus in on one. So we in a past audio blog had A.D. Carson on as well, who is also published with Michigan Press, but put out the first peer reviewed rap album. [Jacob: Huh great.] So it seems like they have a bit of a trend in thinking about some of these questions of sound scholarship. But you touched on trying to explore what was best for your collaboration with Michigan as well. So what would be on you to produce that wouldn't necessarily be on you as the author if you were doing, say, a traditional book manuscript, like you would have an editor working much more closely with your work and helping you produce the work in ways that you maybe have to do a bit more on your own, making a more experimental sound scholarship piece.
This kind of plays into the question I have around what were some of the biggest challenges in the publishing process for working with audio in ESC. And then what did you bring forward out of those challenges into then producing Lightning Birds?
Jacob Smith: [00:16:18] Initially, the biggest challenge was figuring out this different kind of workflow. You know, I mean, if you've been doing academic writing and publishing after a few years, you kind of figure it out and you know, you know how to work your Microsoft Word and you know what the revisions process is gonna be like, and you kind of, you got, you have a sense of the scale of an idea. You know, you have an idea and you're like, oh, that's kind of a 20 minute conference paper scale idea or no, that's, that's more of a 25 page journal article scale idea. , I kind of didn't know any of that. I wasn't sure what this was gonna be. I wasn't sure if it was gonna be like a standalone one hour radio documentary or if it, you know, I didn't, I didn't know what it was gonna be and I didn't know exactly what the work would entail.
[00:17:07] So it took many trials and errors to figure out just how to record voiceover. [Stacey: Mm-hmm]. As I said, I'd been a musician, but I was terrible about paying attention to what engineers and producers were doing, you know, where they were putting microphones and where they were plugging things in. So I was really clueless about all of that stuff. And I ended up doing like a whole reading of the voiceover and then realized, oh, the mic sounds terrible. Oh, you know, uh, did it again, but the room sound is awful. So I had to find, you know, I found out that many podcasters do their voiceover in the closet. [Stacey: Mm-hm.] so I had a great closet and I missed that closet that I, where I did my voiceover.
The one place where I brought in a professional to help me was for mastering. And you know, if you listen to those audio books, ESC, Lightning Birds, if you're listening to it as a radio professional, you're probably thinking, ah, it's a little, some amateur-ish kind of moments here. And I, I was really adamant that that's okay.
[00:18:07] You know, that's my academic voice is the kind of slightly wonky. Sound quality to it, you know? And the fact that I'm not a professional sound effects person, so maybe some things are gonna feel a little bit homemade, but that's okay. You know, that's my, that's where my academic audio work kind of lives. Um, but I did wanna bring in somebody to master it, you know, so it would be my writing and my words and my choices about things like music and sound effects. But then I wanted it to, sound good. And actually paying somebody to master it made a big difference. So I guess that kind of workflow was the biggest challenge, and then it was a little terrifying to, to move into the peer review process. Because, you know, as somebody who's not like a longtime audio professional, I was terrified, you know, if somebody had said, if it was just a written work and they were like, oh, you need to move this section over here and get rid of this paragraph. It would've been like, okay, yeah, no problem. I can do that in an afternoon. But if they would've said, get rid of this 10-minutes over here and can you shrink this? That would've been like days and days and weeks of revision. So the revision process was also different and maybe a little more time consuming. And that was something that I felt like both me and Michigan and Mary Francis, we were all kind of figuring that out together.
Stacey Copeland: [00:19:28] Well, first off, I wanted to share when I was choosing my new apartment, a large closet was very essential. (laughs)
Jacob Smith: I look for closets now too, and we're, you know, looking at apartments like, oh, well hey, that closet's pretty big. That looks good. Yeah.
Stacey Copeland: [00:19:39] So, yeah, I think that'll do. But I wanted to talk a bit about copyright because both of these texts, uh, these two multimodal pieces of scholarship have a lot of different very rich sound materials. So in ESC there is beautiful works by sound composers and soundscape artists like Jana Winderen, for instance. And then in Lightning Birds, we have a ton of beautiful archival work as well as environmental sounds happening. Could you talk a bit about how you navigated copyright as well as accessing a lot of those materials for your work? I think that's something people don't often talk about but is a lot of labor behind these kinds of works.
Jacob Smith: [00:20:21] Yeah. Yeah. And it's also a place where, in the world of the Videographic essay which is a another adjacent world that I'm kind of starting to dip my toes into a little bit. There's some good work there about the legality of fair use and using clips in this context. So I consulted with some lawyer friends for sure. I was getting permission, you know, Jana Winderen, folks like that. Those were, I know I would reach out and get permission.
From all of them to use portions of their work, all the sound artists and esc. Fair use covers a lot, especially if you're using clips to talk about them. And you're just using small portions, not the entire work, but also thinking carefully when you start a project. Like I said, you know, one of the early models for ESC was that kind of DVD commentary, but then those copyright issues came up and it was like, no, okay, you need to rethink that.
Stacey Copeland: [00:21:11] I wanted to bring that forward because it's something I feel like when I'm talking to folks who are interested in sound scholarship, it's something they don't start thinking about until way too late in the process. And it's something again, you mentioned you wanna think about right off the bat and bringing in sound right at the beginning of writing and thinking about the form your research is going to take on its output, and I think Lightning Birds especially, does that thoughtfully done in the way that you weave through sound in that work. You can tell it was something that you were thinking about right from the get-go and I mean, you brought up Videographic essays and I had the chance to talk a bit with Christian and and Jason a few months ago about their work. And could you talk a bit about, just to kind of bring us into the end of our conversation here, where are you taking some of these ideas of multimodal scholarship with your research next?
Jacob Smith: [00:22:02] I really like what you were saying about, you know, thinking about form from the beginning and I think one of the best outcomes for me about trying these different, multimodal forms is just that. Now I feel like I have this kind of range of options. I still write text essays. I'd still like to write textbooks, but now I feel like I have these kind of other options. So an idea that might have just kind of withered on the vine because, oh, I like that idea a lot, or I like that project, but I'm not sure it's a book or I'm not sure it's an article, but, oh actually maybe it's, you know, kind of an audio piece or maybe it's a video essay. I really like, the idea of having these different options, not only for different ideas, but for different audiences. I published my first videographic essay. There's such a exciting explosion of work and scholarship about the Videographic essay, so I feel like that's, even if you're working in sound, being familiar with that material, there's so much overlap and so much to learn from that, that that's, I'm glad that you're interviewing those folks and kind of bringing that into dialog.
[00:23:12] Copyright has been very much at the front of my mind because I have a new book project that I thought was gonna be an audiobook, but actually in the end, copyright issues made it impossible. So now it's kind of evolving in this different form, but that's okay. You know, one of the things I learned from some of these strange , you know, quirky kinds of projects is that sometimes the slowness of that process is a good thing. And sometimes, you know, you, you think it's gonna be this, but then that won't work out for one reason or another. So you have to switch gears and, and start moving to a different form. That tends to be, uh, a productive kind of a process, not only in terms of the idea, but in terms of the form that they take. I think that's exciting.
Stacey Copeland: [00:24:02] Yes. That's slowness, right? That working with sound in particular I think can bring into our research practice and, the new avenues that can come into focus when we take that time to explore other directions we didn't initially think maybe that our, our work would take, I think that's a great place for us to end things off. Thank you so much for joining us Jacob.
Jacob Smith: [00:24:25] Thanks. Yeah, great talking with you. Thanks so much for doing the work that you're doing.
[theme music fades up]
Stacey Copeland: [00:24:34] A big thank you to Jacob Smith for joining us here on Amplified this month. You'll find links to some of Jacob's work and other cool stuff mentioned in this episode in our show notes. If you have comments or additional thoughts on our conversation today or other Amplify topics, please do reach out. We're always interested in hearing from other folks, engaging with scholarly podcasting and other alternative modes of academic publishing. Thanks for listening to Amplified a podcast about the sounds of scholarship coming to you each. From our team here at the Amplify Podcast Network.
[music out]
-
Guest Bio
Jacob Smith is co-founder and director of the Master of Arts in Sound Arts and Industries, and professor in the Department of Radio/Television/Film. He is the author of Vocal Tracks: Performance and Sound Media (University of California Press 2008); Spoken Word: Postwar American Phonograph Cultures (University of California Press 2011); The Thrill Makers: Celebrity, Masculinity, and Stunt Performance (University of California Press 2012); Eco-Sonic Media (University of California Press, 2015); and two experimental audiobooks: ESC: Sonic Adventure in the Anthropocene (University of Michigan Press 2019) and Lightning Birds: An Aeroecology of the Airwaves (University of Michigan Press, 2021). He writes and teaches about the cultural history of media, with a focus on sound and the relationship between media and the environment.
-
Smith, J. (2021). Lightning Birds: An Aeroecology of the Airwaves. Michigan University Press. DOI: 10.3998/mpub.11714652
Smith, J. (2019). ESC: Sonic Adventure in the Anthropocene. Michigan University Press. DOI: 10.3998/mpub.10120795
Sterne, J. (2019). “Multimodal Scholarship in World Soundscape Project Composition: Toward a Different Media-Theoretical Legacy (Or: The WSP as OG DH), In Sound, Media, Ecology, eds. M. Droumeva & R. Jordan. Palgrave Macmillan, Cham, pp. 85-109. https://doi.org/10.1007/978-3-030-16569-7_5
Amplified - Videographic Criticism with Jason Mittell and Christian Keathley
Amplified - Who are the Peers? with Dr. A.D. Carson
But Is Any Of This Legal? Some Notes About Copyright and Fair Use by Jason Mittell - videographicessay.org* (*Note fair use differs country to country)
Fulcrum open source publishing platform - fulcrum.org
Intro + Outro Theme Music: Pxl Cray – Blue Dot Studios (2016)
Written and produced by: Stacey Copeland