Learning with Music Signals: Technology Meets Education

Beat tracking is one of the most fundamental tasks in Music Information Retrieval, reflecting our ability to perceive, follow, and understand music. Existing studies have explored beat tracking in both the audio and symbolic domains. However, there is limited literature addressing the relationship or differences between how beat tracking systems perform in these two domains. We conducted initial experiments using a symbolic beat tracker employing MIDI files as input data, and an audio beat tracker with audio recordings as input data. From the preliminary experiments, we observed the potential benefits of fusing beat tracking in the two domains. However, several factors complicate the experiments, making it difficult to pinpoint where performance improvements originate. Specifically, our current experiments involve various factors, including different data representations (event-based representation vs. frame-based representation), different model architectures (convolutional recurrent neural network vs. bidirectional long-short term memory network), and potential transcription errors when obtaining MIDI files from audio recordings. Additionally, further analysis is needed to quantify the differences between beat trackers in the two domains.

At the Dagstuhl Seminar, we presented our preliminary results, discussed our observations, and explored ways to refine the scope of the study. Participants offered suggestions on how to combine beat tracking in the two domains and recommended new directions for further investigation.

We have seen amazing new techniques for music composition and synthesis, but there is strong evidence that important music concepts and abstractions are not being learned by today’s systems. Simply put, machines are learning to imitate style without a deep understanding of what lies beneath the style and what could be altered or applied in other contexts. In contrast, humans have radically transformed music multiple times throughout history and across cultures. Even if we set aside the avant-garde and consider only music that is almost universally admired, composers consistently invent new kinds of music. This is one of our greatest challenges: to discover the principles that “define” music. I believe these principles are not just a set of commonalities such as “music has pitch and rhythm,” but are much deeper, having more do with temporal and tonal structures interacting with human perception and cognition.

Over the last two decades, the community of modular synthesizer enthusiasts has grown considerably worldwide. For me personally, the COVID-19 pandemic was the trigger to revisit this sub-culture and become fascinated with it. Dabbling with modular synths appeals to those with an interest in electronic music, audio signal processing, electrical engineering, industrial design, and elitist nerdism, spiced with a collector’s frenzy. In the demo session of this Dagstuhl Seminar, I set up three racks with modular synthesis modules to demonstrate to the participants that these devices can be useful tools for teaching undergrad students basic concepts in Signal Processing and Machine Learning. More specifically, I presented three live experiments that could be achieved with quick re-wiring of the systems:

• $\mathrm{■}$

  Auralization of the sampling theorem by running a sinusoidal oscillator through a sample-and-hold module.

• $\mathrm{■}$

  Manual rotation of a phase-shifted signal pair in the two-dimensional plane using a weight matrix module and an oscilloscope.

• $\mathrm{■}$

  Approximation of a cosine signal from a triangle signal using a simple neural network made up of weight matrix and hyperbolic tangent modules.

From cyberspaces to physical spaces, artificial intelligence (AI) is transforming every aspect of human society. Interacting and collaborating with robots that possess human-level intelligence is no longer just a dream but something very likely to occur in the next decade. In addition to helping humans complete mundane or dangerous tasks such as cleaning and searching, becoming human companions is also highly desirable. This requires robots to understand human civilizations and social norms. Music is considered one of the highest forms of human ingenuity and plays an important role in society. Teaching robots to read, play, and compose music will not only advance robotics and AI research but also benefit human society as a whole.

In my stimulus talk, I shared some of my thoughts on why, what, and how to teach music skills to robots. While MIR has made significant progress in cyberspaces toward machine musicianship, much work remains to be done in physical spaces, particularly with instrument-playing robots. I reviewed some existing work on musician robots and their limitations, and then proposed several research directions.

Dagstuhl Seminars provide a very welcome opportunity to consider work in progress, conversation-stimulating topics, and discipline-wide issues. They are particularly useful for topics that require broad “buy-in” from a range of practitioners in the field. As such, they present the perfect opportunity to advance a discussion about a coordination effort I have been informally engaging in with interested parties for a long time. Some high-level observations will help provide the context and motivation for this topic:

• $\mathrm{■}$

  Music computing is not a large field. There are wonderful, enthusiastic practitioners, but relatively few of us compared to more populous fields.

• $\mathrm{■}$

  Relatedly, music computing is somewhat disparate, with practitioners spread out across the world, and sub-fields like MIR, music theory, and music psychology having rather limited interaction, despite their closely shared goals, tasks, and data.

• $\mathrm{■}$

  Educational resources can serve not only as material for preparing specific classes but also as a vehicle for consolidating the field (e.g., the function of textbooks as teaching tools and reference books for experts).

• $\mathrm{■}$

  Code libraries can likewise serve as another gathering point, supporting both newcomers with “how-to” guides and consolidating the field for expert practitioners.

While audio (and signal processing) are relatively well served with code libraries (e.g., McFee et al.’s librosa⁵⁵5https://librosa.org/) and pedagogical materials (e.g., Müller’s Fundamentals of Music Processing⁶⁶6https://audiolabs-erlangen.de/FMP), relatively little exists by way of computational resources for other parts of music computing, including the so-called “symbolic” music.

There are several promising initiatives that bring together algorithms in specific areas of symbolic music computing. These include OMNISIA (for pattern finding in Java), synpy (for rhythmic syncopation in Python), and MIDI Toolbox⁷⁷7https://www.jyu.fi/hytk/fi/laitokset/mutku/en/research/materials/miditoolbox (for melodic contour and more in MATLAB). While these efforts are welcome, they all originate from different research groups, are implemented in different programming languages, and in some cases, are no longer maintained.

Then there are larger libraries that could potentially bring these efforts together. At one time, humdrum⁸⁸8https://www.humdrum.org/ was a central point of reference. It continues to be used and maintained – an impressive 40 years later! However, there are downsides; for example, the “language-neutral” setup is commendable but not very inviting for newcomers who have computational expectations based on the landscape of 2024. Later, music21⁹⁹9https://github.com/cuthbertLab/music21 emerged. First published in 2010/11, it too continues to be maintained and used. That said, the creator/maintainer recently made the explicit decision¹⁰¹⁰10https://groups.google.com/g/music21list/c/HF3tgkMvNWI/m/7vaIHr88BAAJ that it is not/no longer intended to provide the holistic directory function stated here. Instead, it specifically invites sub-projects to operate independently, with or without music21 as a dependency. Partitura is arguably one such project, though it explicitly positions itself in contrast to this approach, suggesting that users “working in computational musicology” should adopt music21 instead.

As a result, students and researchers wishing to “get into” a topic often have to do a lot of “spade-work” to compare any new algorithm with existing work or even to make use of those existing algorithms. In short, given this state of affairs, and following conversations with the maintainers of these code libraries, it is clear that we need a new coordination effort to bring together the work cited above at a higher level. To begin addressing this, we propose a new code library that primarily serves to unify these algorithms. This library should be as user-friendly as possible, serving as a welcoming introduction for newcomers to the field and a valuable reference for those already active. Moreover, it should serve to:

• $\mathrm{■}$

  carefully credit previous work,

• $\mathrm{■}$

  “fill in the gaps” with new implementations where none are readily available, and

• $\mathrm{■}$

  expand into uncharted territory alongside ongoing, separately published research.

Dagstuhl provided a valuable opportunity to advance these ideas and gauge colleagues’ opinions and priorities. We leave Dagstuhl with:

• $\mathrm{■}$

  a clear and shared vision,

• $\mathrm{■}$

  the motivation to populate and maintain this resource,

• $\mathrm{■}$

  a substantially expanded core development team.

Finally, this positive effort complements more specifically and exclusively pedagogical initiatives, notably including the TISMIR education track¹¹¹¹11https://transactions.ismir.net/articles/10.5334/tismir.199 that features prominently elsewhere in this seminar.

At the Dagstuhl Seminar, we shared our recent experiences with four editions of the annual programming contest for “Lyric Apps” since 2020. Lyric apps, as we call them, are a new form of lyric-driven visual art that can render different lyrical content based on user interaction.

We first developed the “Lyric App Framework,” a web-based platform for creating interactive graphical applications that play music and display synchronized lyrics. After releasing this framework as the “TextAlive App API”¹²¹²12https://developer.textalive.jp, we began holding annual programming contests in 2020¹³¹³13https://magicalmirai.com/2023/procon/index_en.html. The contests were open to the public and held online, with the winners announced at the “Magical Mirai” exhibition featuring “Hatsune Miku.” Hatsune Miku is the most popular singing voice synthesis software/character, and we conducted the contests in collaboration with Crypton Future Media, Inc., the company behind Hatsune Miku. Thanks to Hatsune Miku’s popularity, our contests gained significant attention, receiving 159 lyric app submissions over four years. For each contest, we provided a dedicated set of musical pieces and lyrics with appropriate permissions for online streaming, allowing contestants to focus on development and add novel interactive capabilities to existing musical pieces.

We observed that our contests attracted a broader community than the typical creative coding community. One contestant mentioned that they usually did not (or could not) participate in such competitions, but their love for music motivated them to develop and submit a lyric app. While interest in creative coding culture is growing, not all programmers feel confident that they are “creative” enough to enjoy creative coding. Lyric apps allow these programmers to rely on existing musical pieces for the “creative” part, thus kickstarting their programming activities. Programming itself is a creative activity, and we were pleased to see that lyric apps provided opportunities for more people to unleash their creativity. Notably, several contestants reported in the post-contest questionnaire that they were indeed novices, in some cases without any prior experience in web application development. We believe that lyric app development could be a valuable component of computer science education curricula.

Expressive performance is an integral part of music-making. Expert performers shape various parameters such as tempo, timing, dynamics, and articulation in ways that go far beyond the notated score, resulting in an expressive interpretation that emphasizes the dramatic, affective, and emotional qualities of the music, thereby engaging listeners.

Previous work has studied expressive performance parameters in both symbolic and audio modalities [1, 2]. While audio data captures the richness of timbre and dynamics, supporting a holistic understanding of performance, symbolic modalities offer a more precise, structured representation suitable for systematically studying nuanced performance parameters such as timing and articulation. Additionally, in bridging the two modalities, transcribed symbolic (MIDI) datasets have been released in recent years, among other uses, for performance analysis.

Given the availability of different data sources, questions and concerns regarding the variability and reliability of different modalities naturally arise. Specifically, it is important to identify which expressive dimensions and features can be reliably and consistently studied using which data modality. Understanding the advantages and limitations of each data source for different performance parameters will enable a more informed use of existing datasets, both in terms of their technical and musical potential application domains.

As a first concrete step, we conducted a preliminary study [3] to explore the musical quality of different transcribed piano performances. To this end, we proposed musically informed piano transcription evaluation metrics that measure performance aspects such as timing, articulation, harmony, and dynamics in a transcription.

After the successful reintroduction of deep neural network models, using larger models fueled by large-scale datasets has become the standard practice for achieving better results. Recently, very large foundational models pre-trained on massive datasets have gained popularity as building blocks for MIR applications, providing the power of big data and big models. However, this approach leads to immensely high model complexity, making it difficult for users to understand how the resulting model works. From an engineering perspective, the large black-box approach can be tempting because it reduces the effort of building such a big model from scratch, or because one might not have access to the necessary datasets in the first place. This high complexity of models results in fewer opportunities for practitioners to understand the system’s mechanisms, affecting the maintainability and reliability of larger systems that potentially host multiple such pre-trained models.

Alternatives for tackling this issue include model post-hoc explanation, de-biasing, distillation, and so on. Another alternative could be measuring the complexity of the model to understand how unreadable the system is and possibly provide an alternative perspective on assessing the “best” model by incentivizing interpretable models. However, measuring model complexity remains an open question that needs further study, both in the ML and MIR fields. There are several principled ways to measure complexity in simpler probabilistic models, such as logistic regression, but these methods can quickly become complicated as the model’s complexity increases.

During the Dagstuhl Seminar, we explored key works in this area and discussed how they could be applied to MIR-specific models, integrating diverse perspectives from musicology, machine learning, and music theory. Our ultimate aim was to identify classes of more interpretable yet expressive models, which could hopefully enable us to use statistical models as tools for learning from data.

The model of the American “small liberal arts college” (SLAC) may be relatively unknown to those outside of the US, but this model of holistic education across many disciplines, with a low student-to-faculty ratio, offers a unique perspective on teaching and learning. In this talk, four professors from different SLACs around the US discussed the challenges and opportunities their roles provide and how they teach music information retrieval both inside and outside the classroom. The goal of this talk was to stimulate discussion and reflection on different models for equipping and training the next generation of researchers.

Noticing the current AI and generative AI hypes, the surrounding incentives, and the types of papers that make it through the noise and get lauded for their contributions, I am worried. Technical engineering innovations and the (sometimes) marginal improvement of benchmarks are often praised, while the question of whether these innovations serve a true purpose seems to garner much less attention.

Over the past few months, I have served as an ISMIR meta-reviewer, a PhD and student supervisor, and a thesis reading committee member. Simultaneously, I have engaged extensively in public discussions on the societal impacts of AI applications. In my supervision, publication efforts, and public engagement, I have explicitly focused on asking fundamental questions: Is the problem well-articulated? What type of insight would best serve the articulated problem? Often, the answers to these questions have little to do with large-scale AI or generative AI innovations.

At the same time, in my roles as a meta-reviewer and reading committee member, I frequently experienced run-ins with peers when I questioned the extent to which a piece of work was sufficiently grounded in long-term related research, and whether the presented innovation and its evaluation results actually aligned with the underlying problem statement and provided deeper insights. Typically, in such situations, I have been told that the innovation is “cool,” the work resulted in publications, and that other successful published works share similar characteristics. Addressing the actual underlying questions, I was told, would be out of scope or too difficult. Indeed, my students and I struggle significantly more to get our work academically acknowledged, while those who align with the hype more easily gain visibility and funding.

Is this what research is supposed to be about? Am I, in encouraging my students to be critical, inadvertently harming their careers? Or is this precisely what academics, particularly those affiliated with public institutions, should advocate for more strongly?

Open source software provides a common foundation for both research and education. Instead of building tools entirely from scratch, practitioners can leverage existing implementations as building blocks to rapidly compose sophisticated systems and solve complex problems. At the same time, open source software can serve as an educational reference point that students can directly inspect and, with perhaps some effort, understand or modify.

In recent years, so-called “foundation models” have become a common and powerful approach to rapidly developing data-driven systems, especially in situations where task-specific training data may be relatively scarce – which is often the case in MIR. A common work-flow is to use a previously trained model as a “black-box” feature extractor, effectively replacing the role of hand-crafted audio features. While this approach is undoubtedly effective for system development and “pushing the needle,” its pedagogical value remains unclear.

In my Dagstuhl presentation, I highlighted the parallels and differences between traditional open source software and foundation models in the context of MIR. My goal was to stimulate discussion around our use of large pre-trained models, data dependencies, and potential research directions to improve our understanding of how to best use and teach these methods.

In previous work [1], we developed a prototype for an interactive music game called “Sing Your Way.” The prototype resembles a jump-and-run game and uses a gaming controller as the input device. Additionally, it incorporates the player’s singing voice to interact with the game world. For this, we estimate the pitch of a microphone signal in real time and use it to control parts of the game world. For example, by singing a note, a pitch line appears in the game that a player can jump on to add stair-like elements, allowing them to overcome obstacles and reach the end of a level. The game is deliberately kept simple, with a limited set of rules: players can move left, move right, jump, and sing notes, where pitch determines the vertical position in the game world.

Within this simple gaming environment, we consider two specific use cases: singing education and data collection for MIR research. First, the prototype serves as a tool for singing education, offering a fun and motivating environment for practicing singing techniques. Players are challenged with different gaming levels that include various voice-related tasks, such as singing the correct notes, intervals, chords, scales, or melodies. Combining gamification with educational objectives helps players train their ear and improve their vocal control in an engaging and playful manner. Second, the game can also serve as a valuable tool for collecting data on vocal performance and singing style. By capturing detailed information about players’ pitch accuracy, timing, and vocal range, educators and researchers can analyze trends and patterns in singing proficiency. This data can contribute to research in music education and assist in developing new, personalized methods for interactive vocal training and assessment.

At the Dagstuhl Seminar, we presented a demonstrator of our game and invited feedback and discussions on several topics: user experience, educational impact, technical improvements, potential collaborations, and data privacy. By actively playing our music game prototype, the Dagstuhl Seminar participants offered diverse perspectives and ideas on game level design for music education and MIR research.

The Transactions of the International Society for Music Information Retrieval (TISMIR) offers a new education track for articles with a tutorial-style approach, focusing on MIR research and practical applications. This track covers a wide range of topics, including specific techniques, fundamental principles, and practical aspects, reflecting the diverse interests of the MIR community. We published an editorial to introduce this new track, highlighting the key characteristics of educational articles and exploring why the music domain may provide an intuitive and motivating setting for education across various levels and disciplines [1].

At the Dagstuhl Seminar, we discussed the participants’ views on education, their teaching experiences in MIR-related fields, and the needs of their students. Additionally, we reviewed their role in education and the recognition they receive for educational activities in their respective institutions. We believe that writing educational articles benefits both authors and readers by sharing expertise, enriching the knowledge base, and fostering innovation. It also provides researchers and PhD students with opportunities to refine resources, reflect on principles, and receive expert feedback. We discussed incentives for writing educational articles, as well as barriers that may prevent one from doing so, and explore the potential of educational articles in academia. Finally, we explored how to craft effective educational articles for the TISMIR educational track, laying the groundwork for a broader discourse on education within MIR and beyond.

In this talk, we described the technical components and challenges of the human-AI music ensemble and highlighted recent advances in the MIR community. We also presented cases of the human-AI music ensemble in real stage performances. Then, we shared ideas and discussed issues when applying these outcomes to music education settings in public schools. Finally, we envisioned future directions for multimodal integration with language models to achieve further improvements.

In my Dagstuhl Seminar talk, I explored the possibilities of creative learning design through the work of the NYU Music Experience Design Lab (MusEDLab). The lab’s research focuses on developing collaborative web applications and interactive systems that lower barriers to creative expression and sustain musical engagement by fostering curiosity and inquiry. The showcased applications embody these principles and demonstrate the lab’s innovative approaches.

Participants were invited to consider how Music Information Retrieval (MIR) can enhance these methodologies and support creative learning across diverse cultures and communities globally. The philosophical foundations discussed included the balance between design processes of iteration and ideation, the interplay between freedom and friction, and the contrasts between Eastern and Western philosophies. Through these discussions, participants were encouraged to explore how MIR and interactive interface design can create open, creative spaces for music education, promoting sustained engagement and learning.

In connection with this presentation, I want to further summarize some related work. The book [1] outlines a musical journey through Scratch, an accessible programming environment with rich media features, including music and sound. It presents a series of independent musical projects, guiding readers through the process of creating each one. Along the way, readers explore coding techniques and algorithmic music processes. The interactive projects encourage music-making through play and composition. Designed for beginner coders and musicians, the book introduces programming and musical concepts as needed, with projects ranging from basic to advanced. Each project is self-contained, allowing readers to complete them in any order.

While much research has explored Scratch in the context of children creating games and interactive environments, little has focused on its music-specific functionality. Music and sound are central to children’s lives, yet creating music in coding environments often requires deep knowledge of music theory and computing. In [2], we examine the affordances and constraints of Scratch 2.0 for music-making. We analyze the limitations of its music and sound blocks, discuss bottom-up music programming, and break down the creation of a simple drum loop. We argue that current block designs may hinder meaningful music-making for children, Scratch’s core users. Additionally, we touch on the history of educational music coding languages, reference existing Scratch projects, compare Scratch with other music programming tools, and propose new block designs to foster more accessible music creation.

Finally, in [3], we discuss the development of “Sound Thinking,” an interdisciplinary general education course offered by the Departments of Computer Science and Music. The paper focuses on the student outcomes we aim to achieve and the projects designed to help students reach those outcomes. It explains our transition from a web-based environment using HTML and JavaScript to Scratch and discusses how Scratch’s “musical live coding” capability can more effectively reinforce these concepts.

Differentiable Digital Signal Processing (DDSP) is an umbrella term for the concept of including fixed signal processing components in deep learning models, requiring all building blocks to be differentiable with respect to their input. This way, gradients can be back-propagated through the entire system, for example to train a neural network that controls the parameters of a fixed synthesizer. From the more general perspective of hybrid or model-based deep learning [1], this paradigm has two main advantages. First, the parameters of DDSP components are inherently interpretable. As opposed to end-to-end approaches, which tend to learn opaque representations of the training data, these parameters can be directly analyzed and modified with a lower susceptibility for confounding factors and overfitting. Second, incorporating domain knowledge (for example, about the physical properties of the sound generation mechanism) directly into the system architecture can be used as an inductive bias towards a computationally efficient model. Conversely, end-to-end approaches are often intentionally over-parametrized, resulting in unnecessary computational complexity.

Intuitively, it would be desirable to train a DDSP model with an unsupervised analysis-by-synthesis approach, i.e., by learning to estimate parameters for generating an output signal that is as close as possible to a given target signal. However, recent approaches using DDSP for various audio processing tasks [2, 3, 4, 5, 6, 7, 8, 9] typically require additional input information about the target or specifically designed loss terms to converge to meaningful solutions. While various work-arounds for training specific DDSP models have been proposed, we conjecture that fundamental issues may prevent fully unsupervised learning to be successful in estimating meaningful control parameters. At the Dagstuhl Seminar, we discussed the strengths, weaknesses and possible use cases of DDSP and hybrid or model-based deep learning in general, based on three different perspectives on this problem:

1. DDSP as part of the loss function. In many scenarios, a neural network is trained to output the parameters for a fixed DDSP model. For this, the gradient of a loss function between DDSP output and a target signal is first back-propagated through the fixed components, so that the loss for the neural network can equivalently be expressed as the composition of loss function and DDSP model. This view has recently led to new insights into training DDSP-based systems, and it has been shown that the “loss landscape” for individual parameters is often highly irregular, especially for parameters that control the frequency of synthesis components [10, 11, 12]. Further research is required into which combinations of loss functions and DDSP components result in favorable or problematic loss landscapes.

2. DDSP as part of the neural network. Some of the success of deep neural networks for complex tasks has been attributed to the high dimensionality of the loss landscape [13, 14]. While it is often inadequate to translate geometric intuitions to high-dimensional spaces, restricting individual parameters inside the network to have a specific physical meaning may undermine these benefits, since for DDSP components only very specific parameter combinations lead to an appropriate output. On the other hand, some restricting neural network components, like convolutional layers (inducing a bias towards local relationships in the data), are highly successful in a vast amount of application scenarios. This raises the question if there is a natural trade-off between the degree of restriction and learning success and which kinds of model biases are good choices for music and audio data.

3. DDSP as an inverse problem. An inverse problem is the task of determining from a set of observations their cause [15]. In technical terms, the goal is to estimate model parameters from measurements of the model’s output, which is closely related to the structure of many DDSP systems. It is well established that so-called ill-posed inverse problems are generally not easily solvable and require knowledge about the structure of each individual problem to obtain meaningful solutions in the parameter space [15]. While some previous work has drawn initial parallels between DDSP and inverse problems [16, 17], it remains unclear to which degree various DDSP problems are ill-posed and whether methods from the field of inverse problems can be applied to improve training and results. In any case, this perspective warrants further research into specialized solutions for individual DDSP systems.

Generative modeling of music is a challenging task due to factors such as its hierarchical structure and long-term dependencies. Many existing methods, particularly powerful deep learning models, operate exclusively on one of two levels: the symbolic level, e.g., notes, or the waveform level, i.e., outputting raw audio without reference to any musical symbols. We argue that both approaches have fundamental issues that limit their potential for applications in computational creativity, particularly in open-ended creative processes. Specifically, symbolic models lack grounding in the reality of sound, while waveform models lack the level of abstraction necessary to make music creation more accessible.

At the Dagstuhl Seminar, we proposed a hybrid approach that integrates end-to-end modeling at both levels, combining their strengths while overcoming their respective limitations. While similar models already exist, they typically involve separate components that are only combined after training is complete. In contrast, we advocate for fully integrating both levels from the outset. We discussed the advantages and opportunities of this approach, as well as the challenges it presents and potential solutions. Our goal was to inspire other researchers to see the value in fully hybrid modeling of musical data going forward.

MIR of the future will look different than it does today, thanks to the coming deluge of AI-generated music content. The query-document paradigm of MIR is shifting to a prompt-generation paradigm, where music information retrieval has limited applicability – and indeed efficacy – in corpora projected to grow by “billions of [AI-generated] songs per year” [2]. With the proliferation of online commercial services providing music generation via prompting (e.g., suno.com and udio.com), or even generation-to-distribution pipelines (e.g., boomy.com), the sum total of human-crafted musical work is threatened to be swamped by AI-generated music – a “musicpocalypse,” echoing the “textpocalypse” predicted by Matthew Kirschenbaum [1] in his Atlantic article about the impact of large language models (LLMs) like ChatGPT. As he writes, “Our relationship to the written word is fundamentally changing … [These LLMs create] synthetic text devoid of human agency or intent … [and will lead to] a textpocalypse, where machine-written language becomes the norm and human-written prose the exception.”

In this stimulus talk, I presented two papers that deal with the “musicpocalypse” and its implications for MIR (and beyond). The first paper (rejected from ISMIR 2023) involved a collaboration among twelve authors from a variety of disciplines: engineering, musicology and ethnomusicology, sound studies, computational creativity, computer music, and composition. The authors include Bob L. T. Sturm, Ken Déguernel, Rujing S. Huang, André Holzapfel, Ollie Bown, Nick Collins, Jonathan Sterne, Laura Cros Vila, Luca Casini, David Alberto Cabrera Dalmazzo, Eric A. Drott, and Oded Ben-Tal. This paper playfully argues that a new kind of music studies is needed to address the impending flood of AI-generated music. It outlines six aspects for critical analysis and discusses each with reference to the contemporary AI music service Boomy.com: the service or company, the founders and employees, the use of the service, the users, the algorithms, and finally, the resulting music. After rejection, we posted this paper to SocArXiv¹⁴¹⁴14https://osf.io/9pz4x. An updated version has since been accepted to the 2024 AI Music Creativity conference¹⁵¹⁵15https://aimc2024.pubpub.org. We were also motivated to organize our own humanities-focused conference on the subject: “The First International Conference in AI Music Studies,” to be held Dec. 10-12, 2024, in Stockholm¹⁶¹⁶16https://boblsturm.github.io/aimusicstudies2024.

While the paper above was criticized for being “more like the study of a business case,” having a weak technical analysis, and being irrelevant to ISMIR, we decided to write a paper clearly aligned with the technical focus of ISMIR. This paper (rejected from ISMIR 2024) is titled “AI Music: A New Frontier for Music Information Retrieval” and was written by Laura Cros Vila, Bob L. T. Sturm, Luca Casini, Anna-Kaisa Kaila, and David Dalmazzo. In this work (currently being revised for submission to TISMIR), we explore the implications of widespread AI-generated music for MIR. We present two concrete case studies centered on music collections generated by contemporary AI music services. The first case study (“music genre recognition”) analyzes a collection of music recordings generated by one AI music service to attempt to automatically uncover the music styles of the proprietary system. The second case study (“composer identification”) analyzes two collections of music recordings from different AI music services in relation to human-made music to try to identify the origin. Through these case studies, we identify many outstanding questions and promising avenues for future MIR research as it responds to the rising tide of AI-generated music.

The discussion of these two works during the week at Dagstuhl revealed that many participants are convinced of the novel and significant challenges posed to MIR and music in general by AI-generated music. The quality of the generated audio is so high that it is nearly competitive with what one hears on commercial radio. As an unplanned demonstration of that quality, we conducted an experiment when Christof Weiß asked me (on Wednesday afternoon) to generate musical audio from the prompt, “A medium-tempo funky song about music informatics researchers meeting in Germany with Irish-style melodic lines played by bass, drums, acoustic guitar, piano, vocoder, baritone sax, ….” Within 30 minutes, I had a finished 4m09s recording of a catchy piece of music – titled by the system, “Funk and Data in Deutschland” – which a group of us then transcribed and learned on Wednesday evening. We performed the song live at the Thursday evening concert to great applause. For more details on this experiment, see Section 5 of this Dagstuhl report.

In this talk, we described an experiment in designing high-impact research excursions for undergraduate students in Music Information Retrieval (MIR) as a culminating part of their undergraduate research projects. Education research has highlighted the importance of research experiences for undergraduates (REUs) in encouraging students, especially those from underrepresented minorities, to pursue graduate degrees. The Harvey Mudd College MIR Lab partnered with the Semantic Audio Processing Group at the International Audio Laboratories Erlangen (AudioLabs) to host joint workshops aimed at providing a compelling and impactful experience for undergraduates. At the end of their summer REU internship, the students travel to a host university to present their work, engage in technical discussions, learn about current research, and receive professional mentoring and feedback.

So far, we have conducted two joint workshops in 2023 and 2024 involving seven undergraduate students. These events were organized as two-day workshops with various components. First, students presented their work in roughly 15-minute presentations interleaved with and followed by 45-minute intensive discussions. These presentations stimulated deeper discussions not only on the research content but also on the conceptual elements of good scientific research practices and presentation styles. Second, the organizers gave scientific presentations related to the REU topics, bridging the gap to state-of-the-art techniques. Third, the undergraduate students had the opportunity to meet PhD students to discuss recent research directions and gain insights into their research careers and experiences as PhD students. Individual conversations with PhD students proved to be particularly impactful for the participants. Finally, joint meals, hiking, and other social interactions allowed students to meet informally with the organizers and other researchers. The workshop concluded with a day of sightseeing.

At Dagstuhl, we shared our observations and insights from these workshops and discussed with other participants the benefits to students, faculty, and the wider MIR community. In particular, brainstormed how the MIR community can educate, welcome, and encourage the next generation of researchers.

The age of deep learning provides exciting possibilities and progress in MIR but also presents a number of challenges, including:

• $\mathrm{■}$

  The lack of robust, generalizable, and understandable models.

• $\mathrm{■}$

  The lack of large amounts of high-quality, annotated training data.

In several personal discussions at Dagstuhl, we addressed these problems and discussed some preliminary results [1] from a long-term research project at the University ofWürzburg¹⁷¹⁷17https://gepris.dfg.de/gepris/projekt/531250483, where we approach such challenges by exploiting cross-version datasets of classical music. These datasets contain, for the same musical work:

• $\mathrm{■}$

  Several modalities (score and audio).

• $\mathrm{■}$

  Several performances (interpretations, or arrangements).

• $\mathrm{■}$

  Several annotations (multiple experts).

In a broader context, these datasets also comprise:

• $\mathrm{■}$

  Several works by the same composer.

• $\mathrm{■}$

  Several composers from the same historical period.

We use these relationships to test generalization along different dimensions, such as generalizing to other versions of a work or to other works by a composer. Additionally, we can leverage cross-version relationships for learning musical concepts that remain constant or vary across versions in an unsupervised manner.

As a concrete example, we conducted experiments on multi-pitch estimation, addressing unsupervised domain transfer from one instrument (piano) to other, unseen instrumentations (singing/orchestra). We approached this using a teacher–student learning strategy, where we compare teacher annotations across versions (performances) in the target domain and use only consistent annotations as labels to train the student model. Enforcing such consistency across versions during student training helps to improve the challenging domain transfer. In the discussions at Dagstuhl, we received valuable feedback on these ideas and reflected on robustness across different music domains.

To explore this tension, I presented a provocation: What would an audio-based journal for research about audio sound like? Building on Bernd Herzogenrath’s concept of sonic thinking, we proposed the idea of an audio preview, in the form of a one-minute audio clip, for communicating the sonic characteristics of research that can accompany traditional text-based journal articles. By encouraging researchers to communicate their work in an audio-only format, we hope to embrace, celebrate, and share the unique knowledge that can only be communicated through sound.