Manipulations of time-scale relationships between action and perception

in piano performance:

An application of music technology

Peter Q. Pfordresher, University of Texas at San Antonio

Music technology provides a vehicle for measuring and manipulating the conditions of music perception and performance. Such applications have helped inspire a renaissance in music cognition research in the last three decades, especially in the complex domain of piano performance (see Gabrielsson, 1999; Palmer, 1997 for reviews). Through MIDI, and related computer applications, the immense complexity of the acoustic signal can be greatly reduced for the purposes of analyzing fundamental characteristics of keyboard performance. What once took Seashore and his colleagues days to implement and measure now takes a matter of minutes (cf. Seashore, 1967/1938).

The research summarized here applied MIDI technology to the exploration of auditory feedback=s role in piano performance, auditory feedback being the sound of one=s own performance, in keyboard performance. The goal of this research was to better understand the relationships between mental functions underlying perception and performance of music. In these studies, temporal relationships between production and auditory feedback were manipulated using variants of the Delayed Auditory Feedback (DAF) paradigm. In addition to the aforementioned benefits of MIDI to the measurement of performance, the present research also uses the interfacing characteristics of MIDI to manipulate auditory feedback in a more controlled manner than has been possible in past studies.

Background

Past research on the role of auditory feedback in performance has led to two deceptively simple conclusions: synchronization between produced and perceived onsets (e.g., keystrokes and sounded notes) affects the fluency of production, but matches between contents of auditory feedback (e.g., sounded pitches) and produced contents do not. On the one hand, the addition of an absolute time delay to auditory feedback onsets in DAF tasks greatly disrupts performance of music, speech, and tapping (e.g., Black, 1951; Chase, Harvery , Standfast, Rapin & Sutton, 1961; Fairbanks & Guttman, 1958; Finney, 1997; Finney & Warren, in press; Gates, Bradshaw, & Nettleton, 1974; Howell, Powell & Khan, 1983; Lee, 1950; 1951; MacKay, 1987; Pfordresher & Palmer, in press; Robinson, 1972). Disruption comes in the form of slowed production rate (e.g., Howell et al, 1983; Chase et al, 1961; Gates et al, 1974), increased errors (e.g., Black; 1951; Lee, 1950; 1951; Fairbanks & Guttman, 1958; MacKay, 1987; Finney & Warren, in press), and more variable timing (Pfordresher & Palmer, in press). On the other hand, randomization of feedback pitches in piano performance does not significantly disrupt performance (Finney, 1997), and alterations of feedback contents in DAF tasks doesn=t modulate the degree of disruption that one finds when contents aren=t changed (Howell et al, 1983; but see Finney, 1997).

Two shortcomings of this past research led me to further explore the role of auditory feedback in performance. First, the standard DAF paradigm incorporates an absolute time delay during performances in which produced timing varies. Thus, it is generally difficult to determine the exact timing relationships between produced and perceived events that cause more or less disruption; this leads to a gulf in our understanding of the effect. Despite the robustness of DAF disruption, for instance, debate exists as to what kinds of timing relations account for the disruption (see Pfordresher, 2001 for a review). Second, the presumed unimportance of feedback contents is worth revisiting for a simple reason: past studies have not altered feedback pitches in a way that relates to past performance. A serial lag delay in which keystrokes triggered the pitches a pianist has played in the past, for instance, might yield more disruption than past studies have shown.

In three different experiments, relationships between production and feedback were manipulated to differ in terms of onset synchrony, pitch contents, or both. A full report of these experiments has been published by Pfordresher (2003); brief summaries are given here for purposes of demonstration. In each experiment, pianists memorized and performed unfamiliar monophonic melodies (performed with the right hand only) on a Rhodes RD-600 digital piano at a prescribed tempo comprising 500ms inter-onset intervals (IOIs), and listened to auditory feedback over headphones. On an individual trial, the performer would first perform two continuous repetitions a stimulus melody in synchrony with a metronome (heard over headphones) with normal auditory feedback; then the metronome would stop and the performer would continue to repeat the melody continuously at the same tempo while one of the experimental feedback conditions took place. Performances were monitored by a computer and recorded in MIDI format, and were analyzed according to two measures of disruption: timing variability and error rates.

These experiments were able to achieve greater control over manipulations of auditory feedback through the use of new experimental software, FTAP (Finney, 2001). FTAP receives MIDI output from a keyboard or other device, performs manipulations on the MIDI input if desired, and then sends the MIDI data as input to a tone generator. For the present experiments, manipulations on MIDI input involved temporarily storing the feedback pitch of produced events, and then presenting these pitches after a given time delay (Experiment 1) or at a later key press (Experiment 2). FTAP also collected and stored MIDI data from performances into text files for data analyses

Experiment 1

DAF manipulations in Experiment 1 displaced feedback timing by a percentage of produced inter-onset intervals (IOIs); as a result, auditory feedback onsets occurred between successive produced onsets, which shifted the relative timing of perceived and produced events. However, performers always heard the most recently produced pitch in feedback contents. Each condition would result in the sounding of the pitch produced at event 1 sometime in the interval between produced events 1 and 2. The same manipulation of auditory feedback would then be applied to every produced pitch thereafter in a trial. These three types of delay trials were intermingled with other trials in which the performer experienced normal feedback after the metronome stopped, which functioned as a baseline condition.

Overall, trials with DAF revealed higher error rates and more variable produced timing than trials with normal feedback; this effect was more pronounced for timing variability than for error rates. Alterations to feedback timing therefore appear to disrupt mental processes underlying produced timing more so than accuracy. Furthermore, the amount of disruption increased as DAF percentages increased. The fact that disruption increased as feedback onsets occurred at higher percentages within produced IOIs suggests an accumulative effect of disruption as feedback onsets approach the next produced onset (cf. Finney & Warren, in press). Individual differences were not apparent, despite a broad range of musical training among the subject population (a range of 8 years among the 12 pianists participating).

Experiment 2

DAF manipulations in Experiment 2 invoked a serial lag delay: feedback pitches sounded in synchrony with produced key presses but matched pitches of events produced earlier. A new group of performers participated in Experiment 2.

Error rates were higher for DAF trials in Experiment 2 in comparison to trials with normal auditory feedback, with a similar but smaller effect on timing variability. This contradicts the suggestion from past research that similarity between the contents of auditory feedback and produced contents does not affect production (Howell et al, 1983). Unlike Experiment 1, the amount of disruption did not increase with delay amount, which may reflect a difference in how performers process sequential (Experiment 2) versus temporal (Experiment 1) relationships between actions and feedback. Finally, individual differences were evident in Experiment 2: a small subset of the participants (3 out of 21) showed no disruption from these manipulations according to either measure.

Experiment 3

Experiment 3 combined the manipulations used in the first two experiments: a new group of performers heard feedback pitches from events produced earlier that were also temporally displaced to occur between produced onsets. Both error rates and timing variability increased under these conditions, but increases were not as great as those found for measures that revealed maximal disruption in Experiments 1 and 2. Disruption from alterations of feedback onset synchrony and of feedback contents therefore does not appear to summate when these alterations are combined.

Comparisons across experiments confirmed these observations: maximal disruption of produced serial order occurred when the Awhat@ but not the Awhen@ of auditory feedback was altered (Experiment 2), whereas maximal disruption of produced timing occurred when the Awhen@ but not the Awhat@ of auditory feedback was altered (Experiment 1). Results suggest distinct but interacting contributions of timing and serial ordering relationships between auditory feedback and performance. As in Experiment 1, individual differences were not evident in Experiment 3.

Discussion

Relationships between the type of feedback manipulation and performance characteristics that yielded highest disruption reveal commonalties between perception and production, in that manipulation of one aspect of auditory feedback (e.g., synchrony) most disrupts the analogous aspect in performance (e.g., timing variaiblity). This supports other observations in the psychological literature that production and perception may share cognitive resources (e.g., MacKay, 1987; Prinz, 1997). It appears that perception and action are Acoupled@ such that alterations to one affect the other. Moreover, these results support a proposal in past research that small-scale timing mechanisms that guide synchronization in production are separated from large-scale timing mechanisms that guide the ordering of successive events (e.g., successive pitches in piano performance, MacKay, 1987; Palmer, 1997)

The fact that combining DAF types in Experiment 3 did not produce the maximal amount of disruption across experiments was not expected, but has interesting implications. It is possible that performers respond to the overall similarity between produced events and feedback events in a nonlinear fashion. When feedback events differ slightly from produced events, they may trigger unconscious error-correcting mechanisms in the performer, which result in the measures of disruption I obtained. However, when feedback events differ greatly from produced events, the resulting feedback may become perceptually dissociated from performance, allowing the performer to ignore the potentially disruptive feedback. Experiment 3's manipulations may have fit the latter characterization, allowing performers to ignore the altered feedback, because feedback differed in terms of both synchrony and contents. Manipulations of only synchrony or contents in the first two experiments, therefore, may have resulted in greater disruption because of less dissimilarity.

Individual differences were present in Experiment 2, but not in other experiments. It is possible that the use of feedback contents during performance may be more tied to memory and skill level than is the case for feedback timing. High levels of skill in piano performance may result in the ability to ignore feedback contents, relying instead on a memorized plan or Amotor program@ for performance (Lashley, 1951; Schmidt & Lee, 1999). Indeed, the subset of performers that were not affected by manipulations in Experiment 2 had more training and practiced the piano for more hours per week than those who were disrupted. Timing relationships, on the other hand, appear to be much more fundamental to piano performance, given that no performers were unaffected by asynchronous feedback. A valuable course for future research would be to see if this discrepancy is found for performances of instruments in which pitch can be varied continuously (e.g., the violin), as opposed to the discrete pitch representation used in the keyboard.

What are the implications of these data for music learning? I see two possible extensions, both are somewhat indirect. First, these data support the utility of practice environments that contain a minimum of reverberation. Highly reverberatory rooms end up simulating the effects of DAF, especially those in Experiment 1. When first performing in such a room, as often inevitably happens, it would probably be best to slow down the tempo. Reverberation time in concert halls is a constant, and slowing the tempo would make feedback onsets occur at a lower percentage of produced IOIs than would be found at faster tempi. Second, these data have implications for what kind of ensemble pieces may be more difficult to learn than others. Specifically, many fugues and inventions may simulate the disruptive effects of Experiment 2. The current data suggest that individual parts should be very well-learned by performers before pieces are performed as an ensemble.

References

Black, J. W. (1951). The effect of delayed side-tone upon vocal rate and intensity. Journal of           Speech and Hearing Disorders, 16, 56-60.

Chase, R. A., Harvey, S., Standfast, S., Rapin, I. & Sutton, S. (1961). Studies on sensory feedback I: Effect of delayed auditory feedback on speech and keytapping. Quarterly Journal of Experimental Psychology, 13, 141-152.

Fairbanks, G. & Guttman, N. (1958). Effects of delayed auditory feedback upon articulation.          Journal of Speech and Hearing Research, 1, 333-346.

Finney, S. A. (1997). Auditory feedback and musical keyboard performance. Music Perception,     15, 153-174.

Finney, S. A. (2001). FTAP: A linux-based program for tapping and music experiments.                Behavior Research Methods, Instruments & Computers, 33, 65-72.

Finney, S. A., & Warren, W. H. (in press). Delayed auditory feedback and rhythmic tapping: Evidence for a critical interval shift. Perception and Psychophysics.

Gabrielsson, A. (1999). The performance of music. In D. Deutsch (Ed.) The psychology of   music (pp. 501-602). San Diego, CA : Academic Press.

Gates, A., Bradshaw, J., & Nettleton, N. (1974). Effect of different delayed auditory feedback intervals on a music performance task. Perception & Psychophysics, 14, 21-25.

Howell, P., Powell, D. J., & Khan, I. (1983). Amplitude contour of the delayed signal and interference in delayed auditory feedback tasks. Journal of Experimental Psychology: Human Perception & Performance, 9, 772-784.

Lashley, K. (1951). The problem of serial order in behavior. In L. A. Jeffress (Ed.), Cerebral             mechanisms in behavior (pp. 112-136). New York : Wiley.

Lee, B. S. (1950). Effects of delayed speech feedback. Journal of the Acoustical Society of America, 22, 824-826.

Lee, B. S. (1951). Artificial stutter. Journal of Speech and Hearing Disrorders, 16, 53-55.

MacKay, D. G. (1987). The organization of perception and action. New York: Springer-Verlag.

Palmer, C. (1997). Music performance. Annual Review of Psychology, 48, 115-138.

Pfordresher, P.Q. (2001). Auditory feedback in music performance: Serial order and relative         timing. Unpublished doctoral dissertation, the Ohio State University, Columbus, OH.

Pfordresher, P.Q. (2003).  Auditory feedback in music performance: Evidence for a dissociation of sequencing and timing. Journal of Experimental Psychology: Human Perception and Performance, 29, 949-964.

Pfordresher, P.Q. & Palmer, C. (2002). Effects of delayed auditory feedback on timing of music performance. Psychological Research, 66, 71-79.

Prinz, W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9, 129-154.

Robinson, G. M. (1972). The delayed auditory feedback effect is a function of speech rate. Journal of Experimental Psychology, 95, 1-5.

Schmidt, R.A. & Lee, T.D. (1999). Motor control and learning. Champaign, IL : Human Kinetics.

Seashore, C.E. (1967/1938). Psychology of music. New York : Dover.