Mentioning the phrase "tenure review" to a junior faculty member in a university music school or department will likely cause heart palpitations, considerable dizziness, and more than just mild nausea. Additionally, the notion of tenure review evokes visions of thick binders filled with copies of journal articles, published book titles, letters of reference, student evaluations, and a plethora of other data that the candidate has assembled to justify his or her continued existence in the department, otherwise known as "tenure."

Although the traditional evidence presented by a tenure candidate has consisted of published books and journal articles, evidence of important creative activities, a record of high-quality teaching, and acceptable service to the institution and the profession, the increase in opportunities for digital scholarship and technology-related activities over the past few years has created a situation of concern for those faculty involved in technology initiatives. Simply stated, how does an institution measure and evaluate faculty work in instructional technology and digital publishing? How do promotion and tenure review committees evaluate those types of activities in comparison to the more traditional publishing and creative endeavors?

Cartwright (1994) suggests that:

How, then, can faculty members’ work in technology be evaluated fairly so that they will not shy away from the pursuit of digital scholarship and technology-related activity?

Before answering that question, consider the following scenario and subsequent activity. The situation is an annual evaluation (or perhaps a third-year review, or a tenure evaluation). The music faculty member has submitted a list of activities to be evaluated by an administrator and the promotion/tenure committee. Among the series of bulleted items is the following:

First, the candidate’s activity is to be rated from one to five (with one being highest). Second, the evaluating entity (administrator or committee) is only minimally knowledgeable about technology. What is this administrator or committee to do with that item? Two possible reactions are likely:

Thus, the activity is rated as a "1." There might, however, be a second reaction:

For the sake of argument, consider that the activity reported by the faculty member is worth a "2" or a "3" rating. In the first case it was overrated, while in the second case it was underrated. In both cases, the evaluation was inaccurate for the same reason: the evaluator had no clue of the real value of the activity.

This scenario highlights one of several problems inherent in the evaluation of technology-related research in the evaluation process. There are others.

In a second scenario, the young professor has developed an interactive web page to teach the fundamentals of music theory. Students using this page are able to take advantage of a sophisticated palette of graphic routines for the understanding and learning of theory fundamentals through techniques such as mouseovers and other JavaScript techniques. Moreover, their progress through the interactive program is controlled by a sophisticated technique for linking and branching that depends on the amount and speed of their success. Finally, audio examples illustrate the concepts being taught on each page.

For tenure review, though, the candidate is limited to submitting the traditional binder filled with copies of journal articles, letters of recommendation, student evaluations, etc. The candidate’s cutting-edge, interactive website is reduced to a one-dimensional presentation on paper, by which none of the members of the tenure committee can even understand the complexity and sophistication of the candidate’s work.

Are these examples unrealistic or unusual? Very likely not. These scenarios play themselves out yearly in colleges and universities across the nation. Only through the enlightened methods and techniques of administrators and committees can such creative and research initiatives be evaluated properly, and the following paragraphs will offer guidelines toward that end.

• Work with technology is often collaborative. It is not uncommon for teachers on different campuses to link their courses, for example. It is also not uncommon for people working with technology to work closely with others in different areas of campus.
• Work with technology is very time-consuming. People who work with technology in the classroom must spend a portion of their time learning and teaching new software to students and possibly colleagues. They often find themselves providing technical support to students and colleagues outside of class and office hours.
• Technology, particularly the world wide web, is blurring the distinctions between the traditional areas of evaluation for promotion and tenure, i.e., research, teaching, and service.
• Technology is affording new venues for learning about candidates’ work and of assessing the candidate’s role within the profession. For example, a person’s web page may offer outside reviewers a wider lens through which to view a candidate’s research.

In addition, the CCCC offers guidelines for both the tenure committee and the candidate as they each prepare to work through the tenure evaluation process. For the committee, the following guidelines are suggested. It is important that:

• Tenure and promotion committees work with the department hiring committees to ensure that expectations for work with technology and online scholarship (including how technology and online scholarship will be considered in the tenure process) be communicated to prospective new hires.
• Tenure and promotion committees work flexibly to find ways to acknowledge technology work done by candidates for promotion and tenure because the rapid pace of technological change makes it impossible for any set of guidelines to account completely for the ways the technology is making an impact on our profession.
• The candidate’s work be evaluated in the medium in which it was produced.
• Committees consider a candidate’s work with technology in as many categories as possible (teaching, research, service) in order to provide the fullest description of it and thus ensure the fairest evaluation of it.
• The candidate’s work be evaluated by persons knowledgeable about the use of computer technology. If qualified reviewers are not available on the candidate’s home campus, it is appropriate to solicit outside reviewers, particularly for work that is not normally offered as part of a candidate’s tenure file,
• Members of the promotion and tenure committee educate themselves about the candidate’s work.
• The candidate’s work be evaluated with respect to local conditions on campus.

For the candidates, it is important that they:

• Ask about whether and how credit for use of technology and online scholarship will be awarded in the tenure and promotion process when they first negotiate for their tenure track jobs.
• Find ways to tell others on their campus about the work they do as early as possible in the probationary period and on an ongoing basis,.
• Find ways to explain their work in terms of the traditional areas of teaching, research, and service, and also to explain carefully the ways in which their work overlaps or redefines those categories.
• Find others on campus who also work with technology and to network with those people.

Music departments nationwide need to address these issues and would be wise to follow these guidelines as they develop criteria for the evaluation of technology-related activity. There is no question that much of what occurs in the area of technology falls outside the scope of traditional research and creative activities.

For help in comparing technology-related activities to traditional scholarship, Cartwright (1994) suggests that instructional software can be considered scholarly if it:

• Requires a high level of discipline-related experience
• Breaks new ground or is innovative
• Can be replicated or elaborated
• Can be documented
• Can be peer-reviewed
• Has an impact on the discipline itself

Several college and university music schools/departments have utilized these suggestions to develop criteria for evaluating technology-related research in the tenure and promotion process. The University of Miami School of Music Faculty Handbook states: "As with printed textbooks, criteria of assessment, especially for the latter type (interactive software), include: the prestige of the publishing agency; the rigor of the initial review process; the scope of the material’s distribution; and the critical reception given to the publication. The Ohio State University School of Music’s promotion and tenure document states: "Publication of textbooks and pedagogical computer software is considered as research to the degree that pedagogical, programming, design, or subject-specific innovations are involved." One of the most complete descriptions of criteria for use in evaluating technology-related activity was developed at Central Missouri State University (although not addressed specifically to music) and can be found at: http://cstl.semo.edu/itfrr/.

The Department of Music at The University of Texas at San Antonio suggested the following listing of technology related activities that would be accepted as appropriate activity in a tenure or promotion application:

• Develop course website
• Develop course-related materials, e.g., interactive reinforcements, assignments, tests
• Use of course-related materials (lower value than their creation)

• Develop ancillary software for a publisher
• Publish in peer-refereed online journals
• Conference presentations
• Develop and disseminate application software

• Serve as webmaster for department
• Serve on technology committees
• Provide assistance/instruction to colleagues

As is evident from the above items, a great deal of variability is involved in an activity labeled simply "conference presentations," or "develop and disseminate application software." For example, how does development of software that is published by a small publishing company compare with software that is unpublished, yet used by five other institutions? UTSA offers this example of an evaluation rubric for software development, which is based upon a model they had previously devised for CD recordings:

• Software/coursework produced and distributed by a major company (5 points)
• Software/coursework produced and distributed by a small company (4 points)
• Software/coursework produced and distributed by a non-commercial entity (e.g., professional organization, charitable foundation, etc.) (3 points)
• Software/coursework self-produced and self-distributed as a corporate entity (must show sales and/or adoption data) (2 points)
• Software/coursework self-produced (not distributed) (1 point)

Are there potential flaws in the above model? Is there room for abuse, by either the faculty member or the evaluator? One would have to answer "yes" to both questions. It seems almost impossible to "legislate" an exhaustive evaluative tool, however, that would take into account every possible nuance of every conceivable project. As with all such evaluations, the best hope lies with having an evaluator who is honest, sincere, and fair.

In conclusion, it is important for music units in colleges and universities to determine how they will evaluate technology-related activities as they review faculty for tenure and promotion. The suggestions and guidelines presented above can help music administrators and promotion/tenure committees to evaluate such work with the same level of confidence that they bring to the task of evaluating traditional research and creative activity.