Introduction

Researchers in affective computing, human-computer interaction, and the social sciences have long been interested in perceived and experienced emotion. Measurement of these affective constructs is frequently accomplished by collecting holistic ratings at the end of a task, often using discrete emotion categories. However, both subjective experience and observable behavior unfold over time and across numerous affective dimensions. In order to capture all the shifting nuance of such unfolding, affective measurement must be continuous.

Continuous measurement of affect was first popularized by Gottman and Levenson [1] in their studies on marital interactions using the “affect rating dial.” This tool consists of a circular plastic knob mounted to the arm of a chair. Raters are instructed to turn the knob clockwise or counter-clockwise to report their affective experience. A faceplate with a numerical 9-point scale is attached around the knob to indicate to the rater that 1 (i.e., 0 degrees) corresponds to “very negative” and 9 (i.e., 180 degrees) corresponds to “very positive.” Throughout the experiment, an electronic potentiometer captures the dial’s rotation 30 times per second and saves the second-by-second averages to a computer.

The affect rating dial has been used to study dyadic interactions, empathic accuracy, the impact of alcohol on anxiety, and emotional responses to film and music (see [2] for a review). Many of these applications, especially those using time-series analysis, would not have been possible without a continuous representation of affect.

The affect rating dial has demonstrated high validity in a number of ways. For instance, Gottman and Levenson [1] found that participants’ affect ratings during a conversation were consistent with their spouses’ ratings and also with observers’ objective coding. There was also consistency between participants’ physiological measures when they engaged in a conversation and when they later watched a video of that same conversation and used the affect rating dial.

Over time, numerous improvements have been made to the original affect rating dial (see [3] for an example involving colored lights). However, despite such refinements, many researchers would prefer an alternative that does not require them to purchase electronic components and assemble a custom-built device.

Numerous computer programs have been adapted or designed to collect continuous ratings. Many, such as ELAN [4] and ANVIL [5], are large packages capable of much more than continuous rating; as such, they are powerful in the hands of the experienced user but confusing and unwieldy to the uninitiated and impractical for use by study participants. Others, such as the Continuous Measurement System [6], EmuJoy [7], FeelTRACE [8],and Gtrace [9], are more focused on continuous rating. However, most have fallen out of repair and are difficult for non-programmers to implement and customize. EmuJoy and FeelTRACE also require raters to respond on a two-dimensional scale, which is difficult for users and increases the cognitive load of the task [10].

Given increasing interest in continuous affective measurement [11], [12], a new tool is required for the collection of such data. The current study presents CARMA, which fills this need by providing a focused media annotation solution that can be easily customized and used by researchers or participants on a variety of multimedia types.

CARMA was first developed as a computerized version of the affect rating dial to collect observer’s ratings of positive and negative affect in audiovisual recordings of social interactions. We have used previous versions of the software to explore the nonverbal and dyadic aspects of domestic violence [13], psychopathology [14], and other interesting psychological phenomena. By using these ratings as “ground truth” for supervised learning algorithms, we have also trained computerized systems to automatically analyse affect and facial expression intensity from data [15].

Implementation and architecture

CARMA is written in the MATLAB [16] programming language and compiled into a Windows application using the MATLAB Compiler. MathWork’s MATLAB is a commercially- available software package and programming language that is commonly used in psychology, humancomputer interaction, and related fields. Whilst the code for CARMA is released under an open license and can be executed using the free MATLAB Compiler Runtime(MCR) [17] software from MathWorks, any modifications to the code will require the developer to have the MATLAB Compiler to allow the revised code to be compiled and redistributed for use with the MCR. MCR is a stand-alone set of libraries that enables the execution of MATLAB files on computers without MATLAB. The appropriate version of MCR is automatically downloaded and installed with CARMA; the license terms for MCR are available as part of the download.

Graphical interface elements are implemented using MATLAB GUIDE [18], and multimedia playback is implemented using the Windows Media Player [19] ActiveX controller. It can load any multimedia file formats compatible with the version of Windows Media Player installed on the computer, including AVI, MPG, and WMV for video and AIF, CDA, MP3, WAV, and WMA for audio. Additional file types (e.g., M4A, MOV, and MP4) may be made compatible by upgrading Windows Media Player and/or installing additional codecs. Figure 1 shows the dependencies and interactions between CARMA’s components.

CARMA allows users to easily collect continuous measurements on a variety of multimedia file types. When a video file is selected, its soundtrack is played through the speakers and its images are displayed in the multimedia window. When an audio file is selected, its soundtrack is played though the speakers and customizable music visualization is displayed in the multimedia window. Figure 2 shows the main CARMA window during multimedia playback/rating.

Using a slider, whichcan be controlled with a computer mouse or keyboard via the arrow keys, users rate the selected multimedia file in real-time as it plays. Similar to the original affect rating dial, CARMA samples the position of the slider 10 times per second and saves the second-by-second means.

Adjacent to the slider is a customizable rating scale that provides a visual cue for the different slider positions. Users can easily specify the numerical range of this scale, provide labels for its upper and lower bounds, and specify the number of unique positions (or steps) it is composed of; users can also customize the gradient displayed on the scale by selecting any three colors. These settings can optionally be saved as defaults and loaded automatically. Figure 3 shows a screenshot of the CARMA Settings window.

At the conclusion of the multimedia file, the Annotation Viewer window is opened and the collected ratings are displayed (Figure 4). From here, the user may playback the multimedia file and view the collected ratings, or export the ratings to an annotation file. Previously exported annotation files can also be loaded into the Annotation Viewer window for playback or re-exporting.

When exporting an annotation file, the user can select an export location and file format for his or her ratings; export files can be either Comma-Separated Values (CSV) files or a Microsoft Excel Spreadsheet (XLS/XLSX) files (Figure 5). In addition to mean ratings for each second of the multimedia file and corresponding timestamps, export files contain metadata summarizing how CARMA was configured when the measurements were collected.

Sample multimedia and export files are provided in the code repository and step-by-step instructions for its use can be found in the documentation at http://carma.codeplex. com/documentation. Additional support is available at http://carma.codeplex.com/discussions or by emailing jmg174@pitt.edu .

Quality control

Functional testing has been carried out on Microsoft Windows XP (SP3), Microsoft Windows Vista, Microsoft Windows 7, and Microsoft Windows 8. CARMA behaved as expected on all platforms. Usability testing is currently being undertaken with researchers at the University of Pittsburgh and the University of Miami.

Operating system

This software has been successfully tested on Microsoft Windows XP (SP3), Microsoft Windows Vista, Microsoft Windows 7, and Microsoft Windows 8.

It was compiled to can run on both 32-bit and 64-bit machines. Due to its dependency on ActiveX, CARMA does not run on Linux or Mac.

Programming language

MATLAB 8.3 (R2014a)

Dependencies

• MATLAB Compiler Runtime 8.3 (R2014a) 32-bit Windows version
• Windows Media Player (≥9.0.0.0) via ActiveX

List of contributors

Jeffrey M Girard – coding, testing

Archive

Code repository

Language

English