Introduction

Advances in audio processing technology often employ a series of objective and subjective tests to compare performance against previous algorithms. In the case of subjective listening tests, the most commonly used methodologies include; Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) [1], Absolute Category Rating (ACR) [2] and A/B comparison testing [2]. Increasingly, these tests are being conducted online using browser based technology and it has been shown that the results for online tests correlate well with those obtained in controlled laboratory conditions [3]. Typically, a researcher will design a bespoke testing system or extend from an existing framework. The most common frameworks are briefly described here.

The common open-source frameworks developed for research purposes are BeaqleJS [4], Web Audio Evaluation Tool (WAET) [5], webMUSHRA [6], TheFrageboden [7], and the P.808 Toolkit [8]. These JavaScript-based frameworks allow for online listening tests without proprietary access to third party software such as MATLAB [9] and MAX [10]. The advancement in the Web Audio API also made synchronous and flexible playback possible in the browser, which contributes to the versatility of the tests forms available. Most frameworks have incorporated the standard subjective assessment procedures including ITU-R.BS.1284 (for general audio), ITU-R.BS.1116 (for audio with small impairments), and ITU-R.BS.1534 (for audio with intermediate impairments) to ensure the tests are standard complied. The forms of the standard test procedures can also be used for customised research scenarios.

The frameworks differ in their supports for customisation to suit various experiment requirements. BeaqleJS mainly supports A/B and MUSHRA tests or any modified tests based on the two predefined tests forms; the webMUSHRA and WAET allow researchers to build their tests based on a wider range of tests forms and rating scales; TheFragebogen has extra support for behaviour research by incorporating behavioural scales (the NASA task load index (TLX) and the visual analogue scale (VAS)), allowing for free-hand input, and documenting user’s response time; the webMUSHRA provides 2D and 3D GUIs to facilitate the report of localisation judgement for spatial audio assessment. In response to the challenge of the environment control in remote tests, different frameworks also provide solutions accordingly. TheFragebogen’s response time records are used to scrutinize anomalous user behaviours; the P.808 Toolkit implements the standard crowdsourcing methods [11] which test participant’s hearing ability, headphone usage, and the environment to control for the test conditions remotely. The P.808 Toolkit is a welcome addition but currently only runs on the Amazon Mechanical Turk (AMT), a common platform for crowdsource experiments, to integrate the experiment design, setup, and recruitment procedures in one place. These frameworks are admirable but they require coding knowledge and a significant time investment to successfully configure and connect them to a suitable database layer along with deployment on a secure publicly accessible server.

Our goal in creating this system was to allow any researcher to create and share listening tests easily without the need for coding, managing deployment or hosting. To that end, we have created GoListen which is a fully hosted end-to-end listening test platform with support for several popular listening test types along with a variety of survey question types. We are making the source code available but the key advantage of our system is that we are also providing a fully hosted web application. In total, we provide three options for using the system.

• Web Application – Designed for users with no coding skills or those who want the convenience of a fully hosted system. https://golisten.ucd.ie/.
• Docker and AWS images – Designed for users with moderate technical skills who would prefer a self hosted option.
• Source code – Designed for those with expert coding skills who wish to extend or customise the system for specialised use.

Software Features

The system we present here, allows users to create and share a test in minutes with no code or setup required. Currently supported test types include A/B, ACR and MUSHRA with more test types to follow. Survey questions can be inserted before, after and between audio examples. Question types include multiple choice, check box and free input text answers. Audio playback for each question can be set to loop infinitely or a fixed number of times. The subject can also be required to listen to each audio example in full any number of times before being allowed to proceed. Switching between audio examples is fully synchronised, meaning that audio playback continues uninterrupted regardless of which audio condition is being listened to. Subjects may also loop sections of audio if enabled by the test creator. Individual questions can be optional or required for both audio examples and survey questions. Questions and audio examples can be viewed on individual pages or the full test can be configured as a single page view. Audio test order can be randomised for all test types and stimulus presentation order can be randomised within a MUSHRA test. The system also allows for timing each question. In terms of the test builder, questions can be reordered and edited easily. Tests can be created from a blank template or a predefined template. They can also be edited and duplicated. When the results have been collected, they can be downloaded in CSV and JSON format.

In the following sections, we present the user interface and illustrate how it can be used to create tests.

Dashboard View

The user interface is organised in a familiar manner whereby the main navigation is presented in the left pane and the content is presented in the right pane. When a user logs in, they are presented with the dashboard. This is the “home” view and most actions start from here.

Creating Tests

Creating a new test is achieved by tapping the Add Test button, Figure 1. The user has the option of selecting a blank test or a pre-populated template which is created by administrators of the system. Upon selection, the test design screen opens which is discussed in a later section.

Figure 1 

Dashboard screen, where tests can be added, copied, edited and shared.

Copying Tests

Another way to create a test is to copy an existing test which you may have created in the past. Tap on the Copy icon and a new test will appear in your dashboard with the same name as your copied test and “copy” appended to the end. The new test can be renamed and will not contain the responses from the parent test.

Edit a test

Any existing test can be edited from the dashboard by tapping on the Edit icon. This brings the user back into the test design screen which will be discussed below. Note, if a test has already been shared and received responses, it cannot be edited without making a new copy.

Sharing Tests

A test can be shared by tapping on the share icon. This presents a modal allowing the user to copy the public link for the test or to launch the test in a new browser window. The test can be shared from the dashboard or from the test design screen.

Designing Tests

Designing a test is easy using the set of editing tools the system provides. Every test is given a name and description so that the user can locate the test on the dashboard later on, Figure 2.

Figure 2 

Adding Survey Questions.

Add Survey Questions

Survey questions can be added by tapping the Add Question button which reveals a context menu listing various question types, Figure 2. Options include:

• Checkbox Group – Allows for multiple options to be selected simultaneously. Figure 3 shows the question design interface.
• Radio Group – Allows for a single option from a list of options to be selected. Figure 4 shows the question design interface.
• Text Input – Allows answers to be entered as free text. Figure 5 shows the question design interface.
• Text Label – Insert a large text label to introduce a new section in a test.

Figure 3 

Multiple Selection Question.

Figure 4 

Singular Selection Question.

Figure 5 

Text Input Question.

Adding Audio Questions

Audio questions can be added by tapping the Add Question button which reveals a context menu listing various question types, Figure 2. Audio Options include:

• Audio Test – Allows for an AB, ACR or MUSHRA test question to be added. Each audio test type is discussed in more detail in a later section. Figure 6 shows the question design interface for an A/B test. Audio can be uploaded by clicking on or dragging to the dropzone in the interface. The preference question text can be edited as can the options presented to the respondent.
• Audio Training Example – An audio training example is a training step for the respondent. The test designer can provide instructions and an audio example without asking a question related to the example.

Figure 6 

AB Audio Question.

Audio Playback Settings

Each audio question has a settings menu which controls how the respondent will interact with audio playback as seen in Figure 7. The audio formats supported are those described in the HTML5 Audio specification [12]. The options are as follows:

• Set the number of times audio will loop (range 1 to infinity)
• Require the respondent to listen to the audio in full
• Disallow the respondent to skip back and forth in time
• Allow the respondent to use looping controls

Figure 7 

Audio Playback Settings.

Previewing a Test

As you build the test in the test design screen, you may want to check what it will look like for the respondent. This can be achieved by tapping on the preview icon which can be found on the tool bar which appears at the top and bottom of the test design screen as seen in Figure 2.

Global Test Settings

Every test has a set of global parameters which can be accessed by tapping on the gear icon in the tool bar at the top of the test design screen. Options include:

• A test can be shown in a single scrollable web page or each question can be shown on its own screen.
• Each question can also be timed in order to assess the cognitive load on the respondent.

Audio Test Types

The system currently supports the following tests: Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) [1], Absolute Category Rating (ACR) [2] and A/B comparison testing [2]. In all test types, audio switching is synchronous and seamless. This allows the respondent to switch audio conditions without hearing any discontinuities in the audio playback. A loop tool is optional in all test types.

A/B Comparison

An A/B test is used to compare two versions of the same piece of audio where each version has been subjected to a different version of the same process. When presented with the test, the respondent will see an interface similar to that in Figure 8. The respondent may interact in the following ways:

• Play the audio
• Switch between a reference and condition A or B synchronously.
• Set loop start and end points for repetition.
• Change the playback position between the start and end of the clip.
• Select a preference from a radio group containing multiple options.

Figure 8 

AB Participant View.

Absolute Category Rating (ACR)

An ACR test is used to assess the quality of a piece of audio on a labeled 5 point scale. When presented with the test, the respondent will see an interface similar to that in Figure 9. The respondent may interact in the following ways:

• Play the audio.
• Set loop start and end points for repetition.
• Change the playback position between the start and end of the clip.
• Select perceived quality from a 5 point slider

Figure 9 

ACR Participant View.

Multiple Stimuli with Hidden Reference and Anchor (MUSHRA)

A MUSHRA test is used to assess the relative quality of multiple audio stimuli on a continuous scale ranging from 0 to 100. Within the stimuli there is a hidden reference and an anchor. The test is more flexible than the ITU-R BS.1534-3 standard and allows the user to add or omit anchors. When presented with the test, the respondent will see an interface similar to that in Figure 10. The respondent may interact in the following ways:

• Play the audio.
• Switch between a reference and multiple stimuli synchronously.
• Set loop start and end points for repetition.
• Change the playback position between the start and end of the clip.
• Select perceived quality of each of the stimuli on a continuous slider in the range 1–100.

Figure 10 

MUSHRA Test Interface.

Randomisation

Test order randomisation is supported for all audio test types. This is achieved during survey creation by adding a Group Divider before and after a group of audio tests as shown in Figure 11. Randomisation of stimuli within a single MUSHRA audio test is also supported and can be activated within the question settings menu of any MUSHRA audio example as seen in Figure 12.

Figure 11 

Test Order Randomisation.

Figure 12 

Stimuli Randomisation within a MUSHRA test.

Accessing Test Results

Once the link to a test has been shared, the system will collect responses from any respondent who completes the test. The test owner can check the responses at any time from the Responses tab at the top of the test design screen, Figure 2.

The response view shown in Figure 13 allows a user to:

• View and delete individual responses.
• Delete all responses.
• Download the results in CSV format.
• Download the results in JSON format.

Figure 13 

Results View.

Respondents also have the ability to delete their response using a unique link furnished to them at the end of a completed test. In this instance, the test owner will receive a notification requesting that the updated results be downloaded.

Accounts

A test owner may update their password or delete their account from the Accounts screen (see Figure 14) which can be accessed from the navigation menu in the left pane, Figure 1.

Figure 14 

Account Management View.

Implementation and architecture

The software is separated into two independent parts: a front-end and a back-end which serves a database. These two parts are currently served on the same server with different processes but could easily be served independently in a microservices style architecture. The front-end is implemented in React.js and requests data from the back-end which uses Python to make requests to the database which is implemented in MongoDB. The architecture is RESTFul. In addition to source code, we have also provided the software both as a Docker install and an AWS AMI (Amazon Machine Image). Details can be found in the GitHub repository [13].

Quality control

Some components contain unit testing within the code but in general, end-to-end testing was primarily used to ensure the quality of the software. Alpha testing was carried out by 5 researchers at QxLab in University College Dublin. Testers were instructed to create variants of each test type and to provide test responses in order to ensure that the data is collected and stored as expected. This phase of testing also revealed many useful user experience optimisations. Beta testing was carried out by conducting two real-world studies using the software. The software was then released to some invited universities and companies where more feedback was gathered.

Operating system

The software can be run on many operating systems. The following systems were used during testing.

• Ubuntu 18.04 and Ubuntu 16.04
• Mac OSX 10.15 (Mojave)
• Windows 10

Programming languages

• Python 3.8.0
• Node.js 12.16.1
• Node.js (React, NPM)
• NPM 6.13.4
• Typescript 3.9.5
• Python (Tornado, PYMongo)
• MongoDB
• Docker scripts and compose scripts (yaml)
• Nginx configuration file (conf)
• Supervisor configuration file (conf)
• Update delivery scripts (Python)

Additional system requirements

Minimum system requirements:

• 1GB memory
• 8GB disk space
• SSH access to the server which necessarily requires access to the Internet

Dependencies

• material-ui/core: 4.10.0
• material-ui/lab: 4.0.0-alpha.56
• testing-library/jest-dom: 4.2.4
• testing-library/react: 9.5.0
• testing-library/user-event: 7.2.1
• types/jest: 25.2.3
• types/node: 14.0.11
• types/react: 16.9.35
• types/react-dom: 16.9.8
• types/react-router: 5.1.7
• types/react-router-dom: 5.1.5
• axios: 0.19.2
• formik: 2.1.4
• mobx: 5.15.4
• mobx-react: 6.2.2
• mobx-utils: 5.6.1
• react: 16.13.1
• react-dom: 16.13.1
• react-router: 5.2.0
• react-router-dom: 5.2.0
• react-scripts: 3.4.3
• ts-md5: 1.2.7
• typescript: 3.9.5
• uuidv4: 6.1.1
• react-markdown: 5.0.3
• pymongo: 3.11.0
• tornado: 6.0.4

List of contributors

• Qijian Zhang: Lead Developer
• Dan Barry: Project Manager, UX/UI, Developer
• Pheobe Wenyi Sun: Research and Testing
• Alessandro Ragano: Research and Testing
• Andrew Hines: QxLab Director, Research and Testing

All contributors are members of QxLab at University College Dublin, Ireland.

Software location