Introduction

When you encounter a bear in the woods, you can be pretty sure that it’s a bear. You might recognize features from childhood stories, the Discovery channel, or maybe even previous encounters. But then, what if someone asks you to sit down, and write a definition for a bear? How might you start? Well, you might be somewhat confident that it’s a mammal, so you start with those features: having hair or fur, teeth, and being large. We don’t need to work very hard because a lot of work has already gone into defining the features of a mammal. Without listing them all, the creature in question also needs to have sweat and mammary glands, three middle ear bones, a neocortex, and a four chambered heart. Now, even if we could make our bear friend transparent and see into him to answer these questions, we are again left with the same conundrum when we step up to the next level of evaluation – what makes a bear different from any other animal? What features are especially “beary”?

The same conundrum exists for research software. We have a strong sense of what constitutes software – it is some kind of compiled or interpreted program that is run by a computer. But then, what distinguishes “research software” from all other software? And further, do we really care about a set of exact attributes, or are we interested in amassing some significant number of general features? To return to our previous example with bears, I might step back and decide that I care less about identifying the bear, but rather, identifying a creature that might present some danger to me. This changes our way of thinking entirely because instead of thinking about ear bones and mammary glands, we start to consider size, aggression, presence of teeth, and arguably much more useful features in the context of our use case. This brings us to the idea that context is important. If I care about finding an animal to train for a honey commercial, my criteria will be very different than if I care about identifying a beast that might eat me for dinner.

This kind of context is equally important when we discuss research software, as the needs of a group or individual clearly frame any subsequent evaluation. Although efforts such as FAIR [1] exist to ensure that software in the research domain is findable, accessible, interoperable, and reusable (FAIR), and there is work to define the life-cycle [2] or measuring of such software [7], these efforts focus on quality or best practices, which is a different task than definition. There is also often an implied bias that the definition is self explanatory, and that research software is simply software that is used in research [3, 6]. However, the missing component to these efforts is that definition depends on context. Definitions of research software for a specific purpose like applying for a grant or submitting to a journal are typically interested in a subset of software. As an example, we can look at definitions of research software in our research software engineering community to understand why context-specific definitions are important. The Journal of Open Source Software (JoSS), for example, defines research software as

software that: solves complex modeling problems in a scientific context (physics, mathematics, biology, medicine, social science, neuroscience, engineering); supports the functioning of research instruments or the execution of research experiments; extracts knowledge from large data sets; offers a mathematical library, or similar.

This domain-oriented definition would have a hard time including more general software such as application programming interfaces (APIs), supporting code for machine learning models, or databases. We can further look at a sample of rejected papers from JoSS to understand what was not considered research software. As of the writing of this paper, there are 319 rejected papers out of 3,584, and reviewing the first sample of 25 papers, 18 are rejected due to “Not substantial scholarly effort,” 5 for “Does not meet definition of research software,” and 1 for each of a “Desk rejection” and “Minor utility category.” While this is a small sample, the fact that papers can be rejected for these reasons suggests that the authors did not fully understand the submission criteria, or perhaps that the definition can be vague or subjective, or too narrow. If we look across all rejected papers, a total of 83 of the software reviews on GitHub (issues) have some mention of “research software” suggesting that it came up as part of the discussion. If we inspect a crowd-sourced definition [8] of research software engineering, we find that research software engineers work on a set of tools for

reproducibility, reusability, and accuracy of data analysis and applications created for research.

This suggests a different kind of focus on the goals of the software than criteria such as lines of code or domain. Finally, an even more generic definition comes from IGI Global, stating that research software is

A computer-based application that converts inputs into outputs to support the user in one or more research tasks.

The issue arises when we need to define research software in the context of a specific goal. There are several contexts under which we might find ourselves in a position of needing to define a piece of software as research software (or not):

• Funding bodies: If a funding body is evaluating software to determine who receives a grant, they would clearly need to have a definition. There cannot be any gray area about what constitutes research software, and what does not.
• Journals: Journals have traditionally been the means to share academic progress, and as software has been more acknowledged as an important part of research, we now see journals or sections of journals explicitly for research software. However, whether it’s conscious or not, most journals likely have some non-trivial or (externally appearing) subjective way to classify something as research software. Journals need to have transparency in these criteria, and the scope of research software they consider.
• People: In that developing research software is a core part of many individuals’ identities, having a definition is important to them.
• Universities and National Labs: In that these institutions conduct research that is empowered by research software, and need to employ and provide career perspectives for research software engineers and make funding and policy decisions for research, it’s essential that they be able to define it.
• Technology companies & startups: Firms commonly conduct software-based research towards their own business goals, often using tools developed in an academic environment and potentially feeding back to the wider community for example with insights and new, or improved, tools. These businesses would benefit from a definition for research software for instance to distinguish from other software they use or are responsible for, e.g. as a product.

While any particular context-driven definition is not inherently wrong, given the diversity of these different contexts and categories we suggest that the approach to define a single definition of research software is challenging if not impossible. Groups, organizations, or journals that need to more clearly communicate about a definition for research software need a different approach, as with the current approach authors might spend unnecessary time preparing a submission that is deemed to not fit the hard-to-understand definition. While there are currently efforts that eventually will discuss a definition for research software (e.g., The RD Research Alliance [1]), arguably an effort that is fully community driven, open, and has international asynchronous participation on GitHub would help to guarantee that a diversity of opinions across domains of science (e.g., life sciences, social sciences, digital humanities) are taken into account. This is the rationale behind this work for the Research Software Encyclopedia.

Implementation and Architecture

The Research Software Encyclopedia has several different tools and databases with core tools implemented in Python, with data storage in JSON, and web interfaces that use JavaScript, HTML, and CSS styling. These languages and technologies were chosen as they are well known in scientific programming, and would be easy for research software engineers to contribute to. The components include an explicit framework or algorithm to assess a piece of research software, a means to filter criteria points or categories (a taxonomy) for a given use case, and automation and web interfaces for interaction with a community database.

$ pip install rse[all]
$ rse init
$ rse add https://github.com/singularityhub/singularity-hpc
$ rse ls
1 github/singularityhub/singularity-hpc
$ rse add https://github.com/dask/dask
$ rse ls
1 github/dask/dask
2 github/singularityhub/singularity-hpc
$ tree database
database
        github
              dask
                 dask
                    metadata.json
        github
              singularityhub
                 singularity-hpc
                     metadata.json

#!/bin/bash
export RSE_HOST=https://rseng.github.io
export RSE_URL_PREFIX=/software/
export RSE_CONFIG_FILE=rse.ini
rse export --type repos-txt repos.txt --force
rse export --type static-web docs/

Quality control

For the Research Software Encyclopedia, along with the taxonomy and criteria repository, and software database, tests are run via continuous integration for each pull request or merge into the main branch. For the Research Software Encyclopedia command line software, tests include testing the functionality and output of parsers (e.g., GitHub, GitLab), along with ensuring that all commands produce expected output. The criteria and taxonomy repository and tool also tests the functionality of the main commands, and output contents. For the software database, the quality control comes from the sources. For example, papers and associated software that goes into JoSS goes under a formal review process, and the other software databases are curated by teams of research software engineers.

Operating system

The Research Software Encyclopedia should work on most Unix and Linux flavored distributions, or those that can run Docker. The software was developed on Ubuntu 18.04 and 20.04.

Programming language

The Research Software Encyclopedia set of tools supports Python 3.5 and higher. Python 2.x is not supported.

Additional system requirements

To interact with a relational database (e.g., MySQL or Postgres) the system would need to install the database software natively, or run via a Docker container.

Dependencies

The Research Software Encyclopedia requires the Python requests library for basic function, several Flask libraries for advanced use of the interface, and pytest for testing. See the repository version.py file for details.

List of contributors

All authors contributed to the development of criteria and taxonomy items. Vanessa Sochat is the primary author of the software, documentation, and interfaces.

Software location

Name: rse-0.0.34.tar.gz

Persistent identifier: https://zenodo.org/record/5546046#.YVi-uXtMFH4

Package manager: https://pypi.org/project/rse/0.0.34/

License: MPL 2.0

Publisher: Vanessa Sochat

Version published: 0.0.34

Date published: January 28, 2022

Criteria and Taxonomy Archive pypi is used as a package manager for releases.

Name: rseng-0.0.18.tar.gz

Persistent identifier: https://zenodo.org/record/5546052#.YVi-43tMFH4

Package manager: https://pypi.org/project/rseng/0.0.18/

License: MPL 2.0

Publisher: Vanessa Sochat

Version published: 0.0.18

Date published: December 6, 2020

Research Software Encyclopedia Code repository GitHub

Name: https://github.com/rseng/rse

Persistent identifier: https://zenodo.org/record/5546054#.YVi_FHtMFH4

License: MPL 2.0

Date published: November 27, 2020

Taxonomy and Criteria Code repository GitHub

Name: https://github.com/rseng/rseng

Persistent identifier: https://github.com/rseng/rseng

License: MPL 2.0

Date published: September 28, 2020

Language

The software is implemented in Python, with supporting scripts in bash for testing, and configuration files in yaml.