Start Submission Become a Reviewer

Reading: Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (...


A- A+
Alt. Display

Issues in Research Software

Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3)


Daniel S. Katz ,

National Center for Supercomputing Applications (NCSA) & Electrical and Computer Engineering (ECE) Department & School of Information Sciences (iSchool), University of Illinois at Urbana–Champaign, US
X close

Sou-Cheng T. Choi,

NORC at the University of Chicago and Illinois Institute of Technology, US
X close

Kyle E. Niemeyer,

School of Mechanical, Industrial, and Manufacturing Engineering, Oregon State University, US
X close

James Hetherington,

Research Software Development Group, University College London, GB
X close

Frank Löffler,

Center for Computation & Technology, Louisiana State University, US
X close

Dan Gunter,

Lawrence Berkeley National Laboratory, US
X close

Ray Idaszak,

RENCI, University of North Carolina at Chapel Hill, US
X close

Steven R. Brandt,

Lawrence Berkeley National Laboratory, US
X close

Mark A. Miller,

University of California, San Diego, US
X close

Sandra Gessing,

Center for Research Computing, University of Notre Dame, US
X close

Nick D. Jones,

New Zealand eScience Infrastructure (NeSI), University of Auckland, NZ
X close

Nic Weber,

University of Washington, US
X close

Suresh Marru,

Indiana University, US
X close

Gabrielle Allen,

National Center for Supercomputing Applications & Department of Astronomy, University of Illinois at Urbana-Champaign, US
X close

Birgit Penzenstadler,

College of Computer Engineering & Computer Science, California State University, US
X close

Colin C. Venters,

School of Computing and Engineering, University of Huddersfield, GB
X close

Ethan Davis,

UCAR Unidata, US
X close

Lorraine Hwang,

University of California, Davis, US
X close

Ilian Todorov,

Science & Technology Facilities Council, GB
X close

Abani Patra,

Mechanical and Aerospace Engineering, University at Buffalo, US
X close

Miguel de Val-Borro

Department of Astrophysical Sciences, Princeton University, US
X close


This report records and discusses the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3). The report includes a description of the keynote presentation of the workshop, which served as an overview of sustainable scientific software. It also summarizes a set of lightning talks in which speakers highlighted to-the-point lessons and challenges pertaining to sustaining scientific software. The final and main contribution of the report is a summary of the discussions, future steps, and future organization for a set of self-organized working groups on topics including developing pathways to funding scientific software; constructing useful common metrics for crediting software stakeholders; identifying principles for sustainable software engineering design; reaching out to research software organizations around the world; and building communities for software sustainability. For each group, we include a point of contact and a landing page that can be used by those who want to join that group’s future activities. The main challenge left by the workshop is to see if the groups will execute these activities that they have scheduled, and how the WSSSPE community can encourage this to happen.

How to Cite: Katz, D.S., Choi, S.-C.T., Niemeyer, K.E., Hetherington, J., Löffler, F., Gunter, D., Idaszak, R., Brandt, S.R., Miller, M.A., Gessing, S., Jones, N.D., Weber, N., Marru, S., Allen, G., Penzenstadler, B., Venters, C.C., Davis, E., Hwang, L., Todorov, I., Patra, A. and de Val-Borro, M., 2016. Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3). Journal of Open Research Software, 4(1), p.e37. DOI:
  Published on 21 Oct 2016
 Accepted on 05 Sep 2016            Submitted on 06 Feb 2016

(1) Introduction

The Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3)1 was held on 28–29 September 2015 in Boulder, Colorado, USA. Previous events in the WSSSPE series are WSSSPE12 [1, 2], held in conjunction with SC13; WSSSPE1.13, a focused workshop organized jointly with the SciPy conference4; WSSSPE25 [3, 4], held in conjunction with SC14; and WSSSPE2.16, a focused workshop organized again jointly with SciPy7.

Progress in scientific research is dependent on the quality and accessibility of software at all levels. Hence it is critical to address challenges related to development, deployment, maintenance, and overall sustainability of reusable software as well as education around software practices. These challenges can be technological, policy based, organizational, and educational; and are of interest to developers (the software community), users (science disciplines), software-engineering researchers, and researchers studying the conduct of science (science of team science, science of organizations, science of science and innovation policy, and social science communities). The WSSSPE1 workshop engaged a broad scientific community to identify challenges and best practices in areas of interest to creating sustainable scientific software. WSSSPE2 invited the community to propose and discuss specific mechanisms to move towards an imagined future for software development and usage in science and engineering. But WSSSPE2 did not have a good way to enact those mechanisms, or to encourage the attendees to follow through on their intentions.

The WSSSPE3 workshop included multiple mechanisms for participation and encouraged team building around solutions. WSSSPE3 strongly encouraged participation of early-career scientists, postdoctoral researchers, graduate students, early-stage researchers, and those from underrepresented groups, with funds provided to the conference organizers by the Moore Foundation, the National Science Foundation (NSF), and the Software Sustainability Institute (SSI) to support the travel of potential participants who would not otherwise be able to attend the workshop. These funds allowed 16 additional people to attend and participate.

WSSSPE3 also included two professional event organizers/facilitators from Knowinnovation who helped the organizing committee members plan the workshop agenda, and during the workshop, they actively engaged participants with various tools, activities, and reminders.

This report is based on collaborative notes taken during the workshop, which were linked from the GitHub issues that represented the potential and actual working groups8. Overall, the report discusses the organization work done before the workshop (§2), the keynote (§3), and the lightning talks presented at the meeting (§4). The report also gives summaries of action plans proposed by the working groups (§5), then gives longer descriptions of the activities that occurred in each of the working groups that made substantial progress (§6), and provides some conclusions (§7). The appendices contain lists of the organizing committee (Appendix A), the registered attendees (Appendix B), and the travel award recipients (Appendix C).

(2) Calls for Participation

WSSSPE3 was based on the work done in WSSSPE1 and WSSSPE2, but aimed at starting a process to make progress in sustainable software, as the calls for participation said:

The WSSSPE1 workshop engaged the broad scientific community to identify challenges and best practices in areas relevant to sustainable scientific software. WSSSPE2 invited the community to propose and discuss specific mechanisms to move towards an imagined future practice of software development and usage in science and engineering. WSSSPE3 will organize self-directed teams that will collaborate prior to and during the workshop to create vision documents, proposals, papers, and action plans that will help the scientific software community produce software that is more sustainable, including developing sustainable career paths for community members. These teams are intended to lead into working groups that will be active after the workshop, if appropriate, working collaboratively to achieve their goals, and seeking funding to do so if needed.

The first call for participation requested lightning talks, where each author could make a brief statement about work that either had been done or was needed, with the goal of contributing to the discussion of one or more working groups. There were 24 lightning talks submitted; after a peer-review process, 16 were accepted, as discussed further in Section 4.

The first call also discussed the potential action topics that came out of WSSSPE2, and requested additional suggestions. The combination of existing and new topics led to the following 18 potential topics that were advertised in the subsequent calls for participation:

  • Development and Community
    • – Writing a white paper/review paper about best practices in developing sustainable software
    • – Documenting successful models for funding specialist expertise in software collaborations
    • – Creating and curating catalogs for software tools that aid sustainability (perhaps categorized by domain, programming languages, architectures, and/or functions, e.g., for code testing, documentation)
    • – Documenting case studies for academia/industry interaction
    • – Determining effective strategies for refactoring/improving legacy scientific software
    • – Determining principles for engineering design for sustainable software
    • – Create a set of guidance giving examples of specific metrics for the success of scientific software in use, why they are chosen, what they are useful to measure, and any challenges/pitfalls; then publish this as a white paper
  • Training
    • – Writing a white paper on training for developing sustainable software, and coordinating multiple ongoing training-oriented projects
    • – Developing curriculum for software sustainability, and ideas about where such curriculum would be presented, such as a summer training institute
  • Credit
    • – Hacking the credit and citation ecosystem (making it work, or work better, for software)
    • – Developing a taxonomy of contributorship/guidelines for including software contributions in tenure review
    • – Documenting case studies of receiving credit for software contributions
    • – Developing a system of awards and recognitions to encourage sustainable software
  • Publishing
    • – Developing a categorization of journals that publish software papers (building on existing work), and case studies of alternative publishing mechanisms that have been shown to improve software discoverability/reuse, e.g., popular blogs/websites
    • – Determining what journals that publish software paper should provide to their reviewers (e.g., guidelines, mechanisms, metadata standards)
  • Reproducibility and Testing
    • – Building a toolkit that could allow conference organizers to easily add a reproducibility track
    • – Documenting best practices for code testing and code review
  • Documentation
    • – Develop landing pages on the WSSSPE website (or elsewhere) that enable the community to easily find up-to-date information on a WSSSPE topic (e.g., software credit, scientific software metrics, testing scientific software).

(3) Keynote

WSSSPE3 began with a keynote speech delivered by Professor Matthew Turk from the Department of Astronomy, University of Illinois, titled Why Sustain Scientific Software?. Turk is a prolific scientific software practitioner and has extensive experience working on large collaborative projects employing modern computing tools [5]. He also co-organizes and champions WSSSPE events.

In his keynote address, Turk recapped the course of development of WSSSPE workshops over the past few years, alongside his career development from a postdoc to an academic. The first WSSSPE workshop was at the Supercomputing conference (SC13) in 2013, but he observed that the notion of sustainable scientific software drew in an audience beyond supercomputing. In the following year, WSSSPE1.1 at SciPy had speakers talking about how software has been sustained inside the scientific Python community. WSSSPE2 at SC14 had breakout group discussions coming up with actionable items, and WSSSPE2.1 at SciPy 2015 was similar. Turk noted the different atmosphere of the surrounding large conferences, despite similar WSSSPE participants.

WSSSPE3 left the traditional Supercomputing Conference environment this year, and in Turk’s words, this change spoke to the fact that scientific software comes from many different types of inquiries, deployment, strategies for maintenance, users, and ways of measuring the value of a piece of software. It appeared to Turk that the supercomputing community generally adopts some top-down approaches, whereas the SciPy community more often than not uses more bottom-up systems. According to Turk, there is a divergence in views about progress in software: the supercomputing community thinks that software is getting harder, with exascale computing and optimization issues in mind; but the SciPy community thinks that software is becoming better, with emerging tools such as Jupyter and productivity packages for research workflows. Admitting such comparisons are somewhat unfair generalizations, Turk reminded the audience that the different approaches bring different types of ideas to the table, and he welcomed WSSSPE3 being conducted outside existing preconceptions.

Returning to the topic of his talk, Turk invited the audience to picture scientific software as a flower on a landscape under the Sun, which may represent a number of measurable factors such as number of citations; growth of a community and number of contributors; amount of funding; prestigious prizes awarded; stability of the community in terms of leadership transitions, serving community needs, not breaking test suites, and performance on new architectures. But all these metrics are strictly speaking proxies for the values and the impact scientific software bears. What we can measure does not give us direct insight—it just gives us proxies of insight.

Turk then moved onto various different definitions of sustainability. His favorite one was “keeping up with bug reports,” where even if no new features were added, the software remains sustainable. Another definition of sustainability Turk mentioned was “adding of new features,” or “maintaining the software for a long period of time” such as the cases of TEX or LATEX with community help. A notion Turk heard often at supercomputing conferences was that sustainable software “continues to work on new architectures.” Yet another metric was “people continuing to be able to learn how to use and apply the software.” A funder Turk heard talked about sustainability as “continuing to get funded.” Turk also recalled that Greg Wilson, among others, said in WSSSPE1.1 that his view of sustainable software was software that “continued to give the same results over time.” A last measure of sustainability Turk presented was “the ability to transition between different people developing and using a piece of software.”

At WSSSPE1, several models were presented for ensuring sustainability. Turk considered that a familiar one was a funded piece of software where an external agency provided funds to a group who are not necessarily exclusively working on and developing the software, keeps it going, and provides it to the scientific community. The model of productized software, in which a piece of software has grown to the point that research groups or people are willing to support it with some amount of funding, for instance, a subscription to use cloud services that deploy a piece of software, or purchase of a piece of software. A final model Turk felt conflicted about is a volunteer model that is traditional old-school—not modern-day open source—development.

Turk discussed whether productizing scientific software was synonymous with being sustainable and self-sufficient. He thought it was not necessarily the case and furthermore, it could lead to a divergence of interests between users and developers.

Turk reminded the audience that the volunteer model means unpaid labor. On this note, he recommended Ashe Dryden’s blog post on the ethics of unpaid labor and the open source software community9. Oftentimes, a person funded to work full time on a scientific project can spend a small amount of time for working on a piece of software necessary for that project. However, researchers’ abilities to participate in that volunteer community are not always the same and may not always be aligned with their research projects. From Turk’s experience, we cannot always rely on unpaid labor and volunteer time to sustain a piece of software—this came down to the notions of the top-down and the bottom-up approaches, i.e., the funded versus the grassroots. However, Turk pointed out that bottom-up, volunteer-driven projects can be just as large-scale as a top-down software development project.

Turk said that sustaining scientific software really meant to him conducting scientific inquiries, often by some specific software, and sustaining the people we care about, our careers, and the future of our fields. According to Turk, we all have an invested stake in sustaining scientific software. Hence, having “sustained” projects can suffocate new projects, so we need to make sure we don’t cause novel ideas and packages to suffer at the hands of the status quo.

Turk talked about possible reasons why we want to sustain scientific software: devotion to science and interests in pursuing the next stage of research; fun and creative thrill in writing codes and papers; usefulness with measurable impacts, for example, LINPACK and HDF groups providing data storage to satellites, which goes beyond usefulness to necessity. Lastly, Turk presented his wishlist of questions to be answered in the future:

  • How do we ship a product on time when dealing with a mix of funding models and motivations especially when we rely on volunteers?
  • How do we know when it is time to end some software and move on? For example, should we stop sustaining Python and switch to Julia and Javascript?
  • How can productized software balance its future versus its past, or the emerging needs of the customers versus the existing needs of the development community?
  • How can we help avoid burnout and retain the joy in the communities?
  • How can we reduce systemic bias, which goes back to Dryden’s blog post especially on how ethics of unpaid labor disproportionately affect underrepresented communities?

(4) Lightning Talks

After the keynote, WSSSPE3 continued with lightning talks. These short talks were intended to give an opportunity for attendees to quickly highlight an important issue or a potential solution.

  1. Benjamin Tovar and Douglas Thain: Freedom vs. Stability: Facilitating Research Training While Supporting Scientific Research. Benjamin Tovar presented a case study of the Cooperative Computing Lab (CCL)10 at the University of Notre Dame, which is a small group of individuals whose main tasks are collaborating with people that have large-scale computing problems, operating various parallel computer systems, conducting computer science research, and developing open source software. One of the main challenges they face is finding a balance between flexibility/training and stability/quality. Their current solution for ensuring the latter was to add a software engineer (the presenter) to the existing team of faculty and students, who now also serves as a “spring” between flexibility and stability.
  2. Birgit Penzenstadler, Colin Venters, Christoph Becker, Stefanie Betz, Ruzanna Chitchyan, Letícia Duboc, Steve Easterbrook, Guillermo Rodriguez-Navas, and Norbert Seyff: Manifesting the Ghost of the Future: Sustainability. The concept of sustainability has become a topic of interest in the field of computing, which is evidenced by the increase in the number of events that focus on the topic. Nevertheless, it is not well understood yet. Birgit Penzenstadler argued that we often define sustainability too narrowly. Instead, sustainability at its heart is a systemic concept and must be viewed from a range of different dimensions including environmental, economic, individual, social, and technical. She introduced the Karlskrona Manifesto on Software Design [6], which distills knowledge from a broad range of related work on the topic of sustainability into a set of (mis-)perceptions and principles. The manifesto does not proclaim that there is an easy, one-size-fits-all solution around the corner, but rather points out that sustainability is a “wicked problem” and is often misunderstood. Due to these misperceptions, even though sustainability’s importance is increasingly recognized, many software systems are unsustainable. Even more alarming is that most software systems’ broader impacts on sustainability are unknown. To change this, the Karlskrona Manifesto proposes nine principles and commitments. These commitments are not dogmatic laws, but rather commitments to rethink, to move beyond the silo mentality, and to analyze in more depth. As such, they do not restrict, but rather open up a space for discussion.
  3. Abani Patra, Hossein Aghakhani, Nikolay Simakov, Matthew Jones, and Tevfik Kosar: Integrating New Functionality Using Smart Interfaces to Improve Productivity of Legacy Tools. Abani Patra presented an example of how the community using Titan2D, a geoflow simulation software, increased the productivity of their tools by improving both code and data layout [7]. The main obstacles in this change were the non-existence of a common version control system for the source code, coupled with multiple versions of the same code base, the fixed format of input files, that many input values were set as compilation flags, and that the internal data layout was not suitable for modern technologies (e.g., vectorization, accelerators). The approaches of the Titan2D developers included reinforcing the code structure using multiple layers of Python and C++ interfaces, and a redesign of the data layout to be more suitable for modern CPUs and accelerators.
  4. Abigail Cabunoc Mayes, Bill Mills, Arliss Collins, and Kaitlin Thaney: Collaborative Software Development as Sustainable Software: Lessons from Open Source. Abigail Cabunoc Mayes combined two properties of open source that, together, create a suitable habitat for sustainable software. The first of the two properties, public, does not only mean public code. It also includes public discussions, a public process of including contributions, and an open license. The second property, participatory, stresses the importance of reaching out to the community and helping potential new members by providing better documentation and learning experiences, like code review and examples of good first bug reports. Together, Abigail concluded, these two properties not only lead to higher quality, reusability, and ease of understanding, but also eventually, to sustainability.
  5. Louise Kellogg and Lorraine Hwang: Advancing Earth Science through Best Practices in Open Source Software: Computational Infrastructure for Geodynamics. Lorraine Hwang presented experiences with the Computational Infrastructure for Geodynamics (CIG)11, a community software with a worldwide user base. Like others, their main goals include high usability, sustainability, and reproducibility. As a means to achieve these goals, various communication channels have been developed, such as mailing lists, wikis, workshops, hackathons, tutorials, and webinars. In order to contribute to the infrastructure, codes must adhere to specified minimum standards with the desire that all codes are working toward target standards. These include, e.g., the use of version control, certain coding styles, the presence and nature of code tests and documentation, and certain user workflows.
  6. Lorraine Hwang, Joe Dumit, Alison Fish, Louise Kellogg, Mackenzie Smith, and Laura Soito: Software Attribution for Geoscience Applications in the Computational Infrastructure for Geodynamics. In a second talk, Lorraine Hwang mentioned a variety of ways to cite efforts within the SAGA framework12, including science papers, code papers, user manuals, and the CIG website. An analysis of the resulting citations showed that 80% of papers that use CIG codes mention the code name, and about the same number includes a citation. Only about 20% acknowledge CIG. Within the same sample of papers, about one fifth use an URL to cite codes (including non-CIG codes), and only about one eighth specify the version used. Compared to other codes, CIG seems to be much better cited. In part, this is attributed to the fact that CIG requires that donated software provide a citable paper specified in the User Manual. The project is working on tools and methods to generate attribution information automatically.
  7. Mike Hildreth, Jarek Nabrzyski, Da Huo, Peter Ivie, Haiyan Meng, Douglas Thain, and Charles Vardeman: Data And Software Preservation for Open Science (DASPOS). DASPOS13 is an NSF-funded multi-disciplinary effort, located at Notre Dame and Chicago, that links the high energy physics effort to other disciplines such as biology, astrophysics, and digital curation. It includes physicists, digital librarians, as well as computer scientists, and aims to achieve some commonality across disciplines. Examples are meta-data descriptions of archived data, computational descriptions, descriptions of how data was processed, questions such as whether computation replication can be automated, and what the impact of access policies on the preservation infrastructure is. One of the products of this effort is a suite of tools that deals with this preservation [8, 9, 10, 11], and the questions was posed whether that software itself is sustainable. Points that were brought up included the need for a user community depending on a given software, and the need to provide added value for its users. An important method to achieve this was to work with the user community from the start, and to budget that way. For the specific example of the preservation software, this means that besides adding value to the community, it needs to be transparent to their workflows, i.e., not requiring additional effort to preserve “research objects.”
  8. James Hetherington, Jonathan Cooper, Robert Haines, Simon Hettrick, James Spencer, Mark Stillwell, Mike Croucher, Christopher Woods, and Susheel Varma: An update from UK Research Software Engineers. James Hetherington started by listing some of the problems research software faces, which include poor standard of verification and low levels of reuse. For a long time, technical solutions to such technical problems were focus of the eResearch community, including research software distributions, grids, middleware and workflows. Some limited adoption can be seen today in research communities, but the main problems have not been solved to a sufficient level. Hetherington hypothesized that instead of technical solutions, social innovation is needed: a new role in the academic system focused on research software that combines the best parts of a craftsperson and a scholar [12, 13]. However, social innovation in centuries-old institutions is hard. Alternatives to such a new role would have to include rewards for good research software, recognition of software as academic output, and rejecting submissions based on irreproducible computational results. Some advantages of research software engineering (RSE) groups include the possibility of training in reproducible computational research, providing collaborations for researchers who do not want to serve as programmers, and creating synergies with research computing platforms. The success of such a group would be measured by the output and quality of the research software. Members could be part of Research Computing or faculty. They would not be independent researchers, but would have to have a research background. An attempt to form a community of RSE groups within the UK has been underway for several years, including funding from the UK Research Council. An open question is whether this approach can be adopted in other countries, including the USA.
  9. Dan Gunter, Sarah Poon, and Lavanya Ramakrishnan: Bringing the User into Building Sustainable Software for Science. Dan Gunter’s main question was, “What is needed to develop sustainable software?”. Beyond the usual suspects of funding, proficient developers, good design, and software engineering practices, Dan placed the users. He explained this using a traditional software-development model starting from gathering requirements, and reaching release through a design, development, and testing phase. The main deficiency with this approach was pointed out to be the too-late interaction with users. Instead, an alternative approach was proposed that, at first, skips the development phase and repeatedly goes through requirement gathering, design and user interaction/learning phases, and only eventually starts development once an agreement is reached, leading to an increased user satisfaction, higher adoption, and eventually to sustained software.
  10. Dan Gunter, Adam Arkin, Rick Stevens, Robert Cottingham, Sergei Maslov, and the KBase Project: Challenges of a Sustainable Software Platform for Predictive Biology: Lessons Learned on the KBase Project. Dan Gunter presented experiences and lessons learned as part of KBase14, an open software and data platform for addressing the grand challenge of systems biology: predicting and designing biological function. KBase is a unified system that integrates data and analytical tools for comparative functional genomics of microbes, plants, and their communities. However, it is also a collaborative environment for sharing methods and results, and placing those results in the context of knowledge in the field. Being a large, multi-institutional project, one of the big challenges is to agree on standards to enable a single, maintainable system. Working in isolation does not work (anymore) within this field, and the community in the field also does not have standards for software engineering. This is contrasted to computer science research, where software engineering standards shorten design cycles, leading to more time for highly rewarded activities like publishing, performance studies, graduating students, or protecting ideas before publication. Instead of this more traditional approach, KBase uses a variation on the “Scrum” methodology. After picking projects and team members, four to five teams work on projects for about two weeks before a one-week evaluation by the executive committee. Based on their assessment new teams and projects might be chosen before iteratively restarting the “agile” development cycle. This process is intentionally open and documented.
  11. Yolanda Gil, Chris Duffy, Chris Mattmann, Erin Robinson, and Karan Venayagamoorthy: The Geoscience Paper of the Future Initiative: Training Scientists in Best Practices of Software Sharing. Erin Robinson presented an approach to overcome some hurdles in scientific publishing: disconnects among experimental data, research software, and publications. A current effort in the Geosciences is the “Geoscience Paper of the Future15,” which includes four elements [14]. First, it forms a modern paper including text, data, and pointers to supplementary materials. Second, it is reproducible, including data processing, workflow, and visualization tools. Third, it is part of open science, which includes being publicly available under open licenses, and providing meta-data. Last but not least, it uses digital scholarship elements like persistent identifiers for software, data, and authors, and it cites both data and software. OntoSoft is a tool for helping with this effort, providing software stewardship for the Geosciences. As part of this initiative, a special issue of a journal in Geoscience areas is planned to include only Geoscience papers of the future, with submissions open until the end of 2015 [15]. In addition, training sessions are provided to geoscientists to learn best practices in software and data sharing, provenance documentation, and scholarly publication.
  12. Neil Chue Hong: Building a Scientific Software Accreditation Framework. Neil Chue Hong presented a proposal to build a scientific software accreditation framework. One of the aims of such a framework would be to measure how “good” a given piece of software is, and to evaluate how this can be effectively measured in the first place. This can be compared to the effort of the standardized, easy to read, and understandable nutritional labeling of food, which only contain a small set of categories. However, such a framework for software would be more difficult due to different existing community norms. The challenges such a framework faces include that many measurements are subjective, that many metrics are too costly, and that self-assessment needs to be encouraged. Possible categories would include availability, usability, transferability, among others. Such a framework could enable both improvement of specific software, as well as comparisons of similar software. An accreditation by such a framework could then be part of software management plans, ensuring that software is accessible and reusable throughout the proposed project and beyond.
  13. Jeffrey Carver: On the Need for Software Engineering Support for Sustainable Scientific Software. Carver argued that for scientific software to be truly sustainable, there is a need for developers to use appropriate software engineering practices. His experience interacting with scientific teams indicates that choosing and tailoring these practices is not a trivial exercise [16, 17, 18, 19]. There is a general culture clash between software engineering and science that hinders our ability to communicate and choose appropriate methods. In addition, many experienced scientific software developers appear to be unaware of software engineering practices that may be beneficial to them. The most appropriate software engineering practices are those that are lightweight, properly tailored, and focus on the key software development problems faced by scientists. In order to increase the use of software engineering in science, we need more documented success stories. These successes need to be socialized within the scientific community through workshops like the Software Engineering for Science workshop series16 and the new Software Engineering track in Computing in Science and Engineering magazine.
  14. Matthias Bussonnier: User Data Collection in Open Source. This talk highlighted an attempt to solve the common problem for open source development: it is difficult to collect information about how many people use particular software, how often, which version, which parts of the software, or on which operating system. Current solutions include surveys, but these have high uncertainties. A different approach is based on automatic “call-backs” that collect these information at runtime and send it to a central place for analysis. Problems with this approach include obtaining agreement from the user, legal issues, increased maintenance (of servers), ethical questions, and also the lack of a common infrastructure. Some of these problems are of a social nature and have to be solved as such, but the last problem (a missing common infrastructure) is attacked by the sempervirens project [20], which is developing common APIs and a library implementation for common, repeating tasks such as obtaining user consent. Results are uploaded not directly to project servers, but to neutral third parties that only publish aggregated statistics to projects.
  15. Alice Allen: We’re giving away the store! (Merchandise not included). Alice Allen described the Astrophysics Source Code Library17, an increasingly used way to obtain a unique ID for astrophysics software that is indexed by indexing services and can be cited [21]. ASCL offers clones of existing infrastructure, provides server space and computing resources, shares innovations, and permits moves elsewhere. Users provide a domain name, then control and configure their site and use it as intended, gather their codes as they wish, share innovations, and protect the provided computing environment.
  16. Stan Ahalt, Bruce Berriman, Maxine Brown, Jeffrey Carver, Neil Chue Hong, Allison Fish, Ray Idaszak, Greg Newman, Dhabaleswar Panda, Abani Patra, Elbridge Gerry Puckett, Chris Roland, Douglas Thain, Selcuk Uluagac, and Bo Zhang: Scientific Software Success: Developing Metrics While Developing Community. The effort behind this talk given by Ray Idaszak started from a breakout group at an NSF SI2 workshop in 2015, and centers around building a framework for creating metrics for scientific software [22]. This framework would improve both the metrics and the software it evaluates, and could also serve as a tool for building a community around the idea. With especially the last-mentioned idea (building a community) in mind, a software “peer review group” would be created, representing stakeholders who will self-review software created by their respective communities, and will concurrently develop metrics. The whole project should be community-governed, without a single institution overseeing the activities or infrastructure, with the hope to evolving community-generated and adopted standards. The generation of metrics would be tied to the actual evaluation of software, creating an incentive by improving the evaluated software itself during this process. The framework code would provide infrastructure for the creation of metrics and evaluation, and forums for generation of software success metrics. It would also support code reviews of the evaluated software. An open question is whether it is possible to fit the resulting metrics in a common template. So far, this is still in a design phase, with a white paper at the 2015 CSESSP workshop and this talk, but the WSSSPE workshops are seen as a forum for the community to assemble and act, and is planned to be used also in the future to build this community and framework.

(5) Working Groups: Summaries

After the keynote and lightning talks, the workshop facilitators led an exercise to create working groups for the rest of the meeting. The attendees first suggested additional topics beyond those in the call for participation (as listed in §2). One topic that was introduced at this point was “Building Sustainable User Communities for Scientific Software.” The full set of topics was then placed on flipchart-sized pieces of paper around the walls, and the attendees voted on which topics they were strongly interested in working on at WSSSPE3, and which topics they were generally interested in contributing to, but not as strongly. This led to some topics being taken out of the mix for the rest of the workshop, since not enough people wanted to contribute to them to lead to a useful discussion. Additionally, some topics were combined by the participants who felt they were closely linked and that a group could address multiple of them in a single discussion.

After this, the attendees broke up into small working groups to discuss the remaining topics during most of the remaining 1 1/2 days. A high-level summary of each topic and group’s work can be found in the subsections of this section, and for all but one group, more detailed notes on each group’s discussions can be found in §6.

Midway through the first afternoon and between the two days, each group had a chance to talk about what progress they had made. As discussed below (in §5.4), the group that formed to discuss Legacy Software dissolved after the first session as group members left to join other groups. In addition, in the morning of the second day, a small set of reviewers/advisors (external to the groups but chosen by the group members) visited each group to listen to what they group was planning and to provide feedback.

5.1 White paper/journal paper about best practices in developing sustainable software

Reviewing multiple past articles and talks at different meetings like WSSSPEx [2, 4, 23, 24, 25] and analyzing and promoting sustainable scientific software makes it clear that there are several common and recurring ideas that underpin success in developing sustainable software. However, outside of a small community, this knowledge is not widely shared. This is especially true for the large community of scientists who generate most of the software used by scientists but are not primarily software developers. In this scenario, a clear and precise exposition of these best practices collected from many sources and open collaboration among all in the community in a single source (e.g., journal paper, tutorial) that can be widely disseminated is necessary and likely to be very valuable.

5.1.1 Fit with related activities

The creation of such a “best practices” document will build upon the range of activities and topics discussed at WSSSPE3 and associated prior meetings. This working group will attempt to distill the emerging body of knowledge into this document. The large number of articles from the NSF-funded SI2 projects (SSE and SSI), “lightning talks,” “white papers,” and reports from different workshops have created a large if somewhat diffuse source for this report.

5.1.2 Discussion

Core questions that will need to be explored are in reproducibility, reliability, usability, extensibility, knowledge management, as well as continuity (transitions between people). Answers to these questions will guide the group to learn how a software tool becomes part of the core workflow of well-identified users (stakeholders) relating to tool success and hence sustainability. Ideas that may need to be explored include:

  • Requirements engineering to create tools with immediate uptake;
  • When should software “die”?
  • Catering to disruptive developments in environment (e.g., new hardware, new methodology);
  • Dimensions of sustainability: economic, technical, environmental and obsolescence.

Sustainability requires community participation in code development and/or a wide adoption of software. The larger the community base is using a piece of software, the better are the funding possibilities and thus also the sustainability options. Additionally developer commitment to an application is essential and experience shows that software packages with an evangelist imposing strong inspiration and discipline are more likely to achieve sustainability. While a single person can push sustainability to a certain level, open source software also needs sustained commitment from the developer community. Such sustained commitments include diverse tasks and roles, which can be fulfilled by diverse developers with different knowledge levels. Besides developing software and appropriate software management with measures for extensibility and scalability of the software, active (expertise) support for users via a user forum with a quick turnaround is crucial. The barrier to entry for the community as users as well as developers has to be as low as possible.

For additional information about the discussion, see Section 6.1.

5.1.3 Plans

The creation of a document on best practices needs a large and diverse community involved. The group has enlisted over ten contributors from the attendees at the WSSSPE3 and those on the mailing list. The primary mechanism for developing this document will be to examine and analyze the success of several well known community scientific software and organizations supporting scientific software. The group will attempt then to abstract general principles and best practices. Some of the tools identified for such analysis are the general purpose PETSc toolkit for linear system solution, NWChem for computational chemistry and the CIG (Computational Infrastructure for Geodynamics) organization dedicated to supporting an ensemble of related tools for the geodynamics community. The group also established a timeline and a rough outline (see Section 6.1) for the report.


  • 28 Dec: Introduction and scope finished
  • 06 Jan: Sections assigned
  • 31 Jan: Analyzing funding possibilities for survey
  • 31 Jan: First versions of section
  • 15 Feb: Distribution to WSSSPE community
  • 31 Mar: Final version of white paper
  • 30 Apr: Submission of peer-reviewed paper?

5.1.4 Landing Page

The landing page with instructions, timeline and the white paper is here: Discussions can be also continued in

5.2 Funding Research Programmer Expertise

Research Software Engineers (RSEs)—those who contribute to science and scholarship through software development—are an important part of the team needed to deliver 21st century research. However, existing academic structures and systems of funding do not effectively fund and sustain these skills. The resulting high levels of turnover and inappropriate incentives are significant contributing factors to low levels of reliability and readability observed in scientific software. Moreover, the absence of skilled and experienced developers retards progress in key projects, and at times causes important projects to fail completely.

Effective development of software for advanced research requires that researchers work closely with scientific software developers who understand the research domain sufficiently to build meaningful software at a reasonable pace. This requires a collaborative approach—where developers who are fully engaged or invested in the research context are co-developing software with domain academics.

5.2.1 Fit with related activities

The solution this group envisions entails creating an environment where software developers are a stable part of a research team. Such an environment mitigates the risk of losing a key developer at a critical moment in a projects lifetime, and provides the benefits of building a store of institutional knowledge about specific projects as well as about software development for today’s research. The group’s vision is to find a way to promote a university/research institute environment where software developers are stable components of research project teams.

One strategy to promote stability is implementing a mechanism for developers to obtain academic credit for software development work (see §5.8.) With such a mechanism in place, traditional academic funding models and career tracks could properly sustain individuals for whom software development is their primary contribution to research. A contributing factor to the problem with the current academic reward system is the devastating effect on an academic publication record resulting from time in industry; such postings often develop exactly the skills that research software engineers need, yet returns to university positions following an industry role are penalized by the current structures. Retention of senior developers is hard, because these people are high in demand by the economy. However, people who have a PhD in science and enter industry, may desire to return for diverse reasons, and should be welcomed back.

While developing new mechanisms in the current academic reward system is a worthy aspirational goal, such a dramatic change in this structure does not seem likely in a time scale relevant to this working group. Accordingly, the working group sought alternative solutions that may be achievable within the context of existing academic structures. The group felt that developing dedicated research software engineering roles within the university and finding stable funding for those individuals is the most promising mechanism for creating a stable software development staff.

Measures of impact and success for research programming groups, as well as for individual research software engineers, will be required in order to make the case to the university for continued funding. Research software engineers will hopefully not be measured by publications, but by other metrics. Middle-author publications are common for RSEs. Most RSEs welcome co-authorship on papers when the PIs think that the contribution deserves it.

5.2.2 Discussion

It is hard for an individual PI in a university or college to support dedicated research software engineering resources, as the need and funding for these activities are intermittent within a research cycle. To sustain this capacity, therefore, it is necessary to aggregate this work across multiple research groups.

One solution is to fund dedicated software engineering roles for major research software projects at national laboratories or other non-educational institutions. This solution is in place and working well for many well-used scientific codebases. However, this strategy has limited application, as much of the body of software is created and maintained in research universities. Therefore, the group argues that research institutions should develop hybrid academic-technical tracks for this capacity, where employees in this track work with more than one PI, rather than the traditional RA role within a single group. This could be coordinated centrally, as a core facility, perhaps within research computing organizations which have traditionally supported university cyberinfrastructure, library organizations, or research offices. Alternatively, these groups could be organizationally closer to research groups, sitting within academic departments. The most effective model will vary from institution to institution, but the mandate and ways of working should be similar.

Having convinced themselves that this would be a positive innovation, the group members were then faced with the specific question of how to fund the initiation of this activity. A self-sustaining research software group will support itself through collaborations with PIs in the normal grant process, with PIs choosing to fund some amount of research software engineering effort through grants in the usual way. However, to bootstrap such a function to a level where it has sufficient reputation and client base to be self-sustaining will generally require seed investment.

This might come from universities themselves (this was the model that led to the creation of the group in University College London), but more likely, seed funding needs to come from research councils or other funding bodies (as with the Research Software Engineering Fellowship provided by the UK Engineering and Physical Sciences Research Council). The group therefore recommends that funding organizations consider how they might provide such seed funding.

Success, appropriately measured, will help make the case to such funding bodies for further investment. One might expect that metrics such as improved productivity, software adoption rates, and grant success rates would be sufficient arguments in favor of such a model. However, useful measurement of code cleanliness, and the resulting productivity gains, is an unsolved problem in empirical software engineering. To measure “what did not go wrong” because of an intervention is particularly hard.

The working group finally noted that the institutional case for such groups is made easier by having successful examples to point to. In the UK, a collective effort to identify the research software engineering community, with individuals clearly stating “I am a research software engineer,” has been important to the campaign. It will be useful to the global effort to similarly identify emerging research software organizations, and also, importantly, to identify longer-running research software groups, which have in some cases had a long running sui-generis existence, but which now can be identified as part of a wider solution. There remains the problem of how to “sell” the value of this investment to investigators within a university. This is an issue best addressed by the individual organizations that embark on the plan.

For more details on the discussion, see Section 6.2.

5.2.3 Plans

The first step in moving this strategy forward is to gather a list of groups that selfidentify as research software engineering groups, and to reach out to other organizations to see if there may be a widespread community of RSEs who do not identify themselves as such at this time. This working group will collect information about the organizational models under which these groups function, and how they are funded. For example, how many research universities currently fund people in the RSE track, whether they bear the RSE moniker or not. Are these developers paid by the university or through a program supported by research grants/individual PIs? How did they bootstrap the developer track to get this started? How successful is the university in getting investigators to pay for fractional RSEs? The group will author a report describing their findings, should funding be available to conduct the investigation.

5.2.4 Landing Page

A list of known UK research software engineering groups is available at, and a list for the rest of the world is at To add another group to the list, please make a pull request as requested on either of these pages.

5.3 Transition Pathways to Sustainable Software: Industry & Academic Collaboration

Most scientific software is produced as a part of grant-funded research projects typically sponsored by federal governments. If we are interested in the sustainability of scientific software, then we need to understand what exactly happens when that sponsorship ends. More than likely, the project and its resulting software will need to undergo some kind of transition in funding and consequently governance.

At WSSSPE3, this working group was interested in better understanding successful pathways for scientific software to “transition” from grant-funded research projects to industry sponsorship. (This may be an initially awkward phrase—some software projects will begin their life being sponsored by industry, or result in collaboration between industry and academia. In such cases, there is still a need to understand how IP and how maintenance of the software is sustained over time.)

5.3.1 Fit with related activities

Most previous research and discussion of industry and academic collaboration, sharing, and funding of research software has focused on the impact of such arrangements. Examples of these types of reports are:

5.3.2 Discussion

Although sustainability transitions are often studied under the broad umbrella of “technology transfer,” the group believes there are likely to be a number of different ways in which a pathway from initial production to long-term maintenance and secure funding is achieved. In short, industry sponsorship and/or direct participation is an important aspect of sustaining scientific software, but our current understanding of these transitions focuses narrowly on commercial successes or failures of those collaborations.

In looking at existing literature that addresses industry transitions, many reports (such as those listed above) focus on benefits that accrue to the private sector, or to a government that originally sponsored the research project. This literature does not address the impact that these transitions have on the accessibility or usability of the software, or the impact that these transitions have on the career of the researchers involved.

For more detail on the group’s discussion, see Section 6.3.

5.3.3 Plans

Plans for carrying forward are currently unclear—this project would require sustained attention and effort from the group members, and at least some amount of funding in order for those members to be involved for extended periods of time.

The broad goals that the group would like to accomplish are:

  1. To complete a set of case studies which look at successful and unsuccessful transitions between academic researchers and industry
  2. To create a generalizable framework, which might allow for a broader study of different transition pathways (other than between academia and industry)

The main plan for the group going forward is the creation of a white paper on the topic of sustainability transitions.

5.3.4 Landing Page

Transitions Pathways discussions can be posted at or an email be sent to Nic Weber18 to find out more about the group’s efforts and how to participate.

5.4 Legacy Software

This group met only briefly, for one period on the first day. They discussed that it is difficult to define legacy code because there is so much stigma associated with the term. At some point there will be more difficulty and resources wasted trying to keep legacy software supported, but it will eventually be too expensive compared to how much it would be to just rebuild the software or kill it. Most of the group members were not able to attend on the second day, and those who were able to attend joined other groups.

5.5 Principles for Software Engineering Design for Sustainable Software

Principles for software engineering form the basis of methods, techniques, methodologies and tools [26]. However, there is often a mismatch between software engineering theory and practice particularly in the fields of computational science and engineering, which can lead to the development of unsustainable software [27, 28]. Understanding and applying software engineering principles is essential in order to create and maintain sustainable software [29].

5.5.1 Fit with related activities

This group’s discussion focused on identifying existing principles of software engineering design that could be adopted by the computational science and engineering communities.

5.5.2 Discussion

Software engineering principles form the foundation of methods, techniques, methodologies, and tools. Consisting of members from different backgrounds, including quantum chemistry, epidemiology, computer science, software engineering, and microscopy, this group discussed the principles of software engineering design for sustainable software (starting with principles from the Karlskrona Manifesto on Sustainability Design [30], Tate [31], and the Software Engineering Body of Knowledge (SWEBOK) [32]) and their application in various domains including quantum chemistry and epidemiology. The group examined the principles and took a retrospective analysis of what the developers did in practice against how the principles could have made a difference, and asked, what do the principles mean for computational scientific and engineering software, and how do the principles relate to non-functional requirements? It appeared that the sustainable software engineering principles should be mapped to two core quality attributes that underpin technically sustainable software: extensibility, the software’s ability to be extended and the effort level required to implement the extension; and maintainability: the effort required to locate and fix an error in operational software.

For more information about the discussion, see Section 6.4.

5.5.3 Plans

The next steps in this endeavor are to (1) Systematically analyze a number of example systems from different scientific domains with regards to the identified principles, to (2) Identify the commonalities and gaps in applying those principles to different scientific systems, and to (3) Propose a set of guidelines on the principles and examine how they exemplarily apply to scientific software systems. Preliminary work will be carried out through undergraduate or post-graduate student projects.

5.5.4 Landing Page

In the absence of a landing page, the Principles for Software Engineering Design for Sustainable Software working group requests an email be sent to Birgit Penzenstadler19 and Colin C. Venters20 to find out more about the group’s efforts and how to participate.

5.6 Useful Metrics for Scientific Software

Metrics for scientific software are important for many purposes, including tenure and promotion, scientific impact, discovery, reducing duplication, serving as a basis for potential industrial interest in adopting software, prioritizing development and support towards strategic objectives, and making a case for new or continued funding. However, there is no commonly-used standard for collecting or presenting metrics, nor is it known if there is a common set of metrics for scientific software. It is imperative that scientific software stakeholders understand that it is useful to collect metrics.

5.6.1 Fit with related activities

The group discussion focused on identifying existing frameworks and activities for scientific software metrics. The group identified the following related activities:

  • Computational Infrastructure for Geodynamics: Software Development Best Practices21
  • WSSSPE3 Breakout Session: How can we measure the impact of a piece of code on research, and its value to the community?22
  • 2015 NSF SI2 PI Workshop Breakout Session on Framing Success Metrics23
  • 2015 NSF SI2 PI Workshop Breakout Session on Software Metrics24
  • NSF Workshop on Software and Data Citation Breakout Group on Useful Metrics25
  • U.K. Software Sustainability Institute Software Evaluation Guide26
  • U.K. Software Sustainability Institute Blog post: The five stars of research software27
  • Minimal information for reusable scientific software28
  • EPSRC-funded Equipment Data Search Site29
  • Canarie Research Software: Software to accelerate discovery30
  • Canarie Research Software: Research Software Platform Registry31
  • BlackDuck Open HUB32
  • Innovation Policy Platform33

5.6.2 Discussion

The group discussion began by agreeing on the common purpose of creating a set of guidance giving examples of specific metrics for the success of scientific software in use, why they were chosen, what they are useful to measure, and any challenges and pitfalls; then publish this as a white paper. The group discussed many questions related to useful metrics for scientific software including addressing if there is a common set of metrics that can be filtered in some way, can metrics be fit into a common template, which metrics would be the most useful for each stakeholder, which metrics are the most helpful and how would we assess this, how are metrics monitored, and many more. A more complete bulleted list of these questions can be found in Section 6.5. Next, a roadmap for how to proceed was discussed, including creating a set of milestones and tasks. The idea was put forth for the group to interact with the organizing committee of the 2016 NSF Software Infrastructure for Sustained Innovation (SI2) PI workshop in order to send a software metrics survey to all SI2 and related awardees as a targeted and relevant set of stakeholders. The five solicitations for software elements released under the NSF SI2 program all included metrics as a required component with submitters requested to include “a list of tangible metrics, with end user involvement, to be used to measure the success of the software element developed, … ”. These metrics are then reported as part of annual reports to NSF by the projects. Although neither the proposal text describing the metrics nor the reported metric results are publicly available, there is reason to believe that the community will be willing to provide this information through a survey mechanism. This survey would be created by one of the student group members. Similarly, it was suggested that a software metrics survey be sent to the UK SFTF (Software For The Future, led by the Engineering and Physical Sciences Research Council) and TRDF (Tools and Resources Development Fund, led by the Biotechnology and Biological Sciences Research Council) software projects to ask them what metrics would be useful to report. The remainder of the discussion focused mainly on the creation of a white paper on this topic. This resulted in a paper outline and writing assignments with the goal of publishing in venues including WSSSPE4, IEEE CiSE (Institute of Electrical and Electronic Engineers Computing in Science and Engineering magazine), or JORS (Journal of Open Research Software). More information about the group discussion is available in Section 6.5.

5.6.3 Plans

The main plan for the group going forward is the creation of a white paper on the topic of useful metrics for scientific software. The authoring of this white paper would happen in parallel with the creation of a survey by the group with the survey results to be incorporated in the white paper. The timeline for completion of the white paper is approximately one year targeting venues discussed in the previous section.

5.6.4 Landing Page

In lieu of a landing page, the Useful Metrics for Scientific Software working group requests an email be sent to Gabrielle Allen34 to find out more about the group’s efforts and how to participate.

5.7 Training

This group explored a rapidly growing array of training that is seen to contribute to sustainable software. The offerings are diverse, providing training that is more or less directly relevant to sustainable software. While research institutions support professional development for research staff, the skills taught which might impact on sustainable software are limited at best, often lacking a clear and coherent development pathway. Bringing together those involved in leading relevant initiatives on a regular basis could helpfully coordinate this growing array of training opportunities.

5.7.1 Fit with related activities

Three existing venues for discussion of related events are identified:

  • Working towards Sustainable Software for Science: Practice and Experiences (WSSSPE) workshops [33]
  • International Workshop on Software Engineering for High Performance Computing in Computational Science and Engineering (SEHPCCSE) [34]
  • Workshop on Software Engineering for Sustainable Systems [35]

5.7.2 Discussion

Some next steps were identified to quickly test whether there is interest in establishing a community committed to increasing the degree of coordination across training projects. See Section 6.6 for more details about the discussion.

5.7.3 Plans

The main plan for the group is to convene a discussion to explore bringing together regular meetings of those involved in leading relevant training projects.

5.7.4 Landing Page

The Training working group requests an email be sent to Nick Jones35 to find out more about the group’s efforts and how to participate.

5.8 Software Credit Working Group

Modern scientific and engineering research often relies considerably on software, but currently no standard mechanism exists for citing software or receiving credit for developing software akin to receiving credit via citations for writing papers. Ensuring that developers of such scientific software receive credit for their efforts will encourage additional creation and maintenance. Standardizing software citations offers one route to establishing such a citation and credit mechanism. Software is currently eligible for DOI assignment, but DOI metadata fields are not well tuned for software compared to publications. Some software providers apply for DOIs but it is still not widely adopted. Also, there is no mechanism to cite software dependencies within software in the same way papers cite supporting prior work.

5.8.1 Fit with related activities

Publishing Software Working Group (§5.9): publishing a software paper offers one existing mechanism for receiving credit, and further developing new publishing concepts for software will strengthen our activities.

A number of groups external to WSSSPE (although with some overlapping members) are also focused on aspects of software credit, including the FORCE11 Software Citation Working Group (see plans for coordination below). In addition, a Software Credit workshop36 convened in London on October 19, following the conclusion of WSSSPE3. See Section 6.7 for more detailed discussion of related activities.

5.8.2 Discussion

The group discussed a number of topics related to software credit, including a contributorship taxonomy, software citation metadata, standards for citing software in publications, and increasing the value of software in academic promotion and tenure reviews. Although initial discussions both prior to and during WSSSPE3 focused on contribution taxonomy and dividing credit, discussing as an example the Entertainment Identifier Registry [36] used in the entertainment industry, the group decided to prioritize software citation. This decision was motivated by the idea that standardizing citations for software would introduce some initial credit for developers, and later the quantification of credit could be refined based on concepts such as transitive credit [37, 38].

The majority of the remaining discussion focused on standardizing (1) the metadata necessary for software to be cited and (2) the mechanism for citing software in publications. Moreover, discussions also oriented around the indexing of software citations necessary for establishing a software citation network either integrated with the existing paper citation ecosystem or complementary to it. See Section 6.7 for a more detailed summary of the working group’s discussion on these topics.

5.8.3 Plans

The group already merged with the FORCE11 Software Citation Working Group (SCWG), and their efforts will focus (over the next six to nine months) on developing a document describing principles for software citation. Following the publication of that document, the group will focus on outreach to key groups (e.g., journals, publishers, indexers, professional societies). Longer-term plans include working with indexers to ensure that software citations are indexed and pursuing an open/community indexer; these activities may be organized by future FORCE11 working groups.

5.8.4 Landing Page

Since near-term efforts will be shifting to the FORCE11-SCWG, interested readers should go to that group’s existing landing page37 and GitHub repository38.

5.9 Publishing Software Working Group Discussion

This working group explored the value of executable papers (papers whose content includes the code needed to produce their own results), and other forms of publishing which include dynamic electronic content. Transitioning to this type of publication offers possibilities of addressing, or partially addressing, sustainability concerns such as reproducibility, software credit, and best practices.

5.9.1 Fit with related activities

  • Reproducibility: Part of the purpose of these executable paper venues is to (at least partially) address the reproducibility issue by making papers recompute their own results.
  • Software Credit (§5.8): Since these forms of publishing must make their sources explicit in order to execute, they should be easier to trace even if appropriately worded credit for software is not provided. In addition, they make it possible to provide or define additional metadata to make the tracing of credit clearer. Finally, attributions could be added to citations to identify whether a paper extends a result, verifies it, contradicts it, etc.
  • Best Practices (§5.1): Because an executable paper showcases the code, and the code itself is subject to the review process, authors are more likely to pay attention to coding practices. In addition, because the paper must explain what the code does, better documentation is more likely to be achieved.

5.9.2 Discussion

The group felt that the best way to encourage the use of these new publishing concepts would be to create and curate a list of publishing venues that support them. The Software Sustainability Institute agreed to host this list.

See Section 6.8 for more details about the discussion.

5.9.3 Plans

The plan is to create and curate a web page describing executable papers, their value, and a list of what publishers support them. The group expects the page to be available in early January of 2016 on the Software Sustainability Institute’s website.

5.9.4 Landing Page

The aforementioned page will be published on the Software Sustainability Institute website:

5.10 Building Sustainable User Communities for Scientific Software

User communities are the lifeblood of sustainable scientific software. The user community includes the developers, both internal and external, of the software; direct users of the software; other software projects that depend on the software; and any other groups that create or consume data that is specific to the software. Together these groups provide both the reason for sustaining the software and, collectively, the requirements that drive its continued evolution and improvement.

5.10.1 Fit with related activities

There are a number of activities already in progress that are targeted at improving the user community for open-source software, including Mozilla Science’s “Working Open Project Guide” [39] and “UK Collaborative Computational Projects” (CCP)39, or books such as “Art of Community” by Jono Bacon [40].

5.10.2 Discussion

Discussion revolved around a few questions: what are the benefits of having a “community” for software sustainability; what practices and circumstances may lead to having and maintaining a community; how can funding help or hinder this process; and perhaps most importantly, how can best practices be described and distilled into a document that can help new projects.

All the group members agreed on a few points: software must not only offer value, but there must be some support for users; and funding can help pay for that support, in addition to the usual funding for software development. Openness is generally a virtue. An evangelist, either in the form of a single person or some domain-specific group of users, is often the key factor.

Additional details on the group’s discussion can be found in Section 6.9.

5.10.3 Plans

The most important next steps is a “Best Practice” document, which would describe what successful projects with engaged communities look like, how to replicate this type of project, and look at the end of life of a community project. Another next step would be better training to increase recognition of need for science software projects to focus on building and supporting their user communities.

5.10.4 Landing Page

This group does not have a landing page yet. Please send requests to join and contribute by writing to both Dan Gunter40 and Ethan Davis41.

(6) Working Groups: Details

This section captures detailed reports from each working group that made significant progress. Each subsection records the discussion of a group, as written by that group at that time (in the first person and in the present/future tense.) Thus, the subsections are records of what the groups did and planned, as of the end of the WSSSPE3 workshop.

6.1 Best Practices Group Discussion

Sandra Gesing42 will serve as the point of contact for this working group, and be responsible for ensuring timely progress of the planned actions.

6.1.1 Group Members

  • Abani Patra – University at Buffalo
  • Sandra Gesing – University of Notre Dame
  • Neil Chue Hong – Software Sustainability Institute
  • Gregory Tucker – University of Colorado at Boulder
  • Birgit Penzenstadler – California State University Long Beach
  • Abigail Cabunoc Mayes – Mozilla Foundation
  • Frank Löffler – Louisiana State University
  • Colin C. Venters – University of Huddersfield
  • Lorraine Hwang – UC Davis
  • Sou-Cheng Choi – NORC at the University of Chicago & Illinois Institute of Technology
  • Suresh Marru – Indiana University
  • Don Middleton – NCAR
  • Daniel S. Katz – University of Chicago & Argonne National Laboratory
  • Kyle Niemeyer – Oregon State University
  • Jeffrey Carver – University of Alabama
  • Dan Gunter – LBNL
  • Alexander Konovalov – University of St Andrews
  • Tom Crick – Cardiff Metropolitan University

6.1.2 Summary of Discussion

Core questions that will need to be explored are in reliability, reproducibility, usability, extensibility, knowledge management, and continuity (transitions between people). Answers to these will guide us on how a software tool becomes part of the core workflow of well identified users (stakeholders) relating to tool success and hence sustainability. Ideas that may need to be explored include:

  • Requirements engineering to create tools with immediate uptake;
  • When should software “die”?
  • Catering to disruptive developments in environment, e.g., new hardware, new methodology;
  • Dimensions of sustainability: economic, technical, environmental, and obsolescence.

Sustainability requires community participation in code development and/or a wide adoption of software. The larger the community base is using a piece of software, the better are the funding possibilities and thus also the sustainability options. Additionally, the developers’ commitment to an application is essential and experience shows that software packages with an evangelist imposing strong inspiration and discipline are more likely to achieve sustainability. While a single person can push sustainability to a certain level, open source software also needs sustained commitment from the developer community. Such sustained commitments include diverse tasks and roles, which can be fulfilled by diverse developers with different knowledge levels. Besides developing software and appropriate software management with measures for extensibility and scalability of the software, active (expertise) support for users via a user forum with a quick turnaround is crucial. The barrier to entry for the community as users as well as developers has to be as low as possible.

6.1.3 Description of Opportunity, Challenges, and Obstacles

There is an opportunity to collaborate on a white paper, which will be revisited regularly for further improvements, to enhance knowledge of the state of best practices, resulting in a peer-reviewed paper. We would like to reach a wide community by doing this. But these are also the challenges and obstacles – to get everyone to contribute to the paper and to reach the community.

White Paper Outline

  1. Introduction and Scope of White Paper
  2. Related Work
  3. Case Studies
    1. PETSc
    2. NWChem
    3. CIG
  4. Community Related Practices
    1. Findings
    2. Recommendations
  5. Governance and management
    1. Findings
    2. Recommendations
  6. Funding Related
    1. Findings
    2. Recommendations
  7. Metrics for sustainability
  8. Tools
  9. Conclusions

6.1.4 Key Next Steps

The key next steps are to write an introduction, reach out to the co-authors, and to agree on the scope of the white paper.

6.1.5 Plan for Future Organization

Sandra Gesing and Abani Patra are the main editors and will organize the overall communication and the paper. Sections will be assigned to diverse co-authors.

6.1.6 What Else is Needed?

At the moment we do not see any further requirements.

6.1.7 Key Milestones and Responsible Parties

  • 28 Dec: Introduction and scope finished (Abani Patra/Sandra Gesing)
  • 06 Jan: Sections assigned (Abani Patra/Sandra Gesing)
  • 31 Jan: Analyzing funding possibilities for survey
  • 31 Jan: First version of each section
  • 15 Feb: Distribution to the WSSSPE community
  • 31 Mar: Final version of the white paper
  • 30 Apr: Submission to a peer-reviewed journal?

6.1.8 Description of Funding Needed

We might need funding for a journal publication (open-access options).

6.2 Funding Research Programmer Expertise Group Discussion

James Hetherington43 will serve as the point of contact for this working group, and be responsible for ensuring timely progress of the planned actions.

6.2.1 Group Members

The group at WSSSPE:

  • Don Middleton – National Center for Atmospheric Research
  • Joshua Greenberg – Alfred P. Sloan Foundation
  • James Hetherington – University College London
  • Lindsay Powers – The HDF Group
  • Mark A. Miller – San Diego Supercomputer Center
  • Dan Sellars – CANARIE

This was further enhanced by additional discussions at the following GCE15 conference:

  • Lorraine Hwang – UC Davis
  • Simon Trigger – BioTeam, Inc.
  • Nancy Wilkins-Diehr – San Diego Supercomputer Center
  • Alexander Vyushkov – University of Notre Dame
  • Sandra Gesing – University of Notre Dame
  • Ali Swanson – University of Oxford

6.2.2 Summary of Discussion

In addition to the points noted in the main discussion (§5.2), we also discussed the following:

“Are you an RSE or a RA?” is not properly a binary question. Most of us sit at different points on that spectrum, and move along it during our careers (usually from RA to RSE—examples of movement in the other direction from readers would be welcomed). Either way, the label “Research Software Engineer” is now starting to have some power. Many scientists do not want to be writing code; some do, to varying degrees. These groups can usefully support each other.

What is the power of the label? How can we get the word out about RSE support using the label?

Will research science developers be required in the long run? One issue that came up was whether the need for developers was a time bounded one; is it the case that the new generation of computer and software savvy scientists will be so comfortable in developing their own code that the professional developers will not be needed? And this brings up the flip side question, “Do scientists really want to be writing code?”

We also had a little discussion about how to make a career path for research developers. It need not be solely an academic enterprise, but in the past tenure has often been problematic for people of this class.

Skills and resources may vary between teams. To help resolve this, maintaining high levels of communication between groups will be valuable. In the United Kingdom (UK), there are plans to permit resource sharing between institutional RSE groups. Perhaps there are circumstances under which an RSE skill exchange could be arranged, either formally or informally.

Collaborative funding can be crucial to RSE groups, to ensure that research leadership remains with the domain scientists. As an example, at NCAR, university partnerships are required for submission of proposals, so collaboration is an essential part of grant submission, and this will tend to bring developers and scientists together. The UCL group also follows this approach, with all bids requiring an academic collaborator.

Domain scientists and developers are funded together in a single proposal. Another example of a success is the development of semantics and linked data in support of ocean sciences. An EarthCubefunded project pairs domain scientists with RSEs and has been successful; the semantics attached have increased data use and discovery significantly.

An alternative approach has been the provision of programming expertise as part of national compute services. The US XSEDE project’s Extended Collaboration Support Services (ECSS) is a set of developers who are paid with XSEDE funding, and are on “permanent” staff. When PIs request allocations on XSEDE resources, there is a finite pool of developer time that can be awarded, typically for one year only, and at partial effort, typically 20 percent or so. The finite time allowed provides motivation for the scientist and the scientist’s group to work closely with the developer and to become educated in what the developer is doing, so they can sustain the effort once the ECSS period is over. This funding mechanism can be highly efficient for scientific problems, because the developer pool assembled by the research providers are, by definition, expert in the characteristics of their specific resource, and can very quickly assess the scientist’s needs, and what it will take to implement software that meets the user’s needs. However, it does not develop capacity within institutions, and since XSEDE is a time-bounded program, it should not be relied upon as a long-term solution to acquiring this type of capacity.

The UK allows this kind of collaboration to support the creation of scientific software for the large supercomputing resource (ARCHER). However, while the support can come from the staff of the Edinburgh Parallel Computing Centre, who hosts the computer, this “embedded CSE” resource also funds the programming coming from local groups. This has been very helpful in providing funding to establish local groups. These groups work best when they develop good collaborations with national cyberinfrastructure pools. When an organization assembles a developer pool, diversity is developed and skills can be transferred.

We would like to see these models applied outside high performance computing. Most scientific software is not destined to run on national cyberinfrastructure, but needs similar support. The argument regarding making better use of expensive hardware through software improvements has been useful politically, (and many RSE groups are cited in organizations which host clusters for this reason), but the time has come to make the case that software itself is a critical cyberinfrastructure, and, with a much longer shelf-life than hardware, is itself a capital investment.

The CANARIE group (Canada) accepts proposals for providing services to broad communities, integrating people who are doing things that are complementary. The goal is to make the available stack more robust and richer for everyone. They offer short cycles of funding for creating some useful functionality that shows a diversity of input and draws from across disciplines as a key metric, If this metric is met successfully, then more funding may follow. This can apply within or across institutions.

There can be problems communicating across cultural barriers, with domain scientists seeing developers as “other”. Both collaboration and tools to fund, encourage, or motivate collaboration are extremely important.

We think support from non-governmental organizations will be important if RSE groups will become established. The Sloan Foundation is currently funding data science engineers, who work in the context of other software developers at the University of Washington. These scientists work in the e-Science Studio/Data Science Studio, and they help a group of graduate students in solving their problems in data science and data management. During Fall and Spring, a 10-week incubator program allows students to work two days a week on a data-intensive science project. Some fraction of the developer time is dedicated to the developers’ personal interests as well as instruction.

The goal for Sloan is to obtain success stories and demonstrable value of the presence of data scientists on university staff. These stories are the basis for arguments to the host organization. This is an effort to create awareness of the value of research scientist developers. Embedding with scientists, and adding spare capacity is critical to making the innovation possible. This model is essentially to argue for permanent budget lines to support data scientists as part of university staff hires, just as with core facilities. This could become a fee-for-service model requested by grant funding, just as DNA sequencing is for core facilities, if it becomes apparent that this gives competitive advantage to a university’s research effort.

One model that has been helpful in finding funding for RSE groups is the use of funds left over on research grants when RAs have left prematurely – PIs like this arrangement as it is hard to find good staff for short-term positions, so having a pool of research programming staff on hand resolves this problem. We recommend that funders give explicit guidance to grant holders and institutions that such an arrangement is favorable. Framework agreements permitting this to go ahead without checking back every time with funders and/or grant panels would further smooth this. (This also provides more stable jobs for those who hold these skills, but arguments about making life nicer for postdocs will not help persuade funders or PIs!)

There is some question about the most effective duration and percentage of full time for a programmer’s work on a project. At least three months is necessary for the programmer to read into the science (RSEs must not become so disengaged from research that they do not have time to read a few papers – this will result in code which does not meet scientific needs), but too long could result in an RSE losing their flexibility, becoming so engaged in one project that when that project ends, they find it hard to transfer. For this reason, we recommend that 40% is ideal; two projects per developer, with some time for training and infrastructure work. Having two developers per project seems to be ideal, in the sense that software development is enhanced by two pairs of eyes.

There is, as yet, no clear answer as to the scale of aggregation needed to make such a program work. A university wide program allows enough scale to be robust to fluctuations of funding within one field. But a specialization focus on developers to support, for example, physical or biological sciences may be preferable, if the customer base is large enough. The desire to aggregate enough work to make it sustainable, and the need to have domain-relevant research programming skills, are in tension.

In the UK, another source of funding for research software is the Collaborative Computational Projects (CCPs): domain specific communities put forward proposals that are a priority of the community as a whole, for example, biosimulation or plasma physics. These bodies act as custodians of community codes, and a central team also provides software engineering support.

However this area develops, the need for funding for software as a cyberinfrastructure component is clear. Funding that permits code to be refactored, tidied, and optimized is rare; this is often done “on the sly” in a scientifically focused grant. The UK EPSRC’s “software for the future” call, which really permits explicit investment in software as an infrastructure, is so oversubscribed as to have a 4% success rate; the demand is clear!

One opportunity is the idea of co-design, where infrastructural libraries are developed alongside the scientific codes that will call them. However, collaboration is hard to foster here; as incentive structures are still focused on short-term papers. This can cause infrastructure developers to focus more on publications in their areas of mathematics and computer science, the domain developers on the shorter-term needs of their own fields. Genuine collaborative co-construction is harder to foster.

It can be more difficult to help leading domain scientists see the value of engineering effort than those in their teams who are forced to work with difficult-to-use or unreliable software tools, as they do not see the pain. Perhaps a version of “software carpentry” targeted at those PIs who are awarded or apply for software-intensive grants could be valuable here.

RSEs provide a useful contribution to their universities’ teaching missions, as well as research, as they are well placed to deliver the research programming training that many scientists now need. In the longer term, with programming skills taught to all through their careers, we hope specialist scientific developers will be less needed.

6.2.3 Key Next Steps

We will seek to identify and approach existing research programming organizations, to get their permission to list them on a list of research software groups. Casual conversation during the meeting made it clear that although the title is not widely used in the US, this position is not rare. We spoke with several individuals who, at distinct universities, had RSEs (in effect if not in name) who were funded under differing models.

We will also look for examples of groups which have successfully become self-sustaining following initial seed funding.

In this respect, information gathering via a survey and subsequent analysis could be very useful. We would need to assemble a list of targeted individuals. (What positions and ranks are likely to know and care enough to respond?) Perhaps the Science Gateway Institute has already acquired information that could be helpful to advance this issue, and/or craft a proper survey and suggest target individuals.

6.2.4 Plan for Future Organization and Future Needs

The UK RSE community will provide initial facilities to host this list, and continue to work to spread the initiative, but local leadership in the US is needed if this campaign is to succeed. This will require an initial gathering of identified research software organizations in the US to this end.

6.2.5 Description of Funding Needed

Financial support for an initial conference that brings together research software groups to form an organization and create a resource sharing structure would help to further this campaign. Funding to conduct and analyze a survey could also be quite useful as knowing where we stand today, and what models are in use could fuel the ideas for further development of developers in this category.

In the longer term, funding organizations, especially non-governmental organizations with the capability to effect innovation through seed funding, could provide support to nucleate the creation of research software groups. As noted above, Sloan has already initiated one such program, and collaboration with Sloan or at least study of their methods and success or failure could be extremely useful in approaching universities and other institutions in funding this development track. It seems clear that if the value proposition can be made to university administrators, this track could flourish with buy-in at the administrative level.

6.3 Transition Pathways to Sustainable Software: Industry & Academic Collaboration Working Group Discussion

Nic Weber44 will serve as the point of contact for this working group.

6.3.1 Group Members

  • Nic Weber – University of Washington
  • Suresh Marru – Indiana University
  • Jeffrey Carver – University of Alabama
  • Davide DelVento – NCAR/CISL
  • Steven Brandt – Louisiana State University

6.3.2 Summary of Discussion

The group’s initial broad question was, “What makes for successful transitions of scientific software from academia to industry?” There are a number of potential funding transitions that may occur:

  • A project could be re-funded, and development or maintenance of the software continue as planned.
  • A project might locate a new source of funding in which case the software may be further developed or simply maintained as before.
  • The project could transition to a community supported model whereby the software’s ownership, maintenance, and stewardship become similar to peer-production models in open-source (e.g., see Howison [41]).
  • The project could receive some form of industry sponsorship in which case ownership of the intellectual property, licensing, maintenance activities, hosting, etc. may change significantly.
  • The project could gain attention from a industry use case who would potentially make in-kind contributions by having paid staff contribute to the software.

We characterized each of the above potential changes in funding as “transition pathways” to sustainable software (see similar work by Geels and Schot [42]).

Our work at WSSSPE3 included the following three activities (described in more detail below): (1) brainstorming goals for this type of research, (2) imagining potential outcomes of completing a set of case studies on this topic, and (3) generating a set of working definitions for some of the broad concepts we are describing.

First, we discussed the goals of this research, attempting to answer the question What is the goal of doing research on transition pathways? A number of research questions arose: Can we identify collaborations that have occurred and try to understand which were successful, which were unsuccessful, and what factors contributed to these successes/failures? Can we determine what each partner wants to get out of such a collaboration? For example, why would industry be interested in collaborating with academia? Or why would academia be interested in collaborating with industry? How could we design a study that focused on the impact of the software in undergoing this type of transition?

Next, we imagined potential outcomes of research on this topic, involving a set of case studies that look at successful and unsuccessful transitions of researchers between academia and industry. This might address each of the transition types (as described below). Successful transitions are described as those that lead to either weak or strong sustainability (also defined below). In addition, the results from this research might help create a generalizable framework that might allow for the study of different transition pathways (other than academia to industry).

Finally, we created some general definitions for these concepts; we characterize transitions in the following ways:

  • Handoff model: academia initially writes the software, industry (for-profit or nonprofit) then takes over the project.
  • Co-Production Model: industry and academia interact throughout development of the project.
  • Sponsorship Model: academia writes and maintains the software; industry contributes funding for the development/maintenance of software. In this example, industry is also likely a user of the software.
  • Spinoff model: transition to a for-profit or non-profit company owned by or in collaboration with original developers.

We characterized sustainability in the following ways:

  • Weak Sustainability: Software continues to be accessible, useful, and usable.
  • Strong Sustainability: Software meets criteria above, but is also able to be reused for further innovation (i.e., issued non-restrictive open-source license).

We refer readers to Becker et al. [6] for an extended discussion of weak versus strong sustainability.

6.3.3 Description of Opportunity, Challenges, and Obstacles

The opportunity is to create a catalog of success/failure for current and future software projects to be prepared for transitions and achieve sustainability of the software.

The obstacle is more superficial, in finding a champion to gather such information. It will be a challenge to keep this information and surveys updated. With changing rapidly changing industry landscapes, an obsolete survey could be of less or no use.

6.3.4 Key Next Steps

Identify projects that are collaborative, perhaps by reviewing funded projects from programs specifically geared towards industry academic collaborations.

Develop a systematic process for conducting case studies (what kind of data are being gathered about each case).

6.3.5 Plan for Future Organization

No concrete plans have been made at this point. If the community can rally behind this topic, some momentum could be built. Those interested should post at

6.3.6 What Else is Needed?

Nothing at the moment.

6.3.7 Key Milestones and Responsible Parties

A key portion of this effort will require focused surveys of projects which have succeeded and failed in transition. Both these categories will yield good learning on what works and what does not work. The group has identified what needs to be studied further, but has not identified responsible parties to conduct them.

Community members could help in gathering data by means of interviews, historical documents or documentation, and surveys.

An example of data collection is:

  • Origin: Where did project start?
  • People involved: How many people in original project were involved in transition/collaboration?
  • Specs on software
  • Language
  • Size
  • Hardness (age)
  • Lead-up to Transition: How long was project in development before it began transition?
  • Motivation for Transition: Why was transition initiated? By whom?

6.3.8 Description of Funding Needed

Concrete funding needs were not discussed in this working group but a general impression was that some seed funding would motivate members of this group or others in community to launch a survey effort.

6.4 Engineering Design Group Discussion

Birgit Penzenstadler45 and Colin C. Venters46 will serve as the points of contact for this working group, and be responsible for ensuring timely progress of the planned actions.

6.4.1 Group Members

  • Birgit Penzenstadler – California State University, CA, USA
  • Colin C. Venters – University of Huddersfield, Huddersfield, UK
  • Matthias Bussonnier – UC Berkeley, CA, USA
  • Jeff McWhirter – Geode Systems
  • Patrick Nichols – National Center for Atmospheric Research, CO, USA
  • Ilian Todorov – Science & Technology Facilities Council, UK
  • Ian Taylor – Cardiff University, UK
  • Alexander Vyushkov – University of Notre Dame, IN, USA

6.4.2 Summary of Discussion

This group was comprised of members from different backgrounds, including quantum chemistry, epidemiology, microscopy, computer science, and software engineering. Each participant was invited to give their perspective on the topic area and what they thought were the crucial points for discussion. There was a general consensus that there was a need for relating principles to practice for the computational science and engineering community. Furthermore, various members of the group expressed their interest in tools and best practices for facilitating the maintenance and evolution of scientific software systems. It was agreed to identify principles from software engineering and from sustainability design and, based on those lists, discuss what each of those would mean applied to specific example systems from the expert domains of some of the group members. The group identified a number of software engineering principles drawn from the Software Engineering Body of Knowledge (SWEBOK) [32].

Software design principles include abstraction, coupling and cohesion, decomposition and modularization, encapsulation and information hiding, separation of interface and implementation, sufficiency completeness and primitiveness, and separation of concerns. Similarly, user interface design principles include learnability, user familiarity, consistency, minimal surprise, recoverability, user guidance, and user diversity. The sustainability design principles were drawn from the Karlskrona Manifesto on Sustainability Design [6]. The manifesto states that sustainability is systemic, multidimensional, and interdisciplinary; transcends the system’s purpose; applies to both a system and its wider contexts; requires action on multiple levels; requires multiple timescales; changing design to take into account long-term effects does not automatically imply sacrifices; system visibility is a precondition for and enabler of sustainability design. A number of sustainable software engineering principles proposed by Tate [31] were also considered including: continual refinement of product and project practices; a working product at all times; continual emphasis on design; and value defect prevention over defect detection.

This congregated list is an initial collection of principles that could be extended by adding from further related work form separate disciplines within the field of software engineering, including requirements engineering, software architecture, and testing. The group identified two example systems to discuss the application of the principles. The first one was a quantum chemistry system that allows the analysis of the characteristics and capabilities of molecules and solids. The second one was a modeling system for malaria that permitted biologists to analyze a range of datasets across geography, biology, and epidemiology, and add their own datasets. The group then examined the principles and took a retrospective analysis of what the developers did in practice against how the principles could have made a difference. This raised the question, what do the principles mean for computational scientific and engineering software? Similarly, how do the principles relate to non-functional requirements? It was suggested that at the very minimum, that sustainable software engineering principles should be mapped to two core quality attributes that underpin technically sustainable software:

  • Extensibility: the software’s ability to be extended and the level of effort required to implement the extension;
  • Maintainability: the effort required to locate and fix an error in operational software.

These fundamental building blocks could then be extended to include other quality attributes such as portability, reusability, scalability, usability, and energy efficiency etc. Nevertheless, this raises the question of what metrics and measures are suitable to demonstrate the sustainability of the software. In addition, what do the five dimensions of sustainability mean for scientific software, i.e., environmental, economic, social, technical and individual?

6.4.3 Description of Opportunity, Challenges, and Obstacles

The opportunity was identified to distill existing software engineering and sustainability design knowledge into “bite sized” chunks for the Computational Science and Engineering Community. In addition, two primary challenges were identified:

  • Mapping of the principles to best practices.
  • Demonstrating the return on investment of those best practices.

6.4.4 Key Next Steps

In order to achieve the following three goals: (1) a systematic analysis of a number of example systems from different scientific domains with regards to the identified principles, (2) the identification of the commonalities and gaps in applying the principles to different scientific systems, and (3) a proposal of a set of guidelines on the principles, the following next steps were discussed.

6.4.5 Plan for Future Organization

The following plan for future organization was discussed:

  • Identify suitable undergraduate or post-graduate students.
  • Design and pilot study.
  • Organize coordinating online calls via Google Hangout.

6.4.6 What Else is Needed?

  • Ethics committee review panel approval required for data collection.

6.4.7 Key Milestones and Responsible Parties

The following key milestones were discussed as a roadmap for the set of guidelines on software engineering principles:

  • Oct/Nov 2015: Study design and interview guideline
  • Jan/Feb 2016: Interviews conducted and transcribed
  • Mar/Apr 2016: Analysis complete
  • May 2016: Report written

6.4.8 Description of Funding Needed

Specific funding was not discussed in this working group. However, this is a open topic that can be discussed in relation to emerging funding calls from National agencies or grant proposal initiatives.

6.5 Metrics Working Group Discussion

Gabrielle Allen47 will serve as the point of contact for this working group.

6.5.1 Group Members

  • Gabrielle Allen – University of Illinois at Urbana-Champaign
  • Emily Chen – University of Illinois at Urbana-Champaign
  • Neil Chue Hong – U.K. Software Sustainability Institute
  • Ray Idaszak – RENCI, University of North Carolina at Chapel Hill
  • Iain Larmou – Engineering and Physical Sciences Research Council
  • Bernie Randles – University of California, Los Angeles
  • Dan Sellars – Canarie
  • Fraser Watson – National Solar Observatory

6.5.2 Summary of Discussion

The group discussion began by agreeing on the common purpose of creating a set of guidance giving examples of specific metrics for the success of scientific software in use, why they were chosen, what they are useful to measure, and any challenges and pitfalls; then publish this as a white paper. The group discussed many questions related to useful metrics for scientific software as follows:

  • Is there a common set of metrics, that can be filtered in some way
    • – Does this create a large cost
  • Can we fit metrics into a common template (i.e., for collection, for description)
  • Which would be the most useful ones
    • – Which ones would be most useful for each stakeholder
  • Which ones are the most helpful, and how would we assess this
  • How do you monitor
    • – Self-checking – if monitoring is done in the open, then people will call out cheats
  • Should this be published with the software metadata
    • – This would make it easier for public to see the metadata
    • – However, there is no commonly-used standard (DOAP is a good standard but not widely adopted)
    • – The Open Directory Project (ODP) metadata is available for UK infrastructure
  • Intersection of most useful and easiest to collect should be explored
  • How can students/curricula be used as part of a solution
  • Number of users could be affected by other metrics including, e.g., accessibility
  • Assume metrics are collected properly, but guidance should be provided none-the-less
  • Continuum for each metric
    • – Ideal situation is the absolute minimum, so that people can decide on their own what the cost versus usefulness tipping point is
  • Maturity plays a part
    • – Consider different metrics brackets for different maturity levels
  • What are we using metrics for
    • – What software should I use if I have a choice
    • – Where should funders place funding for best impact (e.g., funding two-star software versus three-star) and where there are gaps
    • – How to promote reduction of code proliferation
    • – Metrics used for software panels to provide information
    • – Metrics used for finding problems in their systems
  • Can we use metrics to help people identify the best codes as part of a community effort

Next, a roadmap for how to proceed was discussed including creating a set of milestones and tasks as follows:

  • Can we create a roadmap and milestones for this activity
  • Need to come up with a set of tasks
  • Go to NSF Software Infrastructure for Sustained Innovation (SI2) projects asking them what metrics they defined, and how useful they were
    • – Milestone: Create report which assesses the metrics that SI2 projects used
      • * Ask SI2 PIs to say what metrics they said they would use (copied from proposal)
      • * Ask SI2 PIs what numbers they reported
      • * Ask SI2 PIs what they would have changed
      • * A UIUC student on the project will work on this
    • – Tentatively aim for March 2016
  • Do something similar for UK SFTF and TRDF software projects to ask them what would be useful metrics to report; also eCSE projects
    • – Compare these to understand if there were any implications for including metrics
  • Collaboratively create plan and documentation for doing this
    • – Give some examples from group members projects, and aim to build out some of the measurement continuum
    • – Road-test at the WSSSPE4 meeting
  • Collect the various frameworks together and do a comparison summary

The idea was put forth for the group to interact with the organizing committee of the 2016 NSF Software Infrastructure for Sustained Innovation (SI2) PI workshop in order to email out a software metrics survey to all SI2 and related awardees as a targeted and relevant set of stakeholders. This survey would be created by one of the student group members. Similarly, it was suggested that a software metrics survey be sent to the UK SFTF and TRDF software projects to ask them what metrics would be useful to report. The remainder of the discussion focused mainly on the creation of a white paper on this topic. This resulted in a paper outline and writing assignments with the goal of publishing in venues including WSSSPE4, IEEE CISE, or JORS.

6.5.3 Description of Opportunity, Challenges, and Obstacles

The following opportunities, challenges, and obstacles were discussed:

  • Metrics are important for:
    • – Tenure and promotion
    • – Scientific impact
    • – Discovery
    • – Reducing duplication
    • – Basis for potential industrial interest in adopting software
    • – Make case for funding
  • No commonly used standard for collecting or presenting metrics
  • We do not know if there is a common set of metrics
  • We have to persuade projects that it is useful to collect metrics

6.5.4 Key Next Steps

The following next steps were discussed:

  • Skype phone call to coordinate shortly after the conclusion of the WSSSPE3 workshop
  • Get started on IRB at University of Illinois Urbana-Champaign in anticipation of SI2 project survey (may need more thought into survey)
  • Get started on white paper and associated survey

6.5.5 Plan for Future Organization

The following plan for future organization was discussed:

  • Our group has created a white paper outline with sections assigned to the above individuals; see timeline below.
  • Organize coordinating phone calls.

6.5.6 What Else is Needed?

The following list of what else is needed was discussed:

  • IRB approval/exemption needed for surveys, collecting data
  • Coordination with 2016 NSF SI2 PI workshop organizing committee to possibly piggyback on this event to offer survey to attendees in advance
  • Coordination (mail communication, info page, etc.) via WSSSPE GitHub or other means?

6.5.7 Key Milestones and Responsible Parties

The following items were discussed as a roadmap for the production of a white paper:

  1. October – November 2015: IRB paperwork as appropriate completed (Gabrielle Allen and Emily Chen)
  2. October – December 2015: Draft white paper sections 1-3 (the paper outline has initial writing assignments)
  3. October – December 2015: Run surveys and collect information
    1. Piggyback on planning for 2016 NSF SI2 PIs meeting to be held Feb 16-17, 2016
  4. January – February 2016: Analyze results of data collection from projects
  5. March – April 2016: Draft sections 4-7 of the white paper
  6. May 2016: Draft section 8-9 of the white paper
  7. May – June 2016: Get initial feedback from members of the community and revise
  8. Est. July 2016: By the time of next CFP for WSSSPE, to have a complete draft of the white paper
  9. Est. Sept – Oct 2016: Responses to white paper to be submitted to WSSSPE4

6.5.8 Description of Funding Needed

Funding needs were not discussed in this working group and it was thought that this could potentially be revisited down the road.

6.6 Training Working Group Discussion

Nick Jones48 will serve as the point of contact for this working group, and be responsible for ensuring timely progress of the planned actions.

6.6.1 Group Members

  • Nick Jones – New Zealand eScience Infrastructure
  • Iain Larmour – Engineering & Physical Sciences Research Council, UK
  • Erin Robinson – Foundation for Earth Science

6.6.2 Summary of Discussion

While little training focuses specifically on sustainable software, a variety of training activities could increase researcher awareness of and engagement with software professionals and software engineering practices. Research Software Engineers are being recognized as critical contributors to high quality research; the pathway to acquire and master the relevant skills is not yet clear; equally those skills required by researchers in general are also not commonly understood nor routinely developed.

The group’s discussion explored a rapidly growing array of training that is seen to contribute to sustainable software. The offerings are diverse, including: self-paced online modules focused around specific tools; single and multiple day training workshops that raise awareness of a tool chain to support collaborative and shared software development within a research workflow; block courses specializing on particular methods, technologies, and applications; academic programs at undergraduate and masters levels; doctoral training programs that in part contain requisite skills training activities.

While some of this training focuses on applying software engineering practices within the context of research, meeting the values and goals of research are less often incorporated as explicit learning outcomes. With software (and similarly, data) often being the only tangible artifact of a research method or protocol, the dependency between software applications and the quality of research adds complexity to the learner’s journey. In recognition of the longer term investment required by researchers to integrate such skills into their research practices, many activities are focusing on emotionally engaging researchers and cohorts, to build a sense of shared purpose beyond the obvious goal of technical skill acquisition.

In reviewing current training activities, the group identified a variety of perspectives seen as useful in positioning activities in ways to better communicate why and when best to apply each activity. Training can be categorized on a variety of spectra, with content and delivery ranging across them, for example: programming to research; basic to advanced; technical to emotional; informal to formal; and self-paced to participative. A few attempts have been made to situate a cross section of training activities within such dimensions, creating easier means of communicating the value of any specific opportunity and the pathways across opportunities over time.

Evaluation of training delivery and outcomes is seen as a weakness common to most non-academic training activities. Opportunities for measuring success in delivering training start simply with collecting a Net Promoter Score, which lets those delivering training know whether attendees are likely to recommend the training to others. In looking at the longer term outcomes for the learner, frameworks such as Bloom’s taxonomy and Kirkpatrick’s evaluation model offer possible approaches.

In this latter case of formal evaluation, ownership of evaluation as a component of career development for any researcher appears mostly absent. While academic research institutions have professional development centers to support research staff, the skills taught which might impact on sustainable software are limited at best, and lack a clear and coherent development pathway.

Coordination of these training projects will depend on buy-in from a broad range of training program and activity leaders, suggesting a key opportunity lies in identifying and bringing together these people on a regular basis.

6.6.3 Description of Opportunity, Challenges, and Obstacles

Software skills are needed by an increasing array of researchers and fields. The training arc is not well-defined, with a sometimes baffling array of training opportunities responding to various facets of skill deficit and need. Given this current complexity, coordination across training projects would create common frames of reference, communicating and integrating activities to better serve the needs of researchers.

Building this community could lift the maturity of training projects and capabilities, enabling more advanced approaches to address key gaps in evaluation, career development, and a lift in the standard of research practices.

In aiming at these opportunities, it will be necessary to find the means to support those involved in leading training activities to allocate time to coordination activities, which will often sit beyond their current scope of responsibility.

These activities are also distributed globally, with no single country or region offering a comprehensive set of capabilities and initiatives. Any coordination activity will therefore need to raise the profile of the opportunity gap with relevant research funders and policy makers.

6.6.4 Key Next Steps

The goal of the following next steps is to quickly test whether there is interest in establishing a community committed to increasing the degree of coordination across training projects.

  1. Hold a virtual meeting by December 2015, to bring together a broader group of interest in this topic, with specific goals to:
    1. Identify programs with existing activities aimed at integrating across training projects.
    2. Identify training projects with an interest in participating in coordination efforts.
    3. Identify funding opportunities to bring together training program and project leaders to identify shared goals for future coordination of activities.
    4. Agree on a communications plan to qualify whether programs, projects, and funders are interested in engaging and committing to ongoing activities.
  2. Review progress within 3 months, to establish next steps, if any.

6.6.5 Plan for Future Organization

Continue to track progress by posting comments to WSSSPE3 issue.

6.6.6 What Else is Needed?

If the group moves from early-stage formation into working towards shared goals, expertise will likely be required in pedagogy and training evaluation.

6.6.7 Key Milestones and Responsible Parties

  1. October through December, Nick Jones and Erin Robinson to draft WSSSPE3 report back.
  2. Before February 2016, Nick Jones and Erin Robinson to call a meeting of the broader group, to review key next steps.
  3. Second quarter 2016 – if willing parties are identified, draft workshop proposal and identify a relevant forum, including future WSSSPE events.

6.6.8 Description of Funding Needed

Workshop/RCN travel funding to bring together key program, project, and funder representatives from across North America, EU, UK, Australasia. In addition, funding to support work on better defining the landscape of training activities, the useful perspectives in communicating the value of the varied training projects, and the possible pathways through training activities over time.

6.7 Software Credit Working Group Discussion

Kyle Niemeyer49 will serve as the point of contact for this working group, and be responsible for ensuring timely progress of the planned actions.

6.7.1 Group Members

  • Alice Allen – Astrophysics Source Code Library
  • Sou-Cheng Choi – NORC at University of Chicago, Illinois Institute of Technology
  • James Hetherington – University College London
  • Lorraine Hwang – University of California, Davis
  • Daniel S. Katz – University of Chicago, Argonne National Laboratory
  • Frank Löffler – Louisiana State University
  • Abigail Cabunoc Mayes – Mozilla Science Lab
  • Kyle E. Niemeyer – Oregon State University
  • Grace Peng – National Center for Atmospheric Research
  • Ilian Todorov – Science & Technology Facilities Council, UK

6.7.2 Summary of Discussion

The following section summarizes the working group’s discussion based on contributions prior to the meeting [43] and the collaborative notes taken during the meeting [44]. Please refer to the original sources for the unedited discussions if necessary.

Initial discussions focused on both various mechanisms for, and the philosophical approach behind, crediting software in scientific papers. These began with proposals for various ways to credit software (or other research products including data) that contributed more significantly than a generic citation, including:

  • A hierarchy of citations, with a “substantial” citation category to indicate software or data that played a more significant role in the research;
  • Transitive credit [37, 38], which assigns contriponents (contributors and components) various weights depending on their level of importance;
  • Project CRediT [45], which assigns roles to paper authors based on their specific contributions; and
  • Mozilla Science Lab’s recently introduced Contributorship Badges for Science [46], which provide a badge—associated with an ORCID [47]—that recognizes author contributions using the taxonomy outlined in Project CRediT.

However, as of this writing, only Project CRediT roles [48, 49] and Contributorship Badges [46] have been implemented for published papers, and both of these only provide a single “Software” or “Computation” category associated with software. In addition, neither of these options allows for the citation of software itself, but only provide an author contribution related to software. The discussion quickly focused on transitive credit as a more quantitative measure of allocating credit to both authors and software, although there were some concerns about authors overestimating their own contributions compared to prior work.

The discussion then evolved into philosophical questions about the importance or reliance of a particular work on prior science, materials, or software—in other words, whether there is a difference between depending on prior scientific advances and depending on certain software (or experimental equipment). Multiple contributors converged on the conclusion that unique capabilities require some additional credit. The—albeit limited—consensus was that if a particular study relied on the unique capabilities of software, data, or an experimental apparatus, then the authors or developers that created this capability should be credited somehow.

The group also agreed on the fact that additional data was required to support the assertion that software was not being sufficiently cited in the literature. In particular, this issue seemed to be field-dependent. For example, as shown by a study of Howison and Bullard [50], in the field of biology, the most-cited papers appear to be those describing scientific software. However, this may not—and likely is not—the case in other fields, nor is it clear whether developers of scientific software, even in the case of the biology field, are receiving sufficient credit for their efforts.

In the breakout sessions on the first day of WSSSPE3, the group discussed and deliberated over the Entertainment Identifier Registry (EIDR) [36] as a potential model for scientific software. That system assigns unique Digital Object Identifiers (DOIs)—the same system used for scientific publications—to all content (e.g., movies, television shows) and contributors, along with relevant metadata. One important use of the EIDR system is to track rights and credits for contributors to entertainment works in order to distribute revenues—similar to the proposed transitive credit concept.

The group also discussed separating quantitative measures (e.g., number of citations) from the value of a work in order to give credit, moving towards qualitative or anecdotal evidence of value. Other topics that were brought up included a form of PageRank [51] for citations, based on number of mentions, and using market penetration or adoption rate in a community as a metric, although it was not clear how this would be measured. Finally, the concept a software tool’s uniqueness or indispensability to a community was mentioned, with value being characterized by a particular piece of software either offering unique capabilities or doing something better, faster, or with less computational requirements than other offerings.

On the second day of WSSSPE3, the group decided to put aside the taxonomy of contributions and focus on software citations to ensure developers receive credit (regardless of contribution). Eventually, once software citations are standardized, the goal would be to return to establishing different roles/contributions for this credit. Following this decision, the group identified two necessary actions to move forward:

  1. standardizing a citation file or some other form of metadata associated with software, and
  2. standardizing the way to cite software (used directly) in papers.

For both of these actions, a number of ongoing efforts were identified and discussed.

Software Citation Metadata:

At a minimum, the metadata required for software citation includes:

  • Name of software,
  • DOI for software,
  • Contributors, in the form of names and ORCIDs,
  • Software dependencies, in the form of DOIs, and
  • Other people and artifacts that would be cited or acknowledged in a paper.

This information would then be contained in a citation file, e.g., as part of the GitHub repository. The group also discussed similar efforts such as CodeMeta50, an attempt to codify minimal metadata schemes in JSON and XML for scientific software and code, and implementing transitive credit via JSON-LD [38]. Some questions arose about how this information would be stored for closed-source software.

As one mechanism for constructing accurate contributor lists from existing project contributors, the group discussed associating GitHub accounts—as well as accounts on Bitbucket, CodePlex, and other repositories for open-source scientific software—with ORCID accounts. However, a (quick) response from GitHub (via Arfon Smith) indicated that this might not be possible in the near future: “GitHub doesn’t have any plans to allow ORCID accounts to be associated with GitHub user accounts.”

Citing Software in Publications:

Although far from a standard practice, examples of citing software in publications can be found in various scientific communities—notably, representative samples can be found in astronomy [52] and biology [50]. The group recommended collecting similar examples from other communities, and then developing a software citation principles document in concert with the FORCE11 Software Citation Working Group (see §6.7.5 for more details), following the model of the FORCE11 Data Citation Principles document [53].

The group further discussed briefly whether software used directly in a publication—whether to perform simulation or analysis, or as a dependency for newly developed software—should be distinguished from other references due to the dependence of the study on these research artifacts. Suggestions included a separate list of citations (with DOIs) for software and other research objects that serve this sort of “vital” role. Similar recommendations were made by the credit breakout group at WSSSPE2 [4].

Finally, although a discrete task from software citations, significant discussion focused on ensuring software citations are indexed in the same manner as publications, allowing the construction of a corresponding software citation network. Currently, software releases can receive citable DOIs via Zenodo [54] and figshare [55]; however, these citations are not processed by indexers such as Web of Science, Scopus, or Google Scholar. Thus, either in parallel or following the primary task, the group will need to reach out to these organizations. Initial conversations with Elsevier/Scopus via Michael Taylor during WSSSPE3 clarified that Scopus is not yet DataCite DOI aware, and also does not yet have an internal identifier for software or data (but needs/plans to add this support). Taylor said they prefer a “software article” with the usual article metadata (e.g., authors, citations), and mentioned Zenodo as an example – this proposal seemed to align with our group’s discussions. Taylor also mentioned another benefit of the software and associated DOI on GitHub: in addition to a citation, one could obtain statistics on usage/downloads/forks, which happens to be what Depsy51 is beginning to try to do.

6.7.3 Description of Opportunity, Challenges, and Obstacles

There currently is no standard mechanism for citing software or receiving credit for software (akin to citations for publications). Software is eligible for DOI assignment, but DOI metadata fields are not well tuned or standardized for software (vs. publications). Some software providers apply for DOIs, but this is not widely adopted. Also, there is no mechanism to cite software dependencies within software.

Major obstacles include the fact that indexers (e.g., Scopus, Web of Science, Google Scholar) do not currently support software/data document types or DataCite DOIs. Therefore, even with universal association of scientific software with DOIs and standardized practices for citing software in publications, software citations will not be indexed in the same manner as traditional publications.

Although this working group’s discussions at WSSSPE3 did not focus much on the topic of tenure and professional advancement, the group recognized that there is no standard policy—generally even within a single university—for software products to be included in promotion and tenure dossiers. Thus, it may be difficult to encourage valuing software contributions across the United States or United Kingdom and globally; furthermore, stakeholders are typically not tenured and thus may not be influential enough to change the status quo. However, as discussed in Section 5.2, this is changing for Research Software Engineers, at least in the UK.

6.7.4 Key Next Steps

  1. Hold virtual meeting to determine group members responsible/willing to work on the following tasks, to be organized within one month of the workshop.
  2. Compile best practices of software citation across multiple disciplines, including journals and communities of interest/practice in the research world, to begin by December 2015.
  3. Compile examples of including other products in promotion and tenure dossier, to begin by December 2015.
  4. Draft the Software Citation Principles document (including citation metadata file), by April 2016.
  5. Publish/release the Software Citation Principles document, by August 2016.
  6. Reach out to journals, publishers, teachers/educators, indexers, and professional societieslikely through meetings with key groups, to begin by September 2016.

6.7.5 Plan for Future Organization

The WSSSPE breakout group plans to join efforts related to citing software with the FORCE11 Software Citation Working Group (FORCE11-SCWG)52; Kyle Niemeyer formally requested the merging of these groups following the meeting. However, some future plans of the WSSSPE group fall outside the scope of FORCE11-SCWG, which covers software citation practices. These activities include working with indexers such as Web of Science and Scopus to index software citations archived on, e.g., Zenodo or figshare, and pursuing the development of an open indexing service; such plans will be pursued either separately or through the formation of follow-on FORCE11 working groups.

The group will primarily communicate electronically, with Kyle Niemeyer responsible for ensuring regular progress.

6.7.6 What Else is Needed?

The near-term actions of the group, focused mainly on software citation, do not require any additional resources. However, connections with publishers and indexers will be needed to pursue related activities, although the FORCE11-SCWG may satisfy this need; in addition, some members of the group already reached out to relevant contacts. Funding may be needed to organize meetings or for group members to attend relevant meetings, as discussed further below.

6.7.7 Key Milestones and Responsible Parties

Following the meeting, Kyle Niemeyer formally requested the merging of software citation activities with FORCE11-SCWG. Within a month of the meeting, Niemeyer will organize a virtual meeting of the group and manage the division of responsibilities for compiling existing practices of software citation and including software/products in promotion and tenure dossiers. Building off of these efforts, the next major milestone is drafting the Software Citation Principles document in collaboration with the SCWG, targeted for April 2016. While the existing directors of the SCWG, Arfon Smith and Dan Katz, lead the efforts of that group towards the Software Citation Principles document, Kyle will help coordinate contributions from the WSSSPE group members.

6.7.8 Description of Funding Needed

Some funding would be useful to support primarily travel to conferences for group meetings (e.g., FORCE2016)53, and to hold meetings to bring together both group members and key stakeholders (e.g., journals, publishers, professional societies, indexers). In addition, funding would be desired to support group members’ time to perform work towards the key steps described previously.

6.8 Publishing Software Working Group Discussion

Steven R. Brandt54 will serve as the point of contact for this working group.

6.8.1 Group Members

  • Steven R. Brandt – Louisiana State University
  • Daniel Gunter – LBNL
  • Yuhan Ding – Illinois Institute of Technology
  • Neil Chue Hong – Software Sustainability Institute

6.8.2 Summary of Discussion

A tentative first cut at the list containing executable papers identified the following:

  • ACM Transactions on Mathematical Software (TOMS): provides the additional step of having reviewers validate the code which was submitted with the publication.
  • The Mathematica Journal: publishes Mathematica notebooks (with equations, figures, etc.) directly.
  • O’Reilly Media: announced that it plans to make IPython Notebooks a first-class authoring environment for their publishing program alongside their existing mechanisms.
  • Nature: offers a list of notebooks published alongside more traditional articles, and is looking at ways to make these documents more official. There are, in fact, a number of journals that offer “electronic supplements” to the more traditionally published static articles.
  • IPython: maintained a list of “reproducible academic publications” [56].
  • KBase: offers narratives built on IPython or Jupyter notebooks for assembling publications that are reproducible, and can be commented or annotated.

The group also discussed future possibilities for a new publication format that might provide advantages:

  • Journals could be built around an existing, widely used framework thereby reducing the burden of studying code on the part of reviewers (common bits of infrastructure which are not relevant to the science would be automatically excluded).
  • Journals might be encouraged to use more metadata, making them easier to mine for various analytical purposes.
  • The Research Ideas and Outcomes (RIO) journal is an effort to publish fragmentary results that can subsequently be combined into a single content item.
  • Papers could be made more understandable. Each equation or technical term could be linked to a document/tutorial explaining its origin and/or derivation.
  • So many options for publication currently exist that good science may be getting lost in the noise. Would some sort of “upvote” mechanism be of value?
  • Some sort of Replicated Computation Results badge could be made available to publications that have undergone greater scrutiny (this is already done by TOMS).

6.8.3 Description of Opportunity, Challenges, and Obstacles

The opportunity is to collect a list of executable papers and shine a light on the experiments and development efforts currently underway.

The only obstacle to this is the difficulty in finding and identifying such publications. The Software Sustainability Institute was able to do something similar for publications about software by making a public page on the Software Sustainability Institute’s website ( containing a catalog of these publications and enlisting the help of the community to grow the list.

6.8.4 Key Next Steps

Create the first version of the web page to be displayed on the Software Sustainability Institute’s website: We expect the page to be live in early January of 2016.

An ongoing effort to update the page should follow.

6.8.5 Plan for Future Organization

None at this time.

6.8.6 What Else is Needed?

Nothing else at this time.

6.8.7 Key Milestones and Responsible Parties

Steven R. Brandt has created a first version of the page, and it is in the process of being posted on the Software Sustainability Institute’s website: Neil Chue Hong will take responsibility for the page once it is up.

6.8.8 Description of Funding Needed


6.9 User Community Working Group Discussion

Point of contact: Dan Gunter55 and Ethan Davis56.

6.9.1 Group Members

  • Ethan Davis – UCAR Unidata
  • Dan Gunter – Lawrence Berkeley National Lab
  • Liz Jessup – University of Colorado
  • Mark Miller – University of California, San Diego
  • Lindsey Powers – The HDF Group
  • Daniel Ziskin – NCAR Atmospheric Chemistry Observations and Modeling (ACOM) Laboratory

6.9.2 Summary of Discussion

Discussion revolved around a few questions: what is the benefit of having a “community” for software sustainability, what practices and circumstances lead to having and maintaining a community, how can funding help or hinder this process, and perhaps most importantly, how can best practices be described and distilled into a document that can help new projects.

The benefits of having a community that were brought up were considered largely obvious. In addition to having advocates for the software, and a possible source of “free” contributions to the codebase, the community becomes a good source for requirements, feedback, and metrics. The software community can also act as “cheerleaders” who convince funders or other potential users to fund/use the software, and thus help sustain the software.

Practices and circumstances that lead to a community are first, that the software offers value. But in addition to this, a community will be much more likely to form if they receive (expert) support when they have questions. Additional contributing factors are good usability (not always needed), and an open development process such as IPython developer meetings on YouTube. It was also pointed out that an evangelist for the project, not necessarily but often one of the developers, can often make a big difference.

Funding can help the process by encouraging both value to the community and high-quality user support. Only providing funding for the software development may create good software, but with less likelihood to have a real community. It was discussed that federal laboratories are a good incubator for software communities, and that a general facility like EarthCube is too dispersed to really make a community. Also, domain-specific groups within laboratories or universities might provide as an incubator for software communities.

In describing best practices, the group discussed the different modes for starting a scientific software project: building on an existing product that needs improving, recognizing an unsatisfied need of an existing community, or creating a new solution to a need not yet recognized by the community. The group also thought that the existing books on software communities would need to be evaluated in light of differences between science software projects and general open-source software (OSS) projects in terms of scale, science, acknowledgement and credit, and funding models.

6.9.3 Description of Opportunity, Challenges, and Obstacles

The main opportunity is to increase awareness among scientific software developers and project managers of the importance of developing a community around their project. While this message is fairly well understood in the open source community, the scientific community can be more focused on the science a software project is supporting rather than the software project itself.

As with many of the issues relevant to the sustainability of science software, the main challenge here will be changing the culture and expectations around scientific software.

6.9.4 Key Next Steps

The most important next steps is a “Best Practice” document, which would describe what successful projects with engaged communities look like, how to replicate this type of project, and look at end-of-life on a community project. Inputs to this document would include a software community survey of highly functioning communities such as R Open Science, Python SciPy, OPeNDAP, and Unidata, with analysis of factors that feed into their success. Also references like the “Art of Community” could be adapted and summarized for the science software community.

More specifically, the group would like to take the following steps:

  • Survey successful science software projects
  • Survey community members from the surveyed projects
  • Distill the survey results and document best practices around community engagement
  • Look for ways to raise awareness

Another next step would be increasing recognition of need for science software projects to focus on building and supporting their user communities. Good software engineering practices are not enough, and popular training like Software Carpentry does not currently address this issue head-on.

6.9.5 Plan for Future Organization

No definite plans were agreed upon for future organization. The major ideas discussed were coordinating with another group or adapting some existing text.

Collaboration within the framework of an existing organization seems a good initial path. Mozilla Science maintains a “Working Open Project Guide” [39], the introduction of which states:

Working openly with contributors enables your community to learn how to build and collaborate together. This document is a guideline on how to work openly and involve others in your projects with Mozilla. We want to help you engage your community in a way that encourages contributors and builds other leaders.

Another idea is to form a group that could adapt existing commercial-oriented guidelines for the world of scientific software and top-down funding structures. For example, to distill the “Art of Community” by Jono Bacon [40] for scientific software.

6.9.6 What Else is Needed?

The group had many points of agreement, but there is not currently a dedicated core group of people who have committed to producing the key milestones. Coordination via phone or online would be necessary to build this “community” of contributors.

6.9.7 Key Milestones and Responsible Parties

The key milestones for the group’s activities align closely with the Key Next Steps above:

  • Complete and write up a survey of project members, and community members, for successful science software projects
  • Distill the survey results and document best practices around community engagement

6.9.8 Description of Funding Needed

With a small amount of seed funding, it is possible that members of this group or other parties could spend the time necessary to devise a survey of existing projects and deploy this, probably by traveling to meetings and workshops for the various software communities.

(7) Conclusions

In WSSSPE3, we attempted to take what we learned from WSSSPE1 and WSSSPE2 in how we can collaboratively build a workshop agenda and turn that into an ongoing community activity. The success or failure of these efforts will only become apparent over time.

The workshop had two components, presentations and working groups. The presentations, in the first half day of the workshop, included an inspirational keynote and a set of lightning talks. We used lightning talks for two reasons: first, the need of some participants to have a slot on the agenda to justify their attendance; and second, as a way to get new ideas across to all the attendees. We broke with the tradition of requiring the lightning talk submitters to self-publish their papers, and instead used a common peer-review platform57, choosing to publish their slides on the workshop website instead.

The working groups met for a small part of the first half day and all of the second day, with the exception of some short periods for the groups to report back to the collected workshop attendees. Each group determined a set of activities that the members could do to advance sustainable software in a particular area.

The results of these group sessions made it clear that there are many interlinked challenges in sustainable software, and that while these challenges can be addressed, doing so is difficult because they generally are not the full-time job of any of the attendees. As was the case in WSSSPE2 as well, the participants were willing to dedicate their time to the groups while they were at the meeting, but afterwards, they went back to their (paid) jobs.

We need to determine how to tie the WSSSPE breakout activities to people’s jobs, so that they feel that continuing them is a higher priority than it is now, perhaps through funding the participants, or through funding coordinators for each activity, or perhaps by getting the workshop participants to agree to a specific schedule of activities during the workshop as we have tried to do in WSSSPE3. It remains to be seen, however, if the participants will meet the schedules they set.

The overall challenge left to the sustainable software community is perhaps one of organization: how to combine the small partial efforts of a large number of people to impact a much larger number of people: those who develop and use scientific software. While WSSSPE might help focus the actions of the groups, something more is needed to incentivize the wider community, which is a generalization of the sustainable software problem itself.


36London Software Credit workshop: 

38FORCE11-SCWG GitHub page, 

52FORCE11 Software Citation Working Group, 

Supplementary Files

The supplementary files for this article can be found as follows:


Work by Katz was supported by the National Science Foundation while working at the Foundation. Any opinion, finding, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Choi’s work was supported in part by the National Science Foundation research grant DMS-1522687 and a WSSSPE3 travel award. She thanks the encouragement and discussion with Fred Hickernell. Hetherington was funded by the Software Sustainability Institute, RCUK grants EP/H043160/1 and EP/N006410/1. Work by Gunter was supported by the Office of Science, Office of Biological and Environmental Research, of the U.S. Department of Energy (DOE) under Award Numbers DE-AC02-05CH11231, DE-AC02-06CH11357, DE-AC05-00OR22725, and DE-AC02-98CH10886, as part of the DOE Systems Biology Knowledgebase and by the Office of Science, Office of Advanced Scientific Computing Research (ASCR) of the U.S. Department of Energy under Contract Number DE-AC02-05CH11231 as part of the Template Interfaces for Agile Parallel Data-Intensive Science (TIGRES) project. WSSSPE3 was supported by NSF award 1434218 and funding from the Gordon and Betty Moore Foundation.

Competing Interests

Kyle Niemeyer is an Associate Editor of JORS, although uninvolved in the processing/review of this article.


  1. Katz, DS, Allen, G, Chue Hong, N, Parashar, M and Proctor, D (2013). First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE): Submission and Peer-Review Process, and Results.  arXiv; 1311.3523. 

  2. Katz, DS Choi, SCT Lapp, H Maheshwari, K Löffler, F Turk, M et al. (2014). Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1). Journal of Open Research Software 2(1)DOI: 

  3. Katz, DS Allen, G Chue Hong, N Cranston, K Parashar, M Proctor, D et al. (2014). Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2): Submission, Peer-Review and Sorting Process, and Results.  arXiv; 1411.3464. 

  4. Katz, DS Choi, SCT Wilkins-Diehr, N Chue Hong, N Venters, CC Howison, J et al. (2016). Report on the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). Journal of Open Research Software, Accepted. Available at 

  5. Turk, MJ Smith, BD Oishi, JS Skory, S Skillman, SW Abel, T et al. (2011). yt: A Multi-code Analysis Toolkit for Astrophysical Simulation Data. ApJS Jan 2011192DOI: 9. 

  6. Becker, C Chitchyan, R Duboc, L Easterbrook, S Mahaux, M Penzenstadler, B et al. (2014). The Karlskrona manifesto for sustainability design.  arXiv; 1410.6968. 

  7. Patra, AK Bauer, AC Nichita, CC Pitman, EB Sheridan, MF Bursik, M et al. (2005). Parallel adaptive numerical simulation of dry avalanches over natural terrain. Journal of Volcanology and Geothermal Research. Modeling and Simulation of Geophysical Mass Flows 139(1): 1–21, DOI: 

  8. Meng, H, Kommineni, R, Pham, Q, Gardner, R, Malik, T and Thain, D (2015). An invariant framework for conducting reproducible computational science. Journal of Computational Science. Computational Science at the Gates of Nature 9: 137–142, DOI: 

  9. Meng, H and Thain, D (2015). Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids. Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing. VTDC ’15. New York, NY, USAACM: 23–30, DOI: 

  10. Meng, H, Wolf, M, Ivie, P, Woodard, A, Hildreth, M and Thain, D (2015). A case study in preserving a high energy physics application with Parrot. Journal of Physics: Conference Series 664(3): 032022.DOI: 

  11. Huo, D, Nabrzyski, J and Vardeman, C (2015). An Ontology Design Pattern towards Preservation of Computational Experiments. Proceedings of the 5th Workshop on Linked Science 2015 – Best Practices and the Road Ahead (LISC 2015) co-located with 14th International Semantic Web Conference. 2015, ISWC 

  12. Baxter, R, Chue Hong, N, Gorissen, D, Hetherington, J and Todorov, I (2012). The Research Software Engineer. Digital Research 2012, 

  13. Parr, C (2013). Save your work – give software engineers a career track. Times Higher Education, August 15 2013 

  14. Gil, Y David, CH Demir, I Essawy, BT Fulweiler, RW Goodall, JL et al. (2016). Towards the Geoscience Paper of the Future: Best Practices for Documenting and Sharing Research from Data to Software to Provenance. Earth and Space Science,  

  15. Duffy, CJ, David, C, Peckham, S, Venayagamoorthy, K and Gil, Y (). Geoscience Papers of the Future: An Introduction to the Special Issue. Earth and Space Science, [In press], Accessible from: 

  16. Nanthaamornphong, A and Carver, JC (2015). Test-Driven Development in scientific software: a survey. Software Quality Journal, : 1–30, DOI: 

  17. Heaton, D and Carver, JC (2015). Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology 67: 207–219, DOI: Available from: 

  18. Basili, VR Carver, JC Cruzes, D Hochstein, LM Hollingsworth, JK Shull, F et al. (2008). Understanding the High-Performance-Computing Community: A Software Engineer’s Perspective. IEEE Software July 200825(4): 29–36, DOI: 

  19. Carver, JC, Kendall, RP, Squires, SE and Post, DE (2007). Software Development Environments for Scientific and Engineering Software: A Series of Case Studies. 29th International Conference on Software Engineering (ICSE’07). : 550–559, DOI: 

  20. Sempervirens ().  Accessed: 2015-11-07. 

  21. Allen, A and Schmidt, J (2015). Looking before leaping: Creating a software registry. Journal of Open Research Software 3(1) e15. 

  22. Ahalt, S Berriman, B Brown, M Carver, J Chue Hong, N Fish, A et al. (2015). Toward a Framework for Evaluating Software Success: A Proposed First Step. Computational Science and Engineering Software Sustainability and Productivity Challenges (CSESSP) Workshop, Available from: 

  23. Heroux, MA and Willenbring, JM (2009). Barely sufficient software engineering: 10 practices to improve your CSE software. Software Engineering for Computational Science and Engineering, 2009. SECSE ’09. ICSE Workshop: 15–21.  

  24. Blatt, M (2013). DUNE as an Example of Sustainable Open Source Scientific Software Development.  arXiv; 1309.1783. 

  25. Ahern, S Brugger, E Whitlock, B Meredith, JS Biagas, K Miller, MC et al. (2013). VisIt: Experiences with Sustainable Software.  arXiv; 1309.1796. 

  26. van Vliet, H (2008). Software Engineering: Principles and Practice In: 3rd ed. Wiley Publishing.  

  27. Merali, Z (2010). Computational science: … Error … why scientific programming does not compute. Nature 467: 775–777, DOI: 

  28. Hettrick, S et al. (2014). UK Research Software Survey 2014. DOI: 

  29. Becker, C Betz, S Chitchyan, R Duboc, L Easterbrook, SM Penzenstadler, B et al. (2016). Requirements: The Key to Sustainability In: Software. IEEE, Jan 201633(1)pp. 56–65, DOI: 

  30. Becker, C Chitchyan, R Duboc, L Easterbrook, S Penzenstadler, B Seyff, N et al. (2015). Sustainability Design and Software: The Karlskrona Manifesto. Proc. 2015 Int’l Conf. Software Eng. (ICSE’15). DOI: 

  31. Tate, K (2005). Sustainable Software Development: An Agile Perspective In: Addison-Wesley Professional.  

  32. Bourque, P and Fairley, RE (2014). SWEBOK, version 3.0: Guide to the Software Engineering Body of Knowledge In: IEEE Computer Society Press.  

  33. Working towards Sustainable Software for Science: Practice Experiences ().  Accessed: 2015-12-03. 

  34. International Workshop on Software Engineering for High Performance Computing in Computational Science Engineering ().  Accessed: 2015-12-03. 

  35. Workshop on Software Engineering for Sustainable Systems ().  Accessed: 2015-12-03. 

  36. Entertainment Identifier Registry ().  Accessed: 2015-10-28. 

  37. Katz, DS (2014). Transitive Credit as a Means to Address Social and Technological Concerns Stemming from Citation and Attribution of Digital Products. Journal of Open Research Software Sep 20142(1)DOI: e20. 

  38. Katz, DS and Smith, AM (2015). Transitive Credit and JSON-LD. Journal of Open Research Software 3(1) e7. 

  39. Mayes, AC, zee-moz, Collins, A, Niemeyer, K and Jabbari, A (2015). Leadership-Training: “Working Open” Guide – WSSSPE3 version. DOI: 

  40. Bacon, J (2009). The Art of Community. Building the New Age of Participation.  

  41. Howison, J (2015). Sustaining scientific infrastructures: transitioning from grants to peer production (work-in-progress). iConference 2015 Proceedings. 

  42. Geels, FW and Schot, J (2007). Typology of sociotechnical transition pathways. Research Policy 36(3): 399–417, DOI: 

  43. WSSSPE3 Software Credit Working Group (2015). WSSSPE3 Software Credit Working Group GitHub Issues.  Accessed: 2015-10-1. 

  44. WSSSPE3 Software Credit Working Group (2015). WSSSPE3 Software Credit Working Group Collaborative Notes.  Accessed: 2015-10-1. 

  45. CASRAI (). Project Credit.  Accessed: 2015-03-31. 

  46. Mayes, AC (2015). Contributorship Badges.  Accessed: 2015-10-26. 

  47. Open Researcher Contributor ID (ORCID) ().  Accessed: 2015-03-31. 

  48. McCall, JG Al-Hasani, R Siuda, ER Hong, DY Norris, AJ Ford, CP et al. (2015). CRH Engagement of the Locus Coeruleus Noradrenergic System Mediates Stress-Induced Anxiety. Neuron Aug 201587(3): 605–620, DOI: 

  49. Lin, IC, Okun, M, Carandini, M and Harris, KD (2015). The Nature of Shared Cortical Variability. Neuron Aug 201587(3): 644–656, DOI: 

  50. Howison, J and Bullard, J (2015). Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. Journal of the Association for Information Science and Technology, DOI: In press. 

  51. Brin, S and Page, L (1998). The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems 30(1–7): 107–117.  

  52. SIG SP (2015). Astronomy software citation examples and ideas.  Accessed: 2015-10-29. 

  53. Data Citation Synthesis Group (2014). Joint Declaration of Data Citation Principles In: Martone, M ed.  San Diego, CA: FORCE11. Accessed: 2015-10-28. 

  54. Zenodo ().  Accessed: 2015-10-28. 

  55. Figshare ().  Accessed: 2014-02-03. 

  56. IPython (2015). A gallery of interesting IPython Notebooks.  Accessed: 2015-10-28. 

comments powered by Disqus