Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)

Daniel S. Katz; Kyle E. Niemeyer; Sandra Gesing; Lorraine Hwang; Wolfgang Bangerth; Simon Hettrick; Ray Idaszak; Jean Salac; Neil Chue Hong; Santiago Núñez-Corrales; Alice Allen; R. Stuart Geiger; Jonah Miller; Emily Chen; Anshu Dubey; Patricia Lago

Issues in Research Software

Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)

Authors

Daniel S. Katz
Kyle E. Niemeyer
Sandra Gesing
Lorraine Hwang
Wolfgang Bangerth
Simon Hettrick
Ray Idaszak
Jean Salac
Neil Chue Hong
Santiago Núñez-Corrales
Alice Allen
R. Stuart Geiger
Jonah Miller
Emily Chen
Anshu Dubey
Patricia Lago

Abstract

This article summarizes motivations, organization, and activities of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). The WSSSPE series promotes sustainable research software by positively impacting principles and best practices, careers, learning, and credit. This article discusses the code of conduct; the mission and vision statements that were drafted at the workshop and finalized shortly after it; the keynote and idea papers, position papers, experience papers, demos, and lightning talks presented during the workshop; and a panel discussion on best practices. The main part of the article discusses the set of working groups that formed during the meeting, along with contact information for readers who may want to join a group. Finally, it discusses a survey of the workshop attendees.

Keywords:

Year: 2018

Volume: 6 Issue: 1

Page/Article: 10

DOI: 10.5334/jors.184

Submitted on Jun 18, 2017

Accepted on Jan 25, 2018

Published on Feb 15, 2018

Peer Reviewed

CC BY 4.0

1 Introduction

The Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4) was held over 2 1/2 days on 12–14 September 2016 in Manchester, England, with 68 attendees. This location and date was selected so that WSSSPE4 immediately preceded the First Research Software Engineers (RSE) Conference. Previous WSSSPE events, all of which were in the US, were two general, presentation-focused, one-day workshops held with the SC13 and SC14 conferences [, , , ], two half-day workshops held with SciPy 2014 and 2015 that contained presentations about specific sustainable Python software packages, and a 1-and-1/2-day workshop that included teams that self-assembled and discussed focused software sustainability topics []. Based on the work done at these previous WSSSPE meetings, WSSSPE4 was aimed at producing working groups that would better continue working after the workshop ended.

Specific topics that were discussed in previous meetings and suggested for WSSSPE4 included:

Development and Community: best practices for developing sustainable software; models for funding specialist expertise in software collaborations; software tools that aid sustainability; academia/industry interaction; refactoring/improving legacy scientific software; engineering design for sustainable software; metrics for the success of scientific software; and adaptation of mainstream software practices for scientific software.
Professionalization: career paths; RSE as a brand; RSE outside of the UK or Europe; and Increase incentives in publishing, funding and promotion for better software.
Training: training for developing sustainable software; and curriculum for software sustainability.
Credit: making the existing credit and citation ecosystem work better for software; future credit and citation ecosystem; software contributions as a part of tenure review; case studies of receiving credit for software contributions; and awards and recognition that encourage sustainable software.
Software publishing: journals and alternative venues for publishing software; and review processes for published software.
Software discoverability/reuse: proposals and case studies.
Reproducibility and testing: reproducibility in conferences and journals; and best practices for code testing and code review.

WSSSPE4 included multiple mechanisms for participation and encouraged team building around solutions. It strongly encouraged participation of early-career scientists, postdoctoral researchers, graduate students, early-stage researchers, and those from underrepresented groups, with funds provided to the conference organizers by the National Science Foundation (NSF), the Gordon and Betty Moore Foundation, the Alfred P. Sloan Foundation, and the Software Sustainability Institute (SSI) to support the travel of potential participants who would not otherwise be able to attend the workshop. These funds allowed 29 additional attendees (of 44 applicants), including 11 students and five early-career researchers. WSSSPE4 also included two professional event organizers/facilitators who helped plan the workshop agenda, and who engaged with participants during the workshop.

This paper is a summary of a longer report [] on the workshop and the submitted materials. The remainder of the paper includes the call for papers (§2); the WSSSPE Code of Conduct (§3) and mission and vision (§4); a set of presentations that included a keynote, papers, lighting talks, and a panel (§5); and the activities of a set of working groups (§6). One full day of the workshop was spent with participants in the working groups, which occurred in parallel with each other. And each of the working groups left with a plan for how they could move forward. This report also highlights an attendee survey (§7) before concluding (§8).

2 Call for participation

Post-WSSSPE3 feedback said that attendees had two distinct motivations for attending: to make a better future for research software, and to immediately do better research software development. Thus, WSSSPE4 was planned as two tracks:

Track 1 – Building a sustainable future for open-use research software: defining a vision of the future of open-use research software, and in the workshop, initiating the activities that are needed to get there.

Track 2 – Practices & experiences in sustainable scientific software: improving the quality of today’s research software and the experiences of its developers by sharing best practices and experiences.

The call for participation requested: idea papers, implementable proposals related to Track 1; position papers, longer, not previously published papers discussing what we can do to improve sustainable scientific software in the short term; experience papers, longer papers that discuss current experiences and how they have been used to improve the quality of today’s research software and/or the experiences of its developers; demos, brief papers describing a tool or process that would be demonstrated that improves the quality of today’s research software and/or the experiences of its developers; and extended abstracts describing lightning talks, where each author could make a brief statement about work that either had been done or was needed.

Submissions to WSSSPE4 comprised 19 lightning talks, 4 idea papers, 3 position paper, 5 experience papers, and 3 demos. Most submissions were accepted, since the goal of WSSSPE is always to take in as many relevant inputs as possible, and to encourage their authors to participate in sharing and implementing their ideas. Two submitted lightning talks and one submitted experience paper were rejected, while two more submitted experience papers were accepted as lightning talks. The papers and lightning talks have been published as a volume in the CEUR Workshop Proceedings [].

3 Code of Conduct

WSSSPE4 introduced a Code of Conduct (CoC), which was conceived for the workshop and as guidance for the community of scientists that WSSSPE supports, including their personal and online interactions (e.g., on Twitter, in email lists, in Slack). This CoC is based on the FORCE11 conference CoC [], which is in turn based on the Code4 Lib CoC []. In introducing the CoC, we asked participants to agree to the following main guidelines:

WSSSPE events are community events intended for networking and collaboration as well as learning. We value the participation of every member of the community and want all attendees to have an enjoyable and fulfilling experience. Accordingly, all attendees are expected to show respect and courtesy to other attendees throughout the event and in interactions online associated with the event.

The WSSSPE event organizers are dedicated to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, age, sexual orientation, disability, physical appearance, body size, race, ethnicity, religion (or lack thereof), technology choices, or other group status.

To make clear what is expected, everyone taking part in WSSSPE events and discussions—speakers, helpers, organizers, and participants—is required to conform to the Code of Conduct.

The CoC was discussed at the beginning of WSSSPE4, with the CoC subcommittee and a general email address for reporting concerns or incidents, or asking questions. One concern was raised after the first half day, and we changed the workshop to address this concern.

4 Mission and vision

Going into WSSSPE4, WSSSPE had no formal mission or vision statement. The organizers developed a strawhorse, which was presented to the participants early in the meeting. The presentation included guidelines as well as examples from other similar communities. The guidelines were that in general, an organization’s mission should stand the test of time and state what the organization does; its vision should imagine what the world would look like if the organization is successful; and from which, focus areas could be used to establish the scope of the organization along with its goals.

The participants were invited to comment, and the mission and vision statements, were revised based on this feedback. The committee added focus areas to clarify the organization’s breadth and to keep the mission and vision simple and long-lasting. Comments after the close of the meeting were also incorporated into the statements. A final draft was put on GitHub and advertised via the WSSSPE mailing list, Facebook group, and Twitter. After two weeks without suggested changes, the final statements were added to the WSSSPE website.

5 Presentations

The keynote was given by Patricia Lago and entitled “The legacy of unsustainable software”. Sustainability is broadly associated with natural ecologic systems. When we translate the notion of sustainability to software solutions, however, we often confuse impact in a certain point in time with balanced and durable effects. In addition, software sustainability adds a fourth dimension to environmental, social and economic aspects: technical sustainability, and hence higher complexity []. Lago’s keynote discussed the challenges (and some related ongoing research) of combining technical and environmental sustainability, providing a complementary angle to the workshop discussions.

WSSSPE4 included the presentation of 12 10-minute talks (based on four idea papers, three position papers, two experience papers, and three demos) that addressed a wide range of topics around sustainability for software in science. These talks covered three main areas. The first area was the academic environment, with four papers that address diverse aspects ranging from incentives for quality software to advocating a professional society for research software to roles and degrees for research software engineers [, , , ]. The second area was concerned with characteristics and needs of research software and included five papers [, , , , ]. The third area focused on elucidating lessons learned from or visions for use cases of scientific software and communities working with and/or on a software package, with three papers [, , ].

In addition, 19 lightning talks were presented at WSSSPE4 [, , , , , , , , , , , , , , , , , , ].

The final WSSSPE4 presentation was a panel of five experts with different perspectives on “Best Practices for Scientific Software.” These were: Alice Allen, Editor, Astrophysics Source Code Library, who brought an understanding of the difficulties of organizing a community and curating their software; Mike Croucher from the University of Sheffield and Rob Haines from the University of Manchester who are both Research Software Engineers with decades of experience of writing code for researchers; Patricia Lago from Vrije Universiteit Amsterdam, who brought a fresh perspective on software sustainability from the point of view of its impact on society and business; and Karthik Ram from the Berkeley Institute for Data Science, who brought his perspective on the practicalities of using scientific software to conduct his research as a data scientist.

6 Working groups

After most of the lightning talks and other presentations, WSSSPE used three areas of a large room to let attendees use sticky flip charts and sticky notes on the walls to suggest a vision on any aspect of the work, a gap or challenge, or a project idea. Next, attendees organically formed groups around the flip charts, and 12 groups emerged. Summaries of each group’s activities follow; for more details about these discussions, see []. There is a Slack team for WSSSPE [] that one can join at https://wssspe.signup.team/; some working groups also created channels within this team.

6.1 Verifying best practices and metrics for sustainable research software

Many open source research software projects document their best practices that contribute to the sustainability of the software as well as the metrics that define their research software project’s success. This group will take the outputs of the WSSSPE efforts to identify best development practices and to identify metrics for sustainable research software, and cross-reference them with current open source research software that successfully uses modern software engineering. This will allow the group to identify gaps on both sides and to hypothesize how successful projects can be further improved. One can join this group or obtain more information about it by sending an email to the verifying best practices & metrics for sustainable research software group: <wssspe4-verify-best-practices@googlegroups.com>.

6.2 Software Sustainability Alliance

The Software Sustainability Alliance working group aims to establish an alliance between organizations interested in improving the sustainability of research software at a larger scale than their local university or community, and to distinguish between this group and WSSSPE. Such an alliance would ease the collaboration between member organizations to improve the sharing of expertise, resources and best practices. It is currently seeking feedback on potential activities and member organizations, as well as the aims and scope of this alliance. Currently, point-to-point collaboration exists between organizations, but this inadvertently results in competition or redundancy within the sustainable software community. An alliance of software sustainability organizations would ease inter-organization collaboration and the promotion of software sustainability, andwould improve the pooling of competencies and the sharing of expertise. If interested, one can visit http://softwaresustainability.org/ or email Neil Chue Hong (<N.ChueHong@software.ac.uk>) and Jean Salac (<salac@uchicago.edu>)

6.3 Scientific Software Prototyping Infrastructure (S2PI)

There is a productivity bottleneck—yet to be solved—in HPC from the human perspective, first identified in the 1980s []. Only a fraction of domain-specific scientists use simulations, mostly due to perceptions about the difficulty of using and developing such tools. XSEDE and other resources have become a gateway for successfully increasing the basis of HPC scientific users by facilitating their access to infrastructure and tools [], but the development challenge remains. This group deems the ability to rapidly construct software artifacts that can be trusted in terms of the methods from their design up, easily discarded when wrong and extended when right at low human and computational cost to be one aspect of scientific software sustainability. The group’s objective is development of a prototype tool that allows domain scientists to generate simulation prototypes through a simple, yet expressive declarative strongly-typed programming language close to equational expressions. The result of that specification is both an executable artifact as well as a specification for scientific programmers to later flesh out completely and adapt to particular infrastructures. To join this project, one can visit the GitHub repository (https://github.com/nunezco2/S2PI) or join #wg-sci-soft-proto in the WSSSPE Slack.

6.4 CodeMeta

Research software is often not shared; that which is shared may not have much metadata associated with it, and that which does exist often does not travel further than the website on which the software resides. The CodeMeta project wants to incentivize software developers to release their software, encourage the development of metadata for it, enable credit assignment and citation of research software, increase its discoverability, more easily track dependencies, and enable reuse of software metadata, all goals that WSSSPE attendees have great interest in supporting. CodeMeta seeks in part to create a “Rosetta Stone” for software metadata to facilitate retaining such metadata between repositories, services, registries, indexers, publishers, citation managers, and other entities that create, ingest, use, and/or store metadata about software. The project also wants to establish a JSON-LD schema as a tool for making metadata machine-readable []. While at WSSSPE4, the group greatly expanded the CodeMeta project README file to include a description of the project geared to those with little or no prior knowledge of the project, a list of contributors, information on how one can get involved, a brief project history and who is managing the project, and links to additional information. Though a Google group mailing list has been established for the working group, the easiest way to engage with the CodeMeta project is through its Github repository: https://github.com/codemeta/codemeta.

6.5 White paper on developing sustainable software

Many diverse aspects and dimensions (e.g., economic, technical, environmental, social) of developing sustainable software can be investigated. This group aims to write white papers that focus on scientific environments and their implications, targeted at developers and project managers of scientific software. Given the complexity of this field, it is important to select a subset of sustainability aspects for the white papers. The idea is to create a series of papers instead of trying to tackle all important topics in one paper. While there are already a few papers available on best practices and sustainability of scientific software in general, the group’s goal is to create a series of papers that lead to consensus in the community, tackle many diverse aspects, and stay up-to-date with new trends. The GitHub repository for the white paper can be found at https://github.com/WSSSPE/WG-Best-Practices. For more information and requests, join the #wg-best-practices channel in the WSSSPE Slack.

There is more and more academic research being done on topics related to software sustainability, including work on software engineering practices and management of open source projects. However, academic research in general is often siloed for many reasons, and work on topics relevant to software sustainability is no exception. This working group is motivated by the goal of building better connections between academic researchers who are studying topics in or relating to software sustainability with practitioners, managers, and administrators who are working in the area of software sustainability. Furthermore, the group also recognizes that academic social science and research software engineering are not monolithic, and there is a need to connect people who care about research-driven best practices inside of these two domains with each other as well. The working group plans to briefly survey existing literature to get a better sense of the academic research landscape, facilitate some initial dialog between academic researchers and practitioners at WSSSPE4, then identify needed actions that would be mutually beneficial to researchers and practitioners. To join the group, one can join its mailing list at: https://groups.google.com/forum/#!forum/researchsoftwarestudies.

6.7 Software best practices for undergraduates

This working group was motivated by the perceived prevalence of so-called “hidden code” in scientific communities: code written by individual researchers in an unsustainable way that is never shared with the larger community. The participants hypothesized that the best strategy is to catch researchers while they are still in training and teach them software best practices. Therefore, the working group’s goal is to develop courses on software best practices aimed at undergraduate students studying domain science. The program might be similar to Software and Data Carpentry workshops but focused for domain scientists.

The development of a successful curriculum relies on the expertise of software engineers to describe the best practices, domain experts to describe model problems and workflow, and instructors to formulate the pedagogy. Ideally these people will be brought together for a short workshop or hackathon with the goal of drafting the curriculum. Anyone interested in contributing can join the Google Group/mailing list (https://groups.google.com/forum/#!forum/wssspe4-software-best-practices-for-undergraduates), join the #wg-undergraduatecourse channel in the WSSSPEs Slack, or ask to join the WSSSPE4-undergraduate-course organization (https://github.com/orgs/WSSSPE4-undergraduate-course).

6.8 Meaningful metrics for sustainable software

This group aims to increase the visibility of the quality of scientific software, facilitate the reusability of scientific software, and promote the best software practices by standardizing metrics via interviews with scientific software developers. This working group believes improving the current software metrics system will increase software sustainability. Currently, there are inefficiencies regarding software duplication, sustainability, and selection, as well as others, within the scientific software community. In order to address these inefficiencies, the group aims to create a goal-oriented method to collecting productive metrics by focusing on the developer side of software. The group plans to survey and interview scientific software developers to form metrics from the goals they have for their software, then to publicize the results for more information, one can contact Emily Chen at <echen35@illinois.edu>.

6.9 Coordinating access to continuous integration (CI) for research software

Each developer of software with uncommon needs (hardware, software, libraries, data sets), non-public code, or tests that exceed time limits for free plans must acquire, setup, and maintain their own continuous integration systems because their needs make them ineligible for popular free services such as Travis CI. For example, software groups that develop BIOUNO, CI4SI, or GROMACS have done this. This group is interested in reducing the burden of different projects having to build and maintain their own continuous integration systems (when publicly available CI are not a fit), by coordinating and sharing this burden across multiple projects. However, there are a lot of open issues such as how can the group learn from existing similar work without reinventing the wheel. Some possible goals include to acquire additional hardware such as GPUs, Xeon PHI, FPGAs and add/share them to, e.g., Debian’s testing infrastructure or to extend Debian’s scope to include published but not mature software. For more information, see https://groups.google.com/forum/#!forum/continuous-integration-for-research-software.

6.10 Software engineering processes tailored for research software

This working group is concerned with identifying processes that are not adequately covered by general software engineering, starting with verification and testing. Computational science and engineering applications have many moving parts that need to interoperate with one another. The accuracy and reliability of results produced by scientific software depends not only on the individual components behaving correctly, but also on the validity of their interactions. A big challenge is that those who understand the importance of regular testing do not often find much help from software engineering literature. There is a significant gap between the testing gospel and its applicability to computational science. This gap leads to frustration and abandonment of the good with the bad. This working group will (1) conduct a literature survey to gauge the extent of awareness of the issue in general, (2) generate content useful for the community where needed, and (3) curate the collected and added content for the use of the community. To join this effort, one can use the #wg-testing-in-science channel in the WSSSPE Slack. Additionally, a git repository (https://github.com/WSSSPE/WG-Best-Practices.git) exists for contributing content and reference to, and curation of the existing literature on this topic.

6.11 Open research index

The aim of this group is to investigate the building of an index of research products in an open sustainable manner. Its goal is not to eliminate commercial products, but to build on what is there and provide data and services that are missing. The Open Research Index would take in all research products (papers, software, datasets, workflows, etc.) from their publishers and recorders (journals, societies, domain repositories, government [open access] repositories, preprint servers, general repositories [e.g., figshare, zenodo]) and other services (CrossRef, ORCID). Each product would list authors and citations and allow people to search the resulting network. Users would also be able to interact with their own record and edit it, as Google Scholar allows. The group’s plans are relatively simple to express, though quite complex to undertake; the first is to identify a leader. With a leader and feasible plan, the group would then obtain resources, and then build the index. Because Google and others provide some similar services today, though these services (and the underlying data) could be removed at any time and the community cannot build new services, these companies would ideally be involved in the activities of the group. Alternatively or additionally, the group could build a mailing list for us and discuss further, depending on how receptive others are to this idea. At WSSSPE4, there seemed to be enough interest to do this, so the group set up a #wg-open-research-idx channel within the WSSSPE team Slack for its members and any future members.

6.12 Letters of evaluation for computational scientists

Scientists working on scientific software are often located in disciplinary departments, depending on whether their software originates from the mathematical, physical, chemical, or other disciplines. As a consequence, they are frequently outside the core areas of their science, and their contributions are typically to both the research activity their software enables, as well as on algorithm and implementation aspects. This presents issues when letters of evaluation for hiring, tenure, and promotion do not specifically cover how this is relevant to the discipline. Consequently, letter writers need to be aware of how their letters will be read by committees. It also requires that committees be aware that such letters often look different and may provide a different perspective of how a candidate’s achievements should be assessed. This group has two important strategies that build on each other: (i) raising awareness of the problem beyond just those who are affected by it, and (ii) providing letter writers, letter readers, and evaluating committees with guidance on what criteria are relevant in assessing scientific software authors. Ultimately, building a community large enough to affect change is important; one may contact Wolfgang Bangerth at <bangerth@colostate.edu> if interested in helping.

7 Attendee survey

At the end of WSSSPE4, all remaining participants completed an on-line survey. In general, the respondents were highly satisfied with the meeting and interested in continuing and participating in WSSSPE activities. The mix of topics and types of interactions—talks, panels and discussion—was well balanced with several indicating a desire for time for questions and answers following talks as well for discussions in general. Many respondents indicated that they will remain engaged in WSSSPE working group initiatives. There was weaker demand for a professional organization encompassing WSSSPE interests, with over half interested in joining and a large percentage willing to consider the idea. Respondents were grateful for the opportunity to network, explore new collaborations and for travel support to the conference.

The survey results indicate that future WSSSPE conferences should consider the balance of the attendees. WSSSPE4 had many first time and early career participants who would have benefited from a review of basic concepts and terminology including what it means for software to be sustainable and the roles of research software engineers. Additional emphasis and topics to explore would be the inclusion of more case studies, focus on the decision making process in developing and using software, deeper dives into selected topics, tutorials, software training within and outside of STEM, lowering the barriers to implementing best practices in software development, and progress towards executing WSSSPE’s vision.

8 Conclusions

In WSSSPE4, we heard about a number of interesting projects and ideas, and used those ideas to create and form working groups, intended to start at the workshop and then continue afterwards, to address challenges that arose from the workshop ideas. This workshop has reinforced the lesson from WSSSPE3 that it is relatively easy to get motivated people to attend a meeting and productively spend their time there both doing work and planning more work, but it is very hard to get that additional work after the meeting to take place. The main problem seems to be one of time. Once the attendees have agree to spend their time at the workshop, they put their energy into doing so productively, but they have not really committed themselves to anything more than this, so their energy and effort trails off, as all of their other commitments (particularly, those they are funded to do as part of their jobs) come back to the fore. Without a process to resolve this concern, the utility of further multi-day WSSSPE workshops is unclear, at least to start new activities. On a more positive note, WSSSPE4 was successful as a gathering place to discuss scientific software sustainability, and for groups that are already in place or that can be composed of related funded activities to meet. Future WSSSPE meetings will likely combine these functions: a meeting place for like-minded individuals, a place to share experiences and lessons learned, and a place for both existing groups and sets of related projects to meet in person. Other parts of the WSSSPE infrastructure (the email list and the Slack) will be used both for general discussions, as well as for groups that choose to focus on specific problems.

After WSSSPE4, we also have learned enough to summarize the areas of active work in sustainable scientific software, based on the WSSSPE4 mission and vision statements, the WSSSPE4 call for participation (based on topics in previous WSSSPE meetings), and the working groups formed in WSSSPE4. Following the example of the recently funded US project, “Conceptualizing a US Research Software Sustainability Institute (URSSI),” the high-level version of these areas is: functioning of 1) the individual and team, 2) the research software, and 3) the research field itself, with six specific topics, as shown in Table 1.

Table 1

Scientific Software Sustainability Research Topics and Areas.

Topics\Areas	Individual and team	Research software	Research field

Principles and Best Practices	✔	✔	✔
Careers	✔
Learning	✔
Credit	✔		✔
Organization and community	✔		✔
Common Infrastructure		✔

Notes

http://wssspe.researchcomputing.org.uk/wssspe4/.
The first WSSSPE workshop was named “Working towards Sustainable Software for Science: Practice and Experiences,” which remains the meaning of the WSSSPE group, but the workshops after that were named “Workshop on Sustainable Software for Science: Practice and Experiences.” Together these reflect that WSSSPE is both a community and a set of workshops.
http://wssspe.researchcomputing.org.uk/wssspe1-1/, http://wssspe.researchcomputing.org.uk/wssspe2-1/.
http://wssspe.researchcomputing.org.uk/wssspe4/code-of-conduct/.
http://wssspe.researchcomputing.org.uk/about-wssspe/.
http://codemeta.github.io/.
The extensive use of such letters, and the problems that are associated with it, may be an issue specific to the United States.
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1743188.

Acknowledgements

This material is based upon work supported by the National Science Foundation (ACI-1648293, ACI-1547611), by the Gordon and Betty Moore Foundation (GBMF#5620), and by the Alfred P. Sloan Foundation (G-2016–7214).

Competing Interests

The authors have no competing interests to declare.

References

Katz, D S, Allen, G, Chue Hong, N, Parashar, M and Proctor, D 2013 First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE): Submission and Peer-Review Process, and Results. arXiv, 1311–3523. Available from: http://arxiv.org/abs/1311.3523.
Katz, DS, Choi, SCT, Lapp, H, Maheshwari, K, Löffler, F, Turk, M, et al. 2014 Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1). Journal of Open Research Software, 2(1). DOI: https://doi.org/10.5334/jors.an
Katz, DS, Allen, G, Chue Hong, N, Cranston, K, Parashar, M, Proctor, D, et al. 2014 Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2): Submission, Peer-Review and Sorting Process, and Results. arXiv, 1411–3464. Available from: http://arxiv.org/abs/1411.3464.
Katz, D S, Choi, S T, Wilkins-Diehr, N, Chue Hong, N, Venters, C C, Howison, J, et al. 2016 Report on the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). Journal of Open Research Software, 4(1). e7. DOI: https://doi.org/10.5334/jors.85
Katz, D S, Choi, S T, Niemeyer, K E, Hetherington, J, Löffler, F, Gunter, D, et al. 2016 Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3). Journal of Open Research Software, 4(1). e37. DOI: https://doi.org/10.5334/jors.118
Katz, D S, Niemeyer, K E, Gesing, S, Hwang, L, Bangerth, W, Hettrick, S, et al. 2017 Report on the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). arXiv, 1705–02607. Available from: http://arxiv.org/abs/1705.02607.
Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.) 2016 Proceedings of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences, (WSSSPE4). Aachen. Urn:nbn:de: 0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/.
Ilik, V, Thaney, K, Katz, D S, others. 2016 FORCE11 Conference Code of Conduct. Accessed: 2016-09-15. https://www.force11.org/meetings/force2016/code-conduct.
Code4 Lib 2015 Code Of Conduct 4 Lib, Accessed: 2016-09-15. Git hash 4270594fdbe28eeeb 45256ea6304c8bd800e6a86. https://github.com/code4lib/antiharassment-policy/blob/master/code_of_conduct.md.
Lago, P, Koçak, S A, Crnkovic, I and Penzenstadler, B 2015 Framing sustainability as a property of software quality. Communications of the ACM, 58(10). 70–78. DOI: https://doi.org/10.1145/2714560
Heroux, M 2016 Sustainable & Productive: Improving incentives for quality software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings, Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_10.pdf.
Allen, G 2016 Establishing a professional society for research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings, Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceurws.org/Vol-1686/WSSSPE4_paper_24.pdf.
Philippe, O, Hettrick, S and Chue Hong, N A 2016 A survey of Research Software Engineers in the UK. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_19.pdf.
Gwilliams, C 2016 Using industrial engagement to create and develop research ties within academia. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_1.pdf.
Dubey, A and Riley, K 2016 Software engineering and community codes track in ATPESC. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_17.pdf.
Chue Hong, N 2016 Why do we need to compare research software, and how should we do it? In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings, Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_29.pdf.
Dubey, A and McInnes, L C 2016 The lifecycle of software for scientific simulation. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_16.pdf.
Queiroz, F and Spitz, R 2016 Collaborative gamification design for scientific software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_6.pdf.
Childers, B, Davidson, J, Graves, W, Rous, B and Wilkinson, D 2016 Active curation of artifacts is changing the way digital libraries will operate. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_8.pdf.
Ganguly, D, Garrison, W, Wilkinson, D, Childers, B, Lee, A and Mosse, D 2016 Composing, reproducing, and sharing simulations. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_33.pdf.
Shende, S and Malony, A 2016 Using TAU for performance evaluation of scientific software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_12.pdf.
Sallai, J, Iacovella, C, Klein, C and Ma, T 2016 Development of a software framework for formalizing forcefield atomtyping for molecular simulation. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_35.pdf.
Sufi, S 2016 The Software Sustainability Institute Fellowship programme; supporting the social side of research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_27.pdf.
Gesing, S, Dahan, M, Hayden, LB, Lawrence, K, Marru, S, Pierce, M, et al. 2016 The Science Gateways Community Institute – Supporting communities to achieve sustainability for their science gateways. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_32.pdf.
Ram, K, Ross, N and Chamberlain, S 2016 A model for peer review and onboarding research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, MR, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_13.pdf.
Löffler, F and Brandt, S 2016 A vision of computing in 10 + years. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_9.pdf.
Katz, D S, Niemeyer, K E, Smith, A M and FORCE11 Software Citation Working Group 2016. Software citation: Process, principles, and implementation. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_31.pdf.
Idaszak, R, Tarboton, DG, Yi, H, Stealey, M, Dash, P, Couch, A, et al. 2016 HydroShare: A case study in software engineering best practices and culture change for developing sustainable community software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_5.pdf.
Hwang, L, Bangerth, W, Heister, T and Kellogg, L 2016 ASPECT: Hackathons as an example of sustaining an open source community. In: Allen, G, Carve, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_7.pdf.
Goble, C 2016 A simple profiling framework for software user-producer maturity review. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_2.pdf.
Druskat, S 2016 A proposal for the measurement and documentation of research software sustainability in interactive metadata repositories. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_20.pdf.
Contrastin, M, Danish, M, Orchard, D and Rice, A 2016 Supporting software sustainability with lightweight specifications. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_23.pdf.
Allen, A, Aragon, C, Becker, C, Carver, J, Chis, A, Combemale, B, et al. 2016 I solemnly pledge: A manifesto for personal responsibility in the engineering of academic software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_15.pdf.
Chue Hong, N 2016 Making it easier to understand research software impact. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_30.pdf.
van Hage, W R, Maassen, J and van Nieuwpoort, R 2016 Software impact measurement at the Netherlands eScience Center. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_4.pdf.
Allen, G, Chen, E, Idaszak, R and Katz, D S 2016 Report on software metrics for research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_26.pdf.
Seidel, E L and Allen, G 2016 Bringing techniques from software engineering into scientific software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_28.pdf.
Emsley, I and De Roure, D 2016 Sustaining the social: Connecting the lives of Drupal community. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_34.pdf.
Dongarra, J, Hammarling, S, Higham, N J, Relton, S D, Valero-Lara, P and Zounon, M 2016 Creating a standardised set of batched BLAS routines. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_3.pdf.
Breitwieser, L, Bauer, R, Meglio, A D, Johard, L, Kaiser, M, Manca, M, et al. 2016 The BioDynaMo Project. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_14.pdf.
Aldabjan, A, Haines, R and Jay, C 2016 How should we measure the relationship between code quality and software sustainability? In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_22.pdf.
WSSSPE Organization 2016 WSSSPE Slack. Accessed: 2016-12-11. https://wssspe.signup.team/.
Barstow, D, Duffey, R, Smoliar, S and Vestal, S 1982 An automatic programming system to support an experimental science. In: Proceedings of the 6th international conference on Software engineering. IEEE Computer Society Press, 360–366.
Towns, J, Cockerill, T, Dahan, M, Foster, I, Gaither, K, Grimshaw, A, et al. 2014 XSEDE: accelerating scientific discovery. Computing in Science & Engineering, 16(5). 62–74. DOI: https://doi.org/10.1109/MCSE.2014.80
Jones, M B, Boettiger, C, Mayes, A C, Smith, A, Slaughter, P, Niemeyer, K, et al. 2016 CodeMeta: an exchange schema for software metadata. KNB Data Repository. DOI: https://doi.org/10.5063/SCHEMA/CODEMETA-1.0.

[B1] Katz, D S, Allen, G, Chue Hong, N, Parashar, M and Proctor, D 2013 First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE): Submission and Peer-Review Process, and Results. arXiv, 1311–3523. Available from: http://arxiv.org/abs/1311.3523.

[B2] Katz, DS, Choi, SCT, Lapp, H, Maheshwari, K, Löffler, F, Turk, M, et al. 2014 Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1). Journal of Open Research Software, 2(1). DOI: https://doi.org/10.5334/jors.an

[B3] Katz, DS, Allen, G, Chue Hong, N, Cranston, K, Parashar, M, Proctor, D, et al. 2014 Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2): Submission, Peer-Review and Sorting Process, and Results. arXiv, 1411–3464. Available from: http://arxiv.org/abs/1411.3464.

[B4] Katz, D S, Choi, S T, Wilkins-Diehr, N, Chue Hong, N, Venters, C C, Howison, J, et al. 2016 Report on the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). Journal of Open Research Software, 4(1). e7. DOI: https://doi.org/10.5334/jors.85

[B5] Katz, D S, Choi, S T, Niemeyer, K E, Hetherington, J, Löffler, F, Gunter, D, et al. 2016 Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3). Journal of Open Research Software, 4(1). e37. DOI: https://doi.org/10.5334/jors.118

[B6] Katz, D S, Niemeyer, K E, Gesing, S, Hwang, L, Bangerth, W, Hettrick, S, et al. 2017 Report on the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). arXiv, 1705–02607. Available from: http://arxiv.org/abs/1705.02607.

[B7] Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.) 2016 Proceedings of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences, (WSSSPE4). Aachen. Urn:nbn:de: 0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/.

[B8] Ilik, V, Thaney, K, Katz, D S, others. 2016 FORCE11 Conference Code of Conduct. Accessed: 2016-09-15. https://www.force11.org/meetings/force2016/code-conduct.

[B9] Code4 Lib 2015 Code Of Conduct 4 Lib, Accessed: 2016-09-15. Git hash 4270594fdbe28eeeb 45256ea6304c8bd800e6a86. https://github.com/code4lib/antiharassment-policy/blob/master/code_of_conduct.md.

[B10] Lago, P, Koçak, S A, Crnkovic, I and Penzenstadler, B 2015 Framing sustainability as a property of software quality. Communications of the ACM, 58(10). 70–78. DOI: https://doi.org/10.1145/2714560

[B11] Heroux, M 2016 Sustainable & Productive: Improving incentives for quality software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings, Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_10.pdf.

[B12] Allen, G 2016 Establishing a professional society for research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings, Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceurws.org/Vol-1686/WSSSPE4_paper_24.pdf.

[B13] Philippe, O, Hettrick, S and Chue Hong, N A 2016 A survey of Research Software Engineers in the UK. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_19.pdf.

[B14] Gwilliams, C 2016 Using industrial engagement to create and develop research ties within academia. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_1.pdf.

[B15] Dubey, A and Riley, K 2016 Software engineering and community codes track in ATPESC. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_17.pdf.

[B16] Chue Hong, N 2016 Why do we need to compare research software, and how should we do it? In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings, Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_29.pdf.

[B17] Dubey, A and McInnes, L C 2016 The lifecycle of software for scientific simulation. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_16.pdf.

[B18] Queiroz, F and Spitz, R 2016 Collaborative gamification design for scientific software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_6.pdf.

[B19] Childers, B, Davidson, J, Graves, W, Rous, B and Wilkinson, D 2016 Active curation of artifacts is changing the way digital libraries will operate. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_8.pdf.

[B20] Ganguly, D, Garrison, W, Wilkinson, D, Childers, B, Lee, A and Mosse, D 2016 Composing, reproducing, and sharing simulations. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_33.pdf.

[B21] Shende, S and Malony, A 2016 Using TAU for performance evaluation of scientific software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_12.pdf.

[B22] Sallai, J, Iacovella, C, Klein, C and Ma, T 2016 Development of a software framework for formalizing forcefield atomtyping for molecular simulation. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_35.pdf.

[B23] Sufi, S 2016 The Software Sustainability Institute Fellowship programme; supporting the social side of research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_27.pdf.

[B24] Gesing, S, Dahan, M, Hayden, LB, Lawrence, K, Marru, S, Pierce, M, et al. 2016 The Science Gateways Community Institute – Supporting communities to achieve sustainability for their science gateways. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_32.pdf.

[B25] Ram, K, Ross, N and Chamberlain, S 2016 A model for peer review and onboarding research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, MR, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_13.pdf.

[B26] Löffler, F and Brandt, S 2016 A vision of computing in 10 + years. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_9.pdf.

[B27] Katz, D S, Niemeyer, K E, Smith, A M and FORCE11 Software Citation Working Group 2016. Software citation: Process, principles, and implementation. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_31.pdf.

[B28] Idaszak, R, Tarboton, DG, Yi, H, Stealey, M, Dash, P, Couch, A, et al. 2016 HydroShare: A case study in software engineering best practices and culture change for developing sustainable community software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_5.pdf.

[B29] Hwang, L, Bangerth, W, Heister, T and Kellogg, L 2016 ASPECT: Hackathons as an example of sustaining an open source community. In: Allen, G, Carve, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_7.pdf.

[B30] Goble, C 2016 A simple profiling framework for software user-producer maturity review. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_2.pdf.

[B31] Druskat, S 2016 A proposal for the measurement and documentation of research software sustainability in interactive metadata repositories. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_20.pdf.

[B32] Contrastin, M, Danish, M, Orchard, D and Rice, A 2016 Supporting software sustainability with lightweight specifications. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_23.pdf.

[B33] Allen, A, Aragon, C, Becker, C, Carver, J, Chis, A, Combemale, B, et al. 2016 I solemnly pledge: A manifesto for personal responsibility in the engineering of academic software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_15.pdf.

[B34] Chue Hong, N 2016 Making it easier to understand research software impact. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_30.pdf.

[B35] van Hage, W R, Maassen, J and van Nieuwpoort, R 2016 Software impact measurement at the Netherlands eScience Center. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_4.pdf.

[B36] Allen, G, Chen, E, Idaszak, R and Katz, D S 2016 Report on software metrics for research software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_26.pdf.

[B37] Seidel, E L and Allen, G 2016 Bringing techniques from software engineering into scientific software. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_28.pdf.

[B38] Emsley, I and De Roure, D 2016 Sustaining the social: Connecting the lives of Drupal community. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_34.pdf.

[B39] Dongarra, J, Hammarling, S, Higham, N J, Relton, S D, Valero-Lara, P and Zounon, M 2016 Creating a standardised set of batched BLAS routines. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_3.pdf.

[B40] Breitwieser, L, Bauer, R, Meglio, A D, Johard, L, Kaiser, M, Manca, M, et al. 2016 The BioDynaMo Project. In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_14.pdf.

[B41] Aldabjan, A, Haines, R and Jay, C 2016 How should we measure the relationship between code quality and software sustainability? In: Allen, G, Carver, J, Choi, S C T, Crick, T, Crusoe, M R, Gesing, S, et al. (eds.), Workshop on Sustainable Software for Science: Practice and Experiences, No. 1686 in CEUR Workshop Proceedings. Aachen. Urn:nbn:de:0074-1686-8. Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_22.pdf.

[B42] WSSSPE Organization 2016 WSSSPE Slack. Accessed: 2016-12-11. https://wssspe.signup.team/.

[B43] Barstow, D, Duffey, R, Smoliar, S and Vestal, S 1982 An automatic programming system to support an experimental science. In: Proceedings of the 6th international conference on Software engineering. IEEE Computer Society Press, 360–366.

[B44] Towns, J, Cockerill, T, Dahan, M, Foster, I, Gaither, K, Grimshaw, A, et al. 2014 XSEDE: accelerating scientific discovery. Computing in Science & Engineering, 16(5). 62–74. DOI: https://doi.org/10.1109/MCSE.2014.80

[B45] Jones, M B, Boettiger, C, Mayes, A C, Smith, A, Slaughter, P, Niemeyer, K, et al. 2016 CodeMeta: an exchange schema for software metadata. KNB Data Repository. DOI: https://doi.org/10.5063/SCHEMA/CODEMETA-1.0.

Issues in Research Software

Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)

Abstract

1 Introduction

2 Call for participation

3 Code of Conduct

4 Mission and vision

5 Presentations

6 Working groups

6.1 Verifying best practices and metrics for sustainable research software

6.2 Software Sustainability Alliance

6.3 Scientific Software Prototyping Infrastructure (S2PI)

6.4 CodeMeta

6.5 White paper on developing sustainable software

6.6 Social science for scientific software

6.7 Software best practices for undergraduates

6.8 Meaningful metrics for sustainable software

6.9 Coordinating access to continuous integration (CI) for research software

6.10 Software engineering processes tailored for research software

6.11 Open research index

6.12 Letters of evaluation for computational scientists

7 Attendee survey

8 Conclusions

Notes

Acknowledgements

Competing Interests

References