5.1.1 Fit with related activities

The creation of such a “best practices” document will build upon the range of activities and topics discussed at WSSSPE3 and associated prior meetings. This working group will attempt to distill the emerging body of knowledge into this document. The large number of articles from the NSF-funded SI2 projects (SSE and SSI), “lightning talks,” “white papers,” and reports from different workshops have created a large if somewhat diffuse source for this report.

5.1.2 Discussion

Core questions that will need to be explored are in reproducibility, reliability, usability, extensibility, knowledge management, as well as continuity (transitions between people). Answers to these questions will guide the group to learn how a software tool becomes part of the core workflow of well-identified users (stakeholders) relating to tool success and hence sustainability. Ideas that may need to be explored include:

• Requirements engineering to create tools with immediate uptake;
• When should software “die”?
• Catering to disruptive developments in environment (e.g., new hardware, new methodology);
• Dimensions of sustainability: economic, technical, environmental and obsolescence.

Sustainability requires community participation in code development and/or a wide adoption of software. The larger the community base is using a piece of software, the better are the funding possibilities and thus also the sustainability options. Additionally developer commitment to an application is essential and experience shows that software packages with an evangelist imposing strong inspiration and discipline are more likely to achieve sustainability. While a single person can push sustainability to a certain level, open source software also needs sustained commitment from the developer community. Such sustained commitments include diverse tasks and roles, which can be fulfilled by diverse developers with different knowledge levels. Besides developing software and appropriate software management with measures for extensibility and scalability of the software, active (expertise) support for users via a user forum with a quick turnaround is crucial. The barrier to entry for the community as users as well as developers has to be as low as possible.

For additional information about the discussion, see Section 6.1.

5.1.3 Plans

The creation of a document on best practices needs a large and diverse community involved. The group has enlisted over ten contributors from the attendees at the WSSSPE3 and those on the mailing list. The primary mechanism for developing this document will be to examine and analyze the success of several well known community scientific software and organizations supporting scientific software. The group will attempt then to abstract general principles and best practices. Some of the tools identified for such analysis are the general purpose PETSc toolkit for linear system solution, NWChem for computational chemistry and the CIG (Computational Infrastructure for Geodynamics) organization dedicated to supporting an ensemble of related tools for the geodynamics community. The group also established a timeline and a rough outline (see Section 6.1) for the report.

Timeline:

• 28 Dec: Introduction and scope finished
• 06 Jan: Sections assigned
• 31 Jan: Analyzing funding possibilities for survey
• 31 Jan: First versions of section
• 15 Feb: Distribution to WSSSPE community
• 31 Mar: Final version of white paper
• 30 Apr: Submission of peer-reviewed paper?

5.1.4 Landing Page

The landing page with instructions, timeline and the white paper is here: https://drive.google.com/drive/folders/0B7KZv1TRi06fbnFkZjQ0ZEJKckk. Discussions can be also continued in https://github.com/WSSSPE/meetings/issues/42.

5.2.1 Fit with related activities

The solution this group envisions entails creating an environment where software developers are a stable part of a research team. Such an environment mitigates the risk of losing a key developer at a critical moment in a projects lifetime, and provides the benefits of building a store of institutional knowledge about specific projects as well as about software development for today’s research. The group’s vision is to find a way to promote a university/research institute environment where software developers are stable components of research project teams.

One strategy to promote stability is implementing a mechanism for developers to obtain academic credit for software development work (see §5.8.) With such a mechanism in place, traditional academic funding models and career tracks could properly sustain individuals for whom software development is their primary contribution to research. A contributing factor to the problem with the current academic reward system is the devastating effect on an academic publication record resulting from time in industry; such postings often develop exactly the skills that research software engineers need, yet returns to university positions following an industry role are penalized by the current structures. Retention of senior developers is hard, because these people are high in demand by the economy. However, people who have a PhD in science and enter industry, may desire to return for diverse reasons, and should be welcomed back.

While developing new mechanisms in the current academic reward system is a worthy aspirational goal, such a dramatic change in this structure does not seem likely in a time scale relevant to this working group. Accordingly, the working group sought alternative solutions that may be achievable within the context of existing academic structures. The group felt that developing dedicated research software engineering roles within the university and finding stable funding for those individuals is the most promising mechanism for creating a stable software development staff.

Measures of impact and success for research programming groups, as well as for individual research software engineers, will be required in order to make the case to the university for continued funding. Research software engineers will hopefully not be measured by publications, but by other metrics. Middle-author publications are common for RSEs. Most RSEs welcome co-authorship on papers when the PIs think that the contribution deserves it.

5.2.2 Discussion

It is hard for an individual PI in a university or college to support dedicated research software engineering resources, as the need and funding for these activities are intermittent within a research cycle. To sustain this capacity, therefore, it is necessary to aggregate this work across multiple research groups.

One solution is to fund dedicated software engineering roles for major research software projects at national laboratories or other non-educational institutions. This solution is in place and working well for many well-used scientific codebases. However, this strategy has limited application, as much of the body of software is created and maintained in research universities. Therefore, the group argues that research institutions should develop hybrid academic-technical tracks for this capacity, where employees in this track work with more than one PI, rather than the traditional RA role within a single group. This could be coordinated centrally, as a core facility, perhaps within research computing organizations which have traditionally supported university cyberinfrastructure, library organizations, or research offices. Alternatively, these groups could be organizationally closer to research groups, sitting within academic departments. The most effective model will vary from institution to institution, but the mandate and ways of working should be similar.

Having convinced themselves that this would be a positive innovation, the group members were then faced with the specific question of how to fund the initiation of this activity. A self-sustaining research software group will support itself through collaborations with PIs in the normal grant process, with PIs choosing to fund some amount of research software engineering effort through grants in the usual way. However, to bootstrap such a function to a level where it has sufficient reputation and client base to be self-sustaining will generally require seed investment.

This might come from universities themselves (this was the model that led to the creation of the group in University College London), but more likely, seed funding needs to come from research councils or other funding bodies (as with the Research Software Engineering Fellowship provided by the UK Engineering and Physical Sciences Research Council). The group therefore recommends that funding organizations consider how they might provide such seed funding.

Success, appropriately measured, will help make the case to such funding bodies for further investment. One might expect that metrics such as improved productivity, software adoption rates, and grant success rates would be sufficient arguments in favor of such a model. However, useful measurement of code cleanliness, and the resulting productivity gains, is an unsolved problem in empirical software engineering. To measure “what did not go wrong” because of an intervention is particularly hard.

The working group finally noted that the institutional case for such groups is made easier by having successful examples to point to. In the UK, a collective effort to identify the research software engineering community, with individuals clearly stating “I am a research software engineer,” has been important to the campaign. It will be useful to the global effort to similarly identify emerging research software organizations, and also, importantly, to identify longer-running research software groups, which have in some cases had a long running sui-generis existence, but which now can be identified as part of a wider solution. There remains the problem of how to “sell” the value of this investment to investigators within a university. This is an issue best addressed by the individual organizations that embark on the plan.

For more details on the discussion, see Section 6.2.

5.2.3 Plans

The first step in moving this strategy forward is to gather a list of groups that selfidentify as research software engineering groups, and to reach out to other organizations to see if there may be a widespread community of RSEs who do not identify themselves as such at this time. This working group will collect information about the organizational models under which these groups function, and how they are funded. For example, how many research universities currently fund people in the RSE track, whether they bear the RSE moniker or not. Are these developers paid by the university or through a program supported by research grants/individual PIs? How did they bootstrap the developer track to get this started? How successful is the university in getting investigators to pay for fractional RSEs? The group will author a report describing their findings, should funding be available to conduct the investigation.

5.2.4 Landing Page

A list of known UK research software engineering groups is available at http://www.rse.ac.uk/groups, and a list for the rest of the world is at http://www.rse.ac.uk/international. To add another group to the list, please make a pull request as requested on either of these pages.

5.3.1 Fit with related activities

Most previous research and discussion of industry and academic collaboration, sharing, and funding of research software has focused on the impact of such arrangements. Examples of these types of reports are:

• REF Impact Case Studies: http://impact.ref.ac.uk/CaseStudies/
• Background of projects funded in the UK: http://gtr.rcuk.ac.uk/
• Dowling Review from the UK: addresses complexity of work between these two communities: http://www.raeng.org.uk/policy/dowling-review
• Pathway to Impact – UK report: two pages of grant proposals are asked to forecast what impact they might have (including environmental, academic, economic).

5.3.2 Discussion

Although sustainability transitions are often studied under the broad umbrella of “technology transfer,” the group believes there are likely to be a number of different ways in which a pathway from initial production to long-term maintenance and secure funding is achieved. In short, industry sponsorship and/or direct participation is an important aspect of sustaining scientific software, but our current understanding of these transitions focuses narrowly on commercial successes or failures of those collaborations.

In looking at existing literature that addresses industry transitions, many reports (such as those listed above) focus on benefits that accrue to the private sector, or to a government that originally sponsored the research project. This literature does not address the impact that these transitions have on the accessibility or usability of the software, or the impact that these transitions have on the career of the researchers involved.

For more detail on the group’s discussion, see Section 6.3.

5.3.3 Plans

Plans for carrying forward are currently unclear—this project would require sustained attention and effort from the group members, and at least some amount of funding in order for those members to be involved for extended periods of time.

The broad goals that the group would like to accomplish are:

1. To complete a set of case studies which look at successful and unsuccessful transitions between academic researchers and industry
2. To create a generalizable framework, which might allow for a broader study of different transition pathways (other than between academia and industry)

The main plan for the group going forward is the creation of a white paper on the topic of sustainability transitions.

5.3.4 Landing Page

Transitions Pathways discussions can be posted at https://github.com/WSSSPE/meetings/issues/46 or an email be sent to Nic Weber¹⁸ to find out more about the group’s efforts and how to participate.

5.5.1 Fit with related activities

This group’s discussion focused on identifying existing principles of software engineering design that could be adopted by the computational science and engineering communities.

5.5.2 Discussion

Software engineering principles form the foundation of methods, techniques, methodologies, and tools. Consisting of members from different backgrounds, including quantum chemistry, epidemiology, computer science, software engineering, and microscopy, this group discussed the principles of software engineering design for sustainable software (starting with principles from the Karlskrona Manifesto on Sustainability Design [30], Tate [31], and the Software Engineering Body of Knowledge (SWEBOK) [32]) and their application in various domains including quantum chemistry and epidemiology. The group examined the principles and took a retrospective analysis of what the developers did in practice against how the principles could have made a difference, and asked, what do the principles mean for computational scientific and engineering software, and how do the principles relate to non-functional requirements? It appeared that the sustainable software engineering principles should be mapped to two core quality attributes that underpin technically sustainable software: extensibility, the software’s ability to be extended and the effort level required to implement the extension; and maintainability: the effort required to locate and fix an error in operational software.

For more information about the discussion, see Section 6.4.

5.5.3 Plans

The next steps in this endeavor are to (1) Systematically analyze a number of example systems from different scientific domains with regards to the identified principles, to (2) Identify the commonalities and gaps in applying those principles to different scientific systems, and to (3) Propose a set of guidelines on the principles and examine how they exemplarily apply to scientific software systems. Preliminary work will be carried out through undergraduate or post-graduate student projects.

5.5.4 Landing Page

In the absence of a landing page, the Principles for Software Engineering Design for Sustainable Software working group requests an email be sent to Birgit Penzenstadler¹⁹ and Colin C. Venters²⁰ to find out more about the group’s efforts and how to participate.

5.6.1 Fit with related activities

The group discussion focused on identifying existing frameworks and activities for scientific software metrics. The group identified the following related activities:

• Computational Infrastructure for Geodynamics: Software Development Best Practices²¹
• WSSSPE3 Breakout Session: How can we measure the impact of a piece of code on research, and its value to the community?²²
• 2015 NSF SI2 PI Workshop Breakout Session on Framing Success Metrics²³
• 2015 NSF SI2 PI Workshop Breakout Session on Software Metrics²⁴
• NSF Workshop on Software and Data Citation Breakout Group on Useful Metrics²⁵
• U.K. Software Sustainability Institute Software Evaluation Guide²⁶
• U.K. Software Sustainability Institute Blog post: The five stars of research software²⁷
• Minimal information for reusable scientific software²⁸
• EPSRC-funded Equipment Data Search Site²⁹
• Canarie Research Software: Software to accelerate discovery³⁰
• Canarie Research Software: Research Software Platform Registry³¹
• BlackDuck Open HUB³²
• Innovation Policy Platform³³

5.6.2 Discussion

The group discussion began by agreeing on the common purpose of creating a set of guidance giving examples of specific metrics for the success of scientific software in use, why they were chosen, what they are useful to measure, and any challenges and pitfalls; then publish this as a white paper. The group discussed many questions related to useful metrics for scientific software including addressing if there is a common set of metrics that can be filtered in some way, can metrics be fit into a common template, which metrics would be the most useful for each stakeholder, which metrics are the most helpful and how would we assess this, how are metrics monitored, and many more. A more complete bulleted list of these questions can be found in Section 6.5. Next, a roadmap for how to proceed was discussed, including creating a set of milestones and tasks. The idea was put forth for the group to interact with the organizing committee of the 2016 NSF Software Infrastructure for Sustained Innovation (SI2) PI workshop in order to send a software metrics survey to all SI2 and related awardees as a targeted and relevant set of stakeholders. The five solicitations for software elements released under the NSF SI2 program all included metrics as a required component with submitters requested to include “a list of tangible metrics, with end user involvement, to be used to measure the success of the software element developed, … ”. These metrics are then reported as part of annual reports to NSF by the projects. Although neither the proposal text describing the metrics nor the reported metric results are publicly available, there is reason to believe that the community will be willing to provide this information through a survey mechanism. This survey would be created by one of the student group members. Similarly, it was suggested that a software metrics survey be sent to the UK SFTF (Software For The Future, led by the Engineering and Physical Sciences Research Council) and TRDF (Tools and Resources Development Fund, led by the Biotechnology and Biological Sciences Research Council) software projects to ask them what metrics would be useful to report. The remainder of the discussion focused mainly on the creation of a white paper on this topic. This resulted in a paper outline and writing assignments with the goal of publishing in venues including WSSSPE4, IEEE CiSE (Institute of Electrical and Electronic Engineers Computing in Science and Engineering magazine), or JORS (Journal of Open Research Software). More information about the group discussion is available in Section 6.5.

5.6.3 Plans

The main plan for the group going forward is the creation of a white paper on the topic of useful metrics for scientific software. The authoring of this white paper would happen in parallel with the creation of a survey by the group with the survey results to be incorporated in the white paper. The timeline for completion of the white paper is approximately one year targeting venues discussed in the previous section.

5.6.4 Landing Page

In lieu of a landing page, the Useful Metrics for Scientific Software working group requests an email be sent to Gabrielle Allen³⁴ to find out more about the group’s efforts and how to participate.

5.7.1 Fit with related activities

Three existing venues for discussion of related events are identified:

• Working towards Sustainable Software for Science: Practice and Experiences (WSSSPE) workshops [33]
• International Workshop on Software Engineering for High Performance Computing in Computational Science and Engineering (SEHPCCSE) [34]
• Workshop on Software Engineering for Sustainable Systems [35]

5.7.2 Discussion

Some next steps were identified to quickly test whether there is interest in establishing a community committed to increasing the degree of coordination across training projects. See Section 6.6 for more details about the discussion.

5.7.3 Plans

The main plan for the group is to convene a discussion to explore bringing together regular meetings of those involved in leading relevant training projects.

5.7.4 Landing Page

The Training working group requests an email be sent to Nick Jones³⁵ to find out more about the group’s efforts and how to participate.

5.8.1 Fit with related activities

Publishing Software Working Group (§5.9): publishing a software paper offers one existing mechanism for receiving credit, and further developing new publishing concepts for software will strengthen our activities.

A number of groups external to WSSSPE (although with some overlapping members) are also focused on aspects of software credit, including the FORCE11 Software Citation Working Group (see plans for coordination below). In addition, a Software Credit workshop³⁶ convened in London on October 19, following the conclusion of WSSSPE3. See Section 6.7 for more detailed discussion of related activities.

5.8.2 Discussion

The group discussed a number of topics related to software credit, including a contributorship taxonomy, software citation metadata, standards for citing software in publications, and increasing the value of software in academic promotion and tenure reviews. Although initial discussions both prior to and during WSSSPE3 focused on contribution taxonomy and dividing credit, discussing as an example the Entertainment Identifier Registry [36] used in the entertainment industry, the group decided to prioritize software citation. This decision was motivated by the idea that standardizing citations for software would introduce some initial credit for developers, and later the quantification of credit could be refined based on concepts such as transitive credit [37, 38].

The majority of the remaining discussion focused on standardizing (1) the metadata necessary for software to be cited and (2) the mechanism for citing software in publications. Moreover, discussions also oriented around the indexing of software citations necessary for establishing a software citation network either integrated with the existing paper citation ecosystem or complementary to it. See Section 6.7 for a more detailed summary of the working group’s discussion on these topics.

5.8.3 Plans

The group already merged with the FORCE11 Software Citation Working Group (SCWG), and their efforts will focus (over the next six to nine months) on developing a document describing principles for software citation. Following the publication of that document, the group will focus on outreach to key groups (e.g., journals, publishers, indexers, professional societies). Longer-term plans include working with indexers to ensure that software citations are indexed and pursuing an open/community indexer; these activities may be organized by future FORCE11 working groups.

5.8.4 Landing Page

Since near-term efforts will be shifting to the FORCE11-SCWG, interested readers should go to that group’s existing landing page³⁷ and GitHub repository³⁸.

5.9.1 Fit with related activities

• Reproducibility: Part of the purpose of these executable paper venues is to (at least partially) address the reproducibility issue by making papers recompute their own results.
• Software Credit (§5.8): Since these forms of publishing must make their sources explicit in order to execute, they should be easier to trace even if appropriately worded credit for software is not provided. In addition, they make it possible to provide or define additional metadata to make the tracing of credit clearer. Finally, attributions could be added to citations to identify whether a paper extends a result, verifies it, contradicts it, etc.
• Best Practices (§5.1): Because an executable paper showcases the code, and the code itself is subject to the review process, authors are more likely to pay attention to coding practices. In addition, because the paper must explain what the code does, better documentation is more likely to be achieved.

5.9.2 Discussion

The group felt that the best way to encourage the use of these new publishing concepts would be to create and curate a list of publishing venues that support them. The Software Sustainability Institute agreed to host this list.

See Section 6.8 for more details about the discussion.

5.9.3 Plans

The plan is to create and curate a web page describing executable papers, their value, and a list of what publishers support them. The group expects the page to be available in early January of 2016 on the Software Sustainability Institute’s website.

5.9.4 Landing Page

The aforementioned page will be published on the Software Sustainability Institute website: http://www.software.ac.uk.

5.10.1 Fit with related activities

There are a number of activities already in progress that are targeted at improving the user community for open-source software, including Mozilla Science’s “Working Open Project Guide” [39] and “UK Collaborative Computational Projects” (CCP)³⁹, or books such as “Art of Community” by Jono Bacon [40].

5.10.2 Discussion

Discussion revolved around a few questions: what are the benefits of having a “community” for software sustainability; what practices and circumstances may lead to having and maintaining a community; how can funding help or hinder this process; and perhaps most importantly, how can best practices be described and distilled into a document that can help new projects.

All the group members agreed on a few points: software must not only offer value, but there must be some support for users; and funding can help pay for that support, in addition to the usual funding for software development. Openness is generally a virtue. An evangelist, either in the form of a single person or some domain-specific group of users, is often the key factor.

Additional details on the group’s discussion can be found in Section 6.9.

5.10.3 Plans

The most important next steps is a “Best Practice” document, which would describe what successful projects with engaged communities look like, how to replicate this type of project, and look at the end of life of a community project. Another next step would be better training to increase recognition of need for science software projects to focus on building and supporting their user communities.

5.10.4 Landing Page

This group does not have a landing page yet. Please send requests to join and contribute by writing to both Dan Gunter⁴⁰ and Ethan Davis⁴¹.

6.1.1 Group Members

• Abani Patra – University at Buffalo
• Sandra Gesing – University of Notre Dame
• Neil Chue Hong – Software Sustainability Institute
• Gregory Tucker – University of Colorado at Boulder
• Birgit Penzenstadler – California State University Long Beach
• Abigail Cabunoc Mayes – Mozilla Foundation
• Frank Löffler – Louisiana State University
• Colin C. Venters – University of Huddersfield
• Lorraine Hwang – UC Davis
• Sou-Cheng Choi – NORC at the University of Chicago & Illinois Institute of Technology
• Suresh Marru – Indiana University
• Don Middleton – NCAR
• Daniel S. Katz – University of Chicago & Argonne National Laboratory
• Kyle Niemeyer – Oregon State University
• Jeffrey Carver – University of Alabama
• Dan Gunter – LBNL
• Alexander Konovalov – University of St Andrews
• Tom Crick – Cardiff Metropolitan University

6.1.2 Summary of Discussion

Core questions that will need to be explored are in reliability, reproducibility, usability, extensibility, knowledge management, and continuity (transitions between people). Answers to these will guide us on how a software tool becomes part of the core workflow of well identified users (stakeholders) relating to tool success and hence sustainability. Ideas that may need to be explored include:

• Requirements engineering to create tools with immediate uptake;
• When should software “die”?
• Catering to disruptive developments in environment, e.g., new hardware, new methodology;
• Dimensions of sustainability: economic, technical, environmental, and obsolescence.

Sustainability requires community participation in code development and/or a wide adoption of software. The larger the community base is using a piece of software, the better are the funding possibilities and thus also the sustainability options. Additionally, the developers’ commitment to an application is essential and experience shows that software packages with an evangelist imposing strong inspiration and discipline are more likely to achieve sustainability. While a single person can push sustainability to a certain level, open source software also needs sustained commitment from the developer community. Such sustained commitments include diverse tasks and roles, which can be fulfilled by diverse developers with different knowledge levels. Besides developing software and appropriate software management with measures for extensibility and scalability of the software, active (expertise) support for users via a user forum with a quick turnaround is crucial. The barrier to entry for the community as users as well as developers has to be as low as possible.

6.1.3 Description of Opportunity, Challenges, and Obstacles

There is an opportunity to collaborate on a white paper, which will be revisited regularly for further improvements, to enhance knowledge of the state of best practices, resulting in a peer-reviewed paper. We would like to reach a wide community by doing this. But these are also the challenges and obstacles – to get everyone to contribute to the paper and to reach the community.

White Paper Outline

1. Introduction and Scope of White Paper
2. Related Work
3. Case Studies
   1. PETSc
   2. NWChem
   3. CIG
4. Community Related Practices
   1. Findings
   2. Recommendations
5. Governance and management
   1. Findings
   2. Recommendations
6. Funding Related
   1. Findings
   2. Recommendations
7. Metrics for sustainability
8. Tools
9. Conclusions

6.1.4 Key Next Steps

The key next steps are to write an introduction, reach out to the co-authors, and to agree on the scope of the white paper.

6.1.5 Plan for Future Organization

Sandra Gesing and Abani Patra are the main editors and will organize the overall communication and the paper. Sections will be assigned to diverse co-authors.

6.1.6 What Else is Needed?

At the moment we do not see any further requirements.

6.1.7 Key Milestones and Responsible Parties

• 28 Dec: Introduction and scope finished (Abani Patra/Sandra Gesing)
• 06 Jan: Sections assigned (Abani Patra/Sandra Gesing)
• 31 Jan: Analyzing funding possibilities for survey
• 31 Jan: First version of each section
• 15 Feb: Distribution to the WSSSPE community
• 31 Mar: Final version of the white paper
• 30 Apr: Submission to a peer-reviewed journal?

6.1.8 Description of Funding Needed

We might need funding for a journal publication (open-access options).

6.2.1 Group Members

The group at WSSSPE:

• Don Middleton – National Center for Atmospheric Research
• Joshua Greenberg – Alfred P. Sloan Foundation
• James Hetherington – University College London
• Lindsay Powers – The HDF Group
• Mark A. Miller – San Diego Supercomputer Center
• Dan Sellars – CANARIE

This was further enhanced by additional discussions at the following GCE15 conference:

• Lorraine Hwang – UC Davis
• Simon Trigger – BioTeam, Inc.
• Nancy Wilkins-Diehr – San Diego Supercomputer Center
• Alexander Vyushkov – University of Notre Dame
• Sandra Gesing – University of Notre Dame
• Ali Swanson – University of Oxford

6.2.2 Summary of Discussion

In addition to the points noted in the main discussion (§5.2), we also discussed the following:

“Are you an RSE or a RA?” is not properly a binary question. Most of us sit at different points on that spectrum, and move along it during our careers (usually from RA to RSE—examples of movement in the other direction from readers would be welcomed). Either way, the label “Research Software Engineer” is now starting to have some power. Many scientists do not want to be writing code; some do, to varying degrees. These groups can usefully support each other.

What is the power of the label? How can we get the word out about RSE support using the label?

Will research science developers be required in the long run? One issue that came up was whether the need for developers was a time bounded one; is it the case that the new generation of computer and software savvy scientists will be so comfortable in developing their own code that the professional developers will not be needed? And this brings up the flip side question, “Do scientists really want to be writing code?”

We also had a little discussion about how to make a career path for research developers. It need not be solely an academic enterprise, but in the past tenure has often been problematic for people of this class.

Skills and resources may vary between teams. To help resolve this, maintaining high levels of communication between groups will be valuable. In the United Kingdom (UK), there are plans to permit resource sharing between institutional RSE groups. Perhaps there are circumstances under which an RSE skill exchange could be arranged, either formally or informally.

Collaborative funding can be crucial to RSE groups, to ensure that research leadership remains with the domain scientists. As an example, at NCAR, university partnerships are required for submission of proposals, so collaboration is an essential part of grant submission, and this will tend to bring developers and scientists together. The UCL group also follows this approach, with all bids requiring an academic collaborator.

Domain scientists and developers are funded together in a single proposal. Another example of a success is the development of semantics and linked data in support of ocean sciences. An EarthCubefunded project pairs domain scientists with RSEs and has been successful; the semantics attached have increased data use and discovery significantly.

An alternative approach has been the provision of programming expertise as part of national compute services. The US XSEDE project’s Extended Collaboration Support Services (ECSS) is a set of developers who are paid with XSEDE funding, and are on “permanent” staff. When PIs request allocations on XSEDE resources, there is a finite pool of developer time that can be awarded, typically for one year only, and at partial effort, typically 20 percent or so. The finite time allowed provides motivation for the scientist and the scientist’s group to work closely with the developer and to become educated in what the developer is doing, so they can sustain the effort once the ECSS period is over. This funding mechanism can be highly efficient for scientific problems, because the developer pool assembled by the research providers are, by definition, expert in the characteristics of their specific resource, and can very quickly assess the scientist’s needs, and what it will take to implement software that meets the user’s needs. However, it does not develop capacity within institutions, and since XSEDE is a time-bounded program, it should not be relied upon as a long-term solution to acquiring this type of capacity.

The UK allows this kind of collaboration to support the creation of scientific software for the large supercomputing resource (ARCHER). However, while the support can come from the staff of the Edinburgh Parallel Computing Centre, who hosts the computer, this “embedded CSE” resource also funds the programming coming from local groups. This has been very helpful in providing funding to establish local groups. These groups work best when they develop good collaborations with national cyberinfrastructure pools. When an organization assembles a developer pool, diversity is developed and skills can be transferred.

We would like to see these models applied outside high performance computing. Most scientific software is not destined to run on national cyberinfrastructure, but needs similar support. The argument regarding making better use of expensive hardware through software improvements has been useful politically, (and many RSE groups are cited in organizations which host clusters for this reason), but the time has come to make the case that software itself is a critical cyberinfrastructure, and, with a much longer shelf-life than hardware, is itself a capital investment.

The CANARIE group (Canada) accepts proposals for providing services to broad communities, integrating people who are doing things that are complementary. The goal is to make the available stack more robust and richer for everyone. They offer short cycles of funding for creating some useful functionality that shows a diversity of input and draws from across disciplines as a key metric, If this metric is met successfully, then more funding may follow. This can apply within or across institutions.

There can be problems communicating across cultural barriers, with domain scientists seeing developers as “other”. Both collaboration and tools to fund, encourage, or motivate collaboration are extremely important.

We think support from non-governmental organizations will be important if RSE groups will become established. The Sloan Foundation is currently funding data science engineers, who work in the context of other software developers at the University of Washington. These scientists work in the e-Science Studio/Data Science Studio, and they help a group of graduate students in solving their problems in data science and data management. During Fall and Spring, a 10-week incubator program allows students to work two days a week on a data-intensive science project. Some fraction of the developer time is dedicated to the developers’ personal interests as well as instruction.

The goal for Sloan is to obtain success stories and demonstrable value of the presence of data scientists on university staff. These stories are the basis for arguments to the host organization. This is an effort to create awareness of the value of research scientist developers. Embedding with scientists, and adding spare capacity is critical to making the innovation possible. This model is essentially to argue for permanent budget lines to support data scientists as part of university staff hires, just as with core facilities. This could become a fee-for-service model requested by grant funding, just as DNA sequencing is for core facilities, if it becomes apparent that this gives competitive advantage to a university’s research effort.

One model that has been helpful in finding funding for RSE groups is the use of funds left over on research grants when RAs have left prematurely – PIs like this arrangement as it is hard to find good staff for short-term positions, so having a pool of research programming staff on hand resolves this problem. We recommend that funders give explicit guidance to grant holders and institutions that such an arrangement is favorable. Framework agreements permitting this to go ahead without checking back every time with funders and/or grant panels would further smooth this. (This also provides more stable jobs for those who hold these skills, but arguments about making life nicer for postdocs will not help persuade funders or PIs!)

There is some question about the most effective duration and percentage of full time for a programmer’s work on a project. At least three months is necessary for the programmer to read into the science (RSEs must not become so disengaged from research that they do not have time to read a few papers – this will result in code which does not meet scientific needs), but too long could result in an RSE losing their flexibility, becoming so engaged in one project that when that project ends, they find it hard to transfer. For this reason, we recommend that 40% is ideal; two projects per developer, with some time for training and infrastructure work. Having two developers per project seems to be ideal, in the sense that software development is enhanced by two pairs of eyes.

There is, as yet, no clear answer as to the scale of aggregation needed to make such a program work. A university wide program allows enough scale to be robust to fluctuations of funding within one field. But a specialization focus on developers to support, for example, physical or biological sciences may be preferable, if the customer base is large enough. The desire to aggregate enough work to make it sustainable, and the need to have domain-relevant research programming skills, are in tension.

In the UK, another source of funding for research software is the Collaborative Computational Projects (CCPs): domain specific communities put forward proposals that are a priority of the community as a whole, for example, biosimulation or plasma physics. These bodies act as custodians of community codes, and a central team also provides software engineering support.

However this area develops, the need for funding for software as a cyberinfrastructure component is clear. Funding that permits code to be refactored, tidied, and optimized is rare; this is often done “on the sly” in a scientifically focused grant. The UK EPSRC’s “software for the future” call, which really permits explicit investment in software as an infrastructure, is so oversubscribed as to have a 4% success rate; the demand is clear!

One opportunity is the idea of co-design, where infrastructural libraries are developed alongside the scientific codes that will call them. However, collaboration is hard to foster here; as incentive structures are still focused on short-term papers. This can cause infrastructure developers to focus more on publications in their areas of mathematics and computer science, the domain developers on the shorter-term needs of their own fields. Genuine collaborative co-construction is harder to foster.

It can be more difficult to help leading domain scientists see the value of engineering effort than those in their teams who are forced to work with difficult-to-use or unreliable software tools, as they do not see the pain. Perhaps a version of “software carpentry” targeted at those PIs who are awarded or apply for software-intensive grants could be valuable here.

RSEs provide a useful contribution to their universities’ teaching missions, as well as research, as they are well placed to deliver the research programming training that many scientists now need. In the longer term, with programming skills taught to all through their careers, we hope specialist scientific developers will be less needed.

6.2.3 Key Next Steps

We will seek to identify and approach existing research programming organizations, to get their permission to list them on a list of research software groups. Casual conversation during the meeting made it clear that although the title is not widely used in the US, this position is not rare. We spoke with several individuals who, at distinct universities, had RSEs (in effect if not in name) who were funded under differing models.

We will also look for examples of groups which have successfully become self-sustaining following initial seed funding.

In this respect, information gathering via a survey and subsequent analysis could be very useful. We would need to assemble a list of targeted individuals. (What positions and ranks are likely to know and care enough to respond?) Perhaps the Science Gateway Institute has already acquired information that could be helpful to advance this issue, and/or craft a proper survey and suggest target individuals.

6.2.4 Plan for Future Organization and Future Needs

The UK RSE community will provide initial facilities to host this list, and continue to work to spread the initiative, but local leadership in the US is needed if this campaign is to succeed. This will require an initial gathering of identified research software organizations in the US to this end.

6.2.5 Description of Funding Needed

Financial support for an initial conference that brings together research software groups to form an organization and create a resource sharing structure would help to further this campaign. Funding to conduct and analyze a survey could also be quite useful as knowing where we stand today, and what models are in use could fuel the ideas for further development of developers in this category.

In the longer term, funding organizations, especially non-governmental organizations with the capability to effect innovation through seed funding, could provide support to nucleate the creation of research software groups. As noted above, Sloan has already initiated one such program, and collaboration with Sloan or at least study of their methods and success or failure could be extremely useful in approaching universities and other institutions in funding this development track. It seems clear that if the value proposition can be made to university administrators, this track could flourish with buy-in at the administrative level.

6.3.1 Group Members

• Nic Weber – University of Washington
• Suresh Marru – Indiana University
• Jeffrey Carver – University of Alabama
• Davide DelVento – NCAR/CISL
• Steven Brandt – Louisiana State University

6.3.2 Summary of Discussion

The group’s initial broad question was, “What makes for successful transitions of scientific software from academia to industry?” There are a number of potential funding transitions that may occur:

• A project could be re-funded, and development or maintenance of the software continue as planned.
• A project might locate a new source of funding in which case the software may be further developed or simply maintained as before.
• The project could transition to a community supported model whereby the software’s ownership, maintenance, and stewardship become similar to peer-production models in open-source (e.g., see Howison [41]).
• The project could receive some form of industry sponsorship in which case ownership of the intellectual property, licensing, maintenance activities, hosting, etc. may change significantly.
• The project could gain attention from a industry use case who would potentially make in-kind contributions by having paid staff contribute to the software.

We characterized each of the above potential changes in funding as “transition pathways” to sustainable software (see similar work by Geels and Schot [42]).

Our work at WSSSPE3 included the following three activities (described in more detail below): (1) brainstorming goals for this type of research, (2) imagining potential outcomes of completing a set of case studies on this topic, and (3) generating a set of working definitions for some of the broad concepts we are describing.

First, we discussed the goals of this research, attempting to answer the question What is the goal of doing research on transition pathways? A number of research questions arose: Can we identify collaborations that have occurred and try to understand which were successful, which were unsuccessful, and what factors contributed to these successes/failures? Can we determine what each partner wants to get out of such a collaboration? For example, why would industry be interested in collaborating with academia? Or why would academia be interested in collaborating with industry? How could we design a study that focused on the impact of the software in undergoing this type of transition?

Next, we imagined potential outcomes of research on this topic, involving a set of case studies that look at successful and unsuccessful transitions of researchers between academia and industry. This might address each of the transition types (as described below). Successful transitions are described as those that lead to either weak or strong sustainability (also defined below). In addition, the results from this research might help create a generalizable framework that might allow for the study of different transition pathways (other than academia to industry).

Finally, we created some general definitions for these concepts; we characterize transitions in the following ways:

• Handoff model: academia initially writes the software, industry (for-profit or nonprofit) then takes over the project.
• Co-Production Model: industry and academia interact throughout development of the project.
• Sponsorship Model: academia writes and maintains the software; industry contributes funding for the development/maintenance of software. In this example, industry is also likely a user of the software.
• Spinoff model: transition to a for-profit or non-profit company owned by or in collaboration with original developers.

We characterized sustainability in the following ways:

• Weak Sustainability: Software continues to be accessible, useful, and usable.
• Strong Sustainability: Software meets criteria above, but is also able to be reused for further innovation (i.e., issued non-restrictive open-source license).

We refer readers to Becker et al. [6] for an extended discussion of weak versus strong sustainability.

6.3.3 Description of Opportunity, Challenges, and Obstacles

The opportunity is to create a catalog of success/failure for current and future software projects to be prepared for transitions and achieve sustainability of the software.

The obstacle is more superficial, in finding a champion to gather such information. It will be a challenge to keep this information and surveys updated. With changing rapidly changing industry landscapes, an obsolete survey could be of less or no use.

6.3.4 Key Next Steps

Identify projects that are collaborative, perhaps by reviewing funded projects from programs specifically geared towards industry academic collaborations.

Develop a systematic process for conducting case studies (what kind of data are being gathered about each case).

6.3.5 Plan for Future Organization

No concrete plans have been made at this point. If the community can rally behind this topic, some momentum could be built. Those interested should post at https://github.com/WSSSPE/meetings/issues/46

6.3.6 What Else is Needed?

Nothing at the moment.

6.3.7 Key Milestones and Responsible Parties

A key portion of this effort will require focused surveys of projects which have succeeded and failed in transition. Both these categories will yield good learning on what works and what does not work. The group has identified what needs to be studied further, but has not identified responsible parties to conduct them.

Community members could help in gathering data by means of interviews, historical documents or documentation, and surveys.

An example of data collection is:

• Origin: Where did project start?
• People involved: How many people in original project were involved in transition/collaboration?
• Specs on software
• Language
• Size
• Hardness (age)
• Lead-up to Transition: How long was project in development before it began transition?
• Motivation for Transition: Why was transition initiated? By whom?

6.3.8 Description of Funding Needed

Concrete funding needs were not discussed in this working group but a general impression was that some seed funding would motivate members of this group or others in community to launch a survey effort.

6.4.1 Group Members

• Birgit Penzenstadler – California State University, CA, USA
• Colin C. Venters – University of Huddersfield, Huddersfield, UK
• Matthias Bussonnier – UC Berkeley, CA, USA
• Jeff McWhirter – Geode Systems
• Patrick Nichols – National Center for Atmospheric Research, CO, USA
• Ilian Todorov – Science & Technology Facilities Council, UK
• Ian Taylor – Cardiff University, UK
• Alexander Vyushkov – University of Notre Dame, IN, USA

6.4.2 Summary of Discussion

This group was comprised of members from different backgrounds, including quantum chemistry, epidemiology, microscopy, computer science, and software engineering. Each participant was invited to give their perspective on the topic area and what they thought were the crucial points for discussion. There was a general consensus that there was a need for relating principles to practice for the computational science and engineering community. Furthermore, various members of the group expressed their interest in tools and best practices for facilitating the maintenance and evolution of scientific software systems. It was agreed to identify principles from software engineering and from sustainability design and, based on those lists, discuss what each of those would mean applied to specific example systems from the expert domains of some of the group members. The group identified a number of software engineering principles drawn from the Software Engineering Body of Knowledge (SWEBOK) [32].

Software design principles include abstraction, coupling and cohesion, decomposition and modularization, encapsulation and information hiding, separation of interface and implementation, sufficiency completeness and primitiveness, and separation of concerns. Similarly, user interface design principles include learnability, user familiarity, consistency, minimal surprise, recoverability, user guidance, and user diversity. The sustainability design principles were drawn from the Karlskrona Manifesto on Sustainability Design [6]. The manifesto states that sustainability is systemic, multidimensional, and interdisciplinary; transcends the system’s purpose; applies to both a system and its wider contexts; requires action on multiple levels; requires multiple timescales; changing design to take into account long-term effects does not automatically imply sacrifices; system visibility is a precondition for and enabler of sustainability design. A number of sustainable software engineering principles proposed by Tate [31] were also considered including: continual refinement of product and project practices; a working product at all times; continual emphasis on design; and value defect prevention over defect detection.

This congregated list is an initial collection of principles that could be extended by adding from further related work form separate disciplines within the field of software engineering, including requirements engineering, software architecture, and testing. The group identified two example systems to discuss the application of the principles. The first one was a quantum chemistry system that allows the analysis of the characteristics and capabilities of molecules and solids. The second one was a modeling system for malaria that permitted biologists to analyze a range of datasets across geography, biology, and epidemiology, and add their own datasets. The group then examined the principles and took a retrospective analysis of what the developers did in practice against how the principles could have made a difference. This raised the question, what do the principles mean for computational scientific and engineering software? Similarly, how do the principles relate to non-functional requirements? It was suggested that at the very minimum, that sustainable software engineering principles should be mapped to two core quality attributes that underpin technically sustainable software:

• Extensibility: the software’s ability to be extended and the level of effort required to implement the extension;
• Maintainability: the effort required to locate and fix an error in operational software.

These fundamental building blocks could then be extended to include other quality attributes such as portability, reusability, scalability, usability, and energy efficiency etc. Nevertheless, this raises the question of what metrics and measures are suitable to demonstrate the sustainability of the software. In addition, what do the five dimensions of sustainability mean for scientific software, i.e., environmental, economic, social, technical and individual?

6.4.3 Description of Opportunity, Challenges, and Obstacles

The opportunity was identified to distill existing software engineering and sustainability design knowledge into “bite sized” chunks for the Computational Science and Engineering Community. In addition, two primary challenges were identified:

• Mapping of the principles to best practices.
• Demonstrating the return on investment of those best practices.

6.4.4 Key Next Steps

In order to achieve the following three goals: (1) a systematic analysis of a number of example systems from different scientific domains with regards to the identified principles, (2) the identification of the commonalities and gaps in applying the principles to different scientific systems, and (3) a proposal of a set of guidelines on the principles, the following next steps were discussed.

6.4.5 Plan for Future Organization

The following plan for future organization was discussed:

• Identify suitable undergraduate or post-graduate students.
• Design and pilot study.
• Organize coordinating online calls via Google Hangout.

6.4.6 What Else is Needed?

• Ethics committee review panel approval required for data collection.

6.4.7 Key Milestones and Responsible Parties

The following key milestones were discussed as a roadmap for the set of guidelines on software engineering principles:

• Oct/Nov 2015: Study design and interview guideline
• Jan/Feb 2016: Interviews conducted and transcribed
• Mar/Apr 2016: Analysis complete
• May 2016: Report written

6.4.8 Description of Funding Needed

Specific funding was not discussed in this working group. However, this is a open topic that can be discussed in relation to emerging funding calls from National agencies or grant proposal initiatives.

6.5.1 Group Members

• Gabrielle Allen – University of Illinois at Urbana-Champaign
• Emily Chen – University of Illinois at Urbana-Champaign
• Neil Chue Hong – U.K. Software Sustainability Institute
• Ray Idaszak – RENCI, University of North Carolina at Chapel Hill
• Iain Larmou – Engineering and Physical Sciences Research Council
• Bernie Randles – University of California, Los Angeles
• Dan Sellars – Canarie
• Fraser Watson – National Solar Observatory

6.5.2 Summary of Discussion

The group discussion began by agreeing on the common purpose of creating a set of guidance giving examples of specific metrics for the success of scientific software in use, why they were chosen, what they are useful to measure, and any challenges and pitfalls; then publish this as a white paper. The group discussed many questions related to useful metrics for scientific software as follows:

• Is there a common set of metrics, that can be filtered in some way
  • – Does this create a large cost
• Can we fit metrics into a common template (i.e., for collection, for description)
• Which would be the most useful ones
  • – Which ones would be most useful for each stakeholder
• Which ones are the most helpful, and how would we assess this
• How do you monitor
  • – Self-checking – if monitoring is done in the open, then people will call out cheats
• Should this be published with the software metadata
  • – This would make it easier for public to see the metadata
  • – However, there is no commonly-used standard (DOAP is a good standard but not widely adopted)
  • – The Open Directory Project (ODP) metadata is available for UK infrastructure
• Intersection of most useful and easiest to collect should be explored
• How can students/curricula be used as part of a solution
• Number of users could be affected by other metrics including, e.g., accessibility
• Assume metrics are collected properly, but guidance should be provided none-the-less
• Continuum for each metric
  • – Ideal situation is the absolute minimum, so that people can decide on their own what the cost versus usefulness tipping point is
• Maturity plays a part
  • – Consider different metrics brackets for different maturity levels
• What are we using metrics for
  • – What software should I use if I have a choice
  • – Where should funders place funding for best impact (e.g., funding two-star software versus three-star) and where there are gaps
  • – How to promote reduction of code proliferation
  • – Metrics used for software panels to provide information
  • – Metrics used for finding problems in their systems
• Can we use metrics to help people identify the best codes as part of a community effort

Next, a roadmap for how to proceed was discussed including creating a set of milestones and tasks as follows:

• Can we create a roadmap and milestones for this activity
• Need to come up with a set of tasks
• Go to NSF Software Infrastructure for Sustained Innovation (SI2) projects asking them what metrics they defined, and how useful they were
  • – Milestone: Create report which assesses the metrics that SI2 projects used
    • * Ask SI2 PIs to say what metrics they said they would use (copied from proposal)
    • * Ask SI2 PIs what numbers they reported
    • * Ask SI2 PIs what they would have changed
    • * A UIUC student on the project will work on this
  • – Tentatively aim for March 2016
• Do something similar for UK SFTF and TRDF software projects to ask them what would be useful metrics to report; also eCSE projects
  • – Compare these to understand if there were any implications for including metrics
• Collaboratively create plan and documentation for doing this
  • – Give some examples from group members projects, and aim to build out some of the measurement continuum
  • – Road-test at the WSSSPE4 meeting
• Collect the various frameworks together and do a comparison summary

The idea was put forth for the group to interact with the organizing committee of the 2016 NSF Software Infrastructure for Sustained Innovation (SI2) PI workshop in order to email out a software metrics survey to all SI2 and related awardees as a targeted and relevant set of stakeholders. This survey would be created by one of the student group members. Similarly, it was suggested that a software metrics survey be sent to the UK SFTF and TRDF software projects to ask them what metrics would be useful to report. The remainder of the discussion focused mainly on the creation of a white paper on this topic. This resulted in a paper outline and writing assignments with the goal of publishing in venues including WSSSPE4, IEEE CISE, or JORS.

6.5.3 Description of Opportunity, Challenges, and Obstacles

The following opportunities, challenges, and obstacles were discussed:

• Metrics are important for:
  • – Tenure and promotion
  • – Scientific impact
  • – Discovery
  • – Reducing duplication
  • – Basis for potential industrial interest in adopting software
  • – Make case for funding
• No commonly used standard for collecting or presenting metrics
• We do not know if there is a common set of metrics
• We have to persuade projects that it is useful to collect metrics

6.5.4 Key Next Steps

The following next steps were discussed:

• Skype phone call to coordinate shortly after the conclusion of the WSSSPE3 workshop
• Get started on IRB at University of Illinois Urbana-Champaign in anticipation of SI2 project survey (may need more thought into survey)
• Get started on white paper and associated survey

6.5.5 Plan for Future Organization

The following plan for future organization was discussed:

• Our group has created a white paper outline with sections assigned to the above individuals; see timeline below.
• Organize coordinating phone calls.

6.5.6 What Else is Needed?

The following list of what else is needed was discussed:

• IRB approval/exemption needed for surveys, collecting data
• Coordination with 2016 NSF SI2 PI workshop organizing committee to possibly piggyback on this event to offer survey to attendees in advance
• Coordination (mail communication, info page, etc.) via WSSSPE GitHub or other means?

6.5.7 Key Milestones and Responsible Parties

The following items were discussed as a roadmap for the production of a white paper:

1. October – November 2015: IRB paperwork as appropriate completed (Gabrielle Allen and Emily Chen)
2. October – December 2015: Draft white paper sections 1-3 (the paper outline has initial writing assignments)
3. October – December 2015: Run surveys and collect information
   1. Piggyback on planning for 2016 NSF SI2 PIs meeting to be held Feb 16-17, 2016
4. January – February 2016: Analyze results of data collection from projects
5. March – April 2016: Draft sections 4-7 of the white paper
6. May 2016: Draft section 8-9 of the white paper
7. May – June 2016: Get initial feedback from members of the community and revise
8. Est. July 2016: By the time of next CFP for WSSSPE, to have a complete draft of the white paper
9. Est. Sept – Oct 2016: Responses to white paper to be submitted to WSSSPE4

6.5.8 Description of Funding Needed

Funding needs were not discussed in this working group and it was thought that this could potentially be revisited down the road.

6.6.1 Group Members

• Nick Jones – New Zealand eScience Infrastructure
• Iain Larmour – Engineering & Physical Sciences Research Council, UK
• Erin Robinson – Foundation for Earth Science

6.6.2 Summary of Discussion

While little training focuses specifically on sustainable software, a variety of training activities could increase researcher awareness of and engagement with software professionals and software engineering practices. Research Software Engineers are being recognized as critical contributors to high quality research; the pathway to acquire and master the relevant skills is not yet clear; equally those skills required by researchers in general are also not commonly understood nor routinely developed.

The group’s discussion explored a rapidly growing array of training that is seen to contribute to sustainable software. The offerings are diverse, including: self-paced online modules focused around specific tools; single and multiple day training workshops that raise awareness of a tool chain to support collaborative and shared software development within a research workflow; block courses specializing on particular methods, technologies, and applications; academic programs at undergraduate and masters levels; doctoral training programs that in part contain requisite skills training activities.

While some of this training focuses on applying software engineering practices within the context of research, meeting the values and goals of research are less often incorporated as explicit learning outcomes. With software (and similarly, data) often being the only tangible artifact of a research method or protocol, the dependency between software applications and the quality of research adds complexity to the learner’s journey. In recognition of the longer term investment required by researchers to integrate such skills into their research practices, many activities are focusing on emotionally engaging researchers and cohorts, to build a sense of shared purpose beyond the obvious goal of technical skill acquisition.

In reviewing current training activities, the group identified a variety of perspectives seen as useful in positioning activities in ways to better communicate why and when best to apply each activity. Training can be categorized on a variety of spectra, with content and delivery ranging across them, for example: programming to research; basic to advanced; technical to emotional; informal to formal; and self-paced to participative. A few attempts have been made to situate a cross section of training activities within such dimensions, creating easier means of communicating the value of any specific opportunity and the pathways across opportunities over time.

Evaluation of training delivery and outcomes is seen as a weakness common to most non-academic training activities. Opportunities for measuring success in delivering training start simply with collecting a Net Promoter Score, which lets those delivering training know whether attendees are likely to recommend the training to others. In looking at the longer term outcomes for the learner, frameworks such as Bloom’s taxonomy and Kirkpatrick’s evaluation model offer possible approaches.

In this latter case of formal evaluation, ownership of evaluation as a component of career development for any researcher appears mostly absent. While academic research institutions have professional development centers to support research staff, the skills taught which might impact on sustainable software are limited at best, and lack a clear and coherent development pathway.

Coordination of these training projects will depend on buy-in from a broad range of training program and activity leaders, suggesting a key opportunity lies in identifying and bringing together these people on a regular basis.

6.6.3 Description of Opportunity, Challenges, and Obstacles

Software skills are needed by an increasing array of researchers and fields. The training arc is not well-defined, with a sometimes baffling array of training opportunities responding to various facets of skill deficit and need. Given this current complexity, coordination across training projects would create common frames of reference, communicating and integrating activities to better serve the needs of researchers.

Building this community could lift the maturity of training projects and capabilities, enabling more advanced approaches to address key gaps in evaluation, career development, and a lift in the standard of research practices.

In aiming at these opportunities, it will be necessary to find the means to support those involved in leading training activities to allocate time to coordination activities, which will often sit beyond their current scope of responsibility.

These activities are also distributed globally, with no single country or region offering a comprehensive set of capabilities and initiatives. Any coordination activity will therefore need to raise the profile of the opportunity gap with relevant research funders and policy makers.

6.6.4 Key Next Steps

The goal of the following next steps is to quickly test whether there is interest in establishing a community committed to increasing the degree of coordination across training projects.

1. Hold a virtual meeting by December 2015, to bring together a broader group of interest in this topic, with specific goals to:
   1. Identify programs with existing activities aimed at integrating across training projects.
   2. Identify training projects with an interest in participating in coordination efforts.
   3. Identify funding opportunities to bring together training program and project leaders to identify shared goals for future coordination of activities.
   4. Agree on a communications plan to qualify whether programs, projects, and funders are interested in engaging and committing to ongoing activities.
2. Review progress within 3 months, to establish next steps, if any.

6.6.5 Plan for Future Organization

Continue to track progress by posting comments to WSSSPE3 issue.

6.6.6 What Else is Needed?

If the group moves from early-stage formation into working towards shared goals, expertise will likely be required in pedagogy and training evaluation.

6.6.7 Key Milestones and Responsible Parties

1. October through December, Nick Jones and Erin Robinson to draft WSSSPE3 report back.
2. Before February 2016, Nick Jones and Erin Robinson to call a meeting of the broader group, to review key next steps.
3. Second quarter 2016 – if willing parties are identified, draft workshop proposal and identify a relevant forum, including future WSSSPE events.

6.6.8 Description of Funding Needed

Workshop/RCN travel funding to bring together key program, project, and funder representatives from across North America, EU, UK, Australasia. In addition, funding to support work on better defining the landscape of training activities, the useful perspectives in communicating the value of the varied training projects, and the possible pathways through training activities over time.

6.7.1 Group Members

• Alice Allen – Astrophysics Source Code Library
• Sou-Cheng Choi – NORC at University of Chicago, Illinois Institute of Technology
• James Hetherington – University College London
• Lorraine Hwang – University of California, Davis
• Daniel S. Katz – University of Chicago, Argonne National Laboratory
• Frank Löffler – Louisiana State University
• Abigail Cabunoc Mayes – Mozilla Science Lab
• Kyle E. Niemeyer – Oregon State University
• Grace Peng – National Center for Atmospheric Research
• Ilian Todorov – Science & Technology Facilities Council, UK

6.7.2 Summary of Discussion

The following section summarizes the working group’s discussion based on contributions prior to the meeting [43] and the collaborative notes taken during the meeting [44]. Please refer to the original sources for the unedited discussions if necessary.

Initial discussions focused on both various mechanisms for, and the philosophical approach behind, crediting software in scientific papers. These began with proposals for various ways to credit software (or other research products including data) that contributed more significantly than a generic citation, including:

• A hierarchy of citations, with a “substantial” citation category to indicate software or data that played a more significant role in the research;
• Transitive credit [37, 38], which assigns contriponents (contributors and components) various weights depending on their level of importance;
• Project CRediT [45], which assigns roles to paper authors based on their specific contributions; and
• Mozilla Science Lab’s recently introduced Contributorship Badges for Science [46], which provide a badge—associated with an ORCID [47]—that recognizes author contributions using the taxonomy outlined in Project CRediT.

However, as of this writing, only Project CRediT roles [48, 49] and Contributorship Badges [46] have been implemented for published papers, and both of these only provide a single “Software” or “Computation” category associated with software. In addition, neither of these options allows for the citation of software itself, but only provide an author contribution related to software. The discussion quickly focused on transitive credit as a more quantitative measure of allocating credit to both authors and software, although there were some concerns about authors overestimating their own contributions compared to prior work.

The discussion then evolved into philosophical questions about the importance or reliance of a particular work on prior science, materials, or software—in other words, whether there is a difference between depending on prior scientific advances and depending on certain software (or experimental equipment). Multiple contributors converged on the conclusion that unique capabilities require some additional credit. The—albeit limited—consensus was that if a particular study relied on the unique capabilities of software, data, or an experimental apparatus, then the authors or developers that created this capability should be credited somehow.

The group also agreed on the fact that additional data was required to support the assertion that software was not being sufficiently cited in the literature. In particular, this issue seemed to be field-dependent. For example, as shown by a study of Howison and Bullard [50], in the field of biology, the most-cited papers appear to be those describing scientific software. However, this may not—and likely is not—the case in other fields, nor is it clear whether developers of scientific software, even in the case of the biology field, are receiving sufficient credit for their efforts.

In the breakout sessions on the first day of WSSSPE3, the group discussed and deliberated over the Entertainment Identifier Registry (EIDR) [36] as a potential model for scientific software. That system assigns unique Digital Object Identifiers (DOIs)—the same system used for scientific publications—to all content (e.g., movies, television shows) and contributors, along with relevant metadata. One important use of the EIDR system is to track rights and credits for contributors to entertainment works in order to distribute revenues—similar to the proposed transitive credit concept.

The group also discussed separating quantitative measures (e.g., number of citations) from the value of a work in order to give credit, moving towards qualitative or anecdotal evidence of value. Other topics that were brought up included a form of PageRank [51] for citations, based on number of mentions, and using market penetration or adoption rate in a community as a metric, although it was not clear how this would be measured. Finally, the concept a software tool’s uniqueness or indispensability to a community was mentioned, with value being characterized by a particular piece of software either offering unique capabilities or doing something better, faster, or with less computational requirements than other offerings.

On the second day of WSSSPE3, the group decided to put aside the taxonomy of contributions and focus on software citations to ensure developers receive credit (regardless of contribution). Eventually, once software citations are standardized, the goal would be to return to establishing different roles/contributions for this credit. Following this decision, the group identified two necessary actions to move forward:

1. standardizing a citation file or some other form of metadata associated with software, and
2. standardizing the way to cite software (used directly) in papers.

For both of these actions, a number of ongoing efforts were identified and discussed.

Software Citation Metadata:

At a minimum, the metadata required for software citation includes:

• Name of software,
• DOI for software,
• Contributors, in the form of names and ORCIDs,
• Software dependencies, in the form of DOIs, and
• Other people and artifacts that would be cited or acknowledged in a paper.

This information would then be contained in a citation file, e.g., as part of the GitHub repository. The group also discussed similar efforts such as CodeMeta⁵⁰, an attempt to codify minimal metadata schemes in JSON and XML for scientific software and code, and implementing transitive credit via JSON-LD [38]. Some questions arose about how this information would be stored for closed-source software.

As one mechanism for constructing accurate contributor lists from existing project contributors, the group discussed associating GitHub accounts—as well as accounts on Bitbucket, CodePlex, and other repositories for open-source scientific software—with ORCID accounts. However, a (quick) response from GitHub (via Arfon Smith) indicated that this might not be possible in the near future: “GitHub doesn’t have any plans to allow ORCID accounts to be associated with GitHub user accounts.”

Citing Software in Publications:

Although far from a standard practice, examples of citing software in publications can be found in various scientific communities—notably, representative samples can be found in astronomy [52] and biology [50]. The group recommended collecting similar examples from other communities, and then developing a software citation principles document in concert with the FORCE11 Software Citation Working Group (see §6.7.5 for more details), following the model of the FORCE11 Data Citation Principles document [53].

The group further discussed briefly whether software used directly in a publication—whether to perform simulation or analysis, or as a dependency for newly developed software—should be distinguished from other references due to the dependence of the study on these research artifacts. Suggestions included a separate list of citations (with DOIs) for software and other research objects that serve this sort of “vital” role. Similar recommendations were made by the credit breakout group at WSSSPE2 [4].

Finally, although a discrete task from software citations, significant discussion focused on ensuring software citations are indexed in the same manner as publications, allowing the construction of a corresponding software citation network. Currently, software releases can receive citable DOIs via Zenodo [54] and figshare [55]; however, these citations are not processed by indexers such as Web of Science, Scopus, or Google Scholar. Thus, either in parallel or following the primary task, the group will need to reach out to these organizations. Initial conversations with Elsevier/Scopus via Michael Taylor during WSSSPE3 clarified that Scopus is not yet DataCite DOI aware, and also does not yet have an internal identifier for software or data (but needs/plans to add this support). Taylor said they prefer a “software article” with the usual article metadata (e.g., authors, citations), and mentioned Zenodo as an example – this proposal seemed to align with our group’s discussions. Taylor also mentioned another benefit of the software and associated DOI on GitHub: in addition to a citation, one could obtain statistics on usage/downloads/forks, which happens to be what Depsy⁵¹ is beginning to try to do.

6.7.3 Description of Opportunity, Challenges, and Obstacles

There currently is no standard mechanism for citing software or receiving credit for software (akin to citations for publications). Software is eligible for DOI assignment, but DOI metadata fields are not well tuned or standardized for software (vs. publications). Some software providers apply for DOIs, but this is not widely adopted. Also, there is no mechanism to cite software dependencies within software.

Major obstacles include the fact that indexers (e.g., Scopus, Web of Science, Google Scholar) do not currently support software/data document types or DataCite DOIs. Therefore, even with universal association of scientific software with DOIs and standardized practices for citing software in publications, software citations will not be indexed in the same manner as traditional publications.

Although this working group’s discussions at WSSSPE3 did not focus much on the topic of tenure and professional advancement, the group recognized that there is no standard policy—generally even within a single university—for software products to be included in promotion and tenure dossiers. Thus, it may be difficult to encourage valuing software contributions across the United States or United Kingdom and globally; furthermore, stakeholders are typically not tenured and thus may not be influential enough to change the status quo. However, as discussed in Section 5.2, this is changing for Research Software Engineers, at least in the UK.

6.7.4 Key Next Steps

1. Hold virtual meeting to determine group members responsible/willing to work on the following tasks, to be organized within one month of the workshop.
2. Compile best practices of software citation across multiple disciplines, including journals and communities of interest/practice in the research world, to begin by December 2015.
3. Compile examples of including other products in promotion and tenure dossier, to begin by December 2015.
4. Draft the Software Citation Principles document (including citation metadata file), by April 2016.
5. Publish/release the Software Citation Principles document, by August 2016.
6. Reach out to journals, publishers, teachers/educators, indexers, and professional societieslikely through meetings with key groups, to begin by September 2016.

6.7.5 Plan for Future Organization

The WSSSPE breakout group plans to join efforts related to citing software with the FORCE11 Software Citation Working Group (FORCE11-SCWG)⁵²; Kyle Niemeyer formally requested the merging of these groups following the meeting. However, some future plans of the WSSSPE group fall outside the scope of FORCE11-SCWG, which covers software citation practices. These activities include working with indexers such as Web of Science and Scopus to index software citations archived on, e.g., Zenodo or figshare, and pursuing the development of an open indexing service; such plans will be pursued either separately or through the formation of follow-on FORCE11 working groups.

The group will primarily communicate electronically, with Kyle Niemeyer responsible for ensuring regular progress.

6.7.6 What Else is Needed?

The near-term actions of the group, focused mainly on software citation, do not require any additional resources. However, connections with publishers and indexers will be needed to pursue related activities, although the FORCE11-SCWG may satisfy this need; in addition, some members of the group already reached out to relevant contacts. Funding may be needed to organize meetings or for group members to attend relevant meetings, as discussed further below.

6.7.7 Key Milestones and Responsible Parties

Following the meeting, Kyle Niemeyer formally requested the merging of software citation activities with FORCE11-SCWG. Within a month of the meeting, Niemeyer will organize a virtual meeting of the group and manage the division of responsibilities for compiling existing practices of software citation and including software/products in promotion and tenure dossiers. Building off of these efforts, the next major milestone is drafting the Software Citation Principles document in collaboration with the SCWG, targeted for April 2016. While the existing directors of the SCWG, Arfon Smith and Dan Katz, lead the efforts of that group towards the Software Citation Principles document, Kyle will help coordinate contributions from the WSSSPE group members.

6.7.8 Description of Funding Needed

Some funding would be useful to support primarily travel to conferences for group meetings (e.g., FORCE2016)⁵³, and to hold meetings to bring together both group members and key stakeholders (e.g., journals, publishers, professional societies, indexers). In addition, funding would be desired to support group members’ time to perform work towards the key steps described previously.

6.8.1 Group Members

• Steven R. Brandt – Louisiana State University
• Daniel Gunter – LBNL
• Yuhan Ding – Illinois Institute of Technology
• Neil Chue Hong – Software Sustainability Institute

6.8.2 Summary of Discussion

A tentative first cut at the list containing executable papers identified the following:

• ACM Transactions on Mathematical Software (TOMS): provides the additional step of having reviewers validate the code which was submitted with the publication.
• The Mathematica Journal: publishes Mathematica notebooks (with equations, figures, etc.) directly.
• O’Reilly Media: announced that it plans to make IPython Notebooks a first-class authoring environment for their publishing program alongside their existing mechanisms.
• Nature: offers a list of notebooks published alongside more traditional articles, and is looking at ways to make these documents more official. There are, in fact, a number of journals that offer “electronic supplements” to the more traditionally published static articles.
• IPython: maintained a list of “reproducible academic publications” [56].
• KBase: offers narratives built on IPython or Jupyter notebooks for assembling publications that are reproducible, and can be commented or annotated.

The group also discussed future possibilities for a new publication format that might provide advantages:

• Journals could be built around an existing, widely used framework thereby reducing the burden of studying code on the part of reviewers (common bits of infrastructure which are not relevant to the science would be automatically excluded).
• Journals might be encouraged to use more metadata, making them easier to mine for various analytical purposes.
• The Research Ideas and Outcomes (RIO) journal is an effort to publish fragmentary results that can subsequently be combined into a single content item.
• Papers could be made more understandable. Each equation or technical term could be linked to a document/tutorial explaining its origin and/or derivation.
• So many options for publication currently exist that good science may be getting lost in the noise. Would some sort of “upvote” mechanism be of value?
• Some sort of Replicated Computation Results badge could be made available to publications that have undergone greater scrutiny (this is already done by TOMS).

6.8.3 Description of Opportunity, Challenges, and Obstacles

The opportunity is to collect a list of executable papers and shine a light on the experiments and development efforts currently underway.

The only obstacle to this is the difficulty in finding and identifying such publications. The Software Sustainability Institute was able to do something similar for publications about software by making a public page on the Software Sustainability Institute’s website (http://www.software.ac.uk) containing a catalog of these publications and enlisting the help of the community to grow the list.

6.8.4 Key Next Steps

Create the first version of the web page to be displayed on the Software Sustainability Institute’s website: http://www.software.ac.uk. We expect the page to be live in early January of 2016.

An ongoing effort to update the page should follow.

6.8.5 Plan for Future Organization

None at this time.

6.8.6 What Else is Needed?

Nothing else at this time.

6.8.7 Key Milestones and Responsible Parties

Steven R. Brandt has created a first version of the page, and it is in the process of being posted on the Software Sustainability Institute’s website: http://www.software.ac.uk. Neil Chue Hong will take responsibility for the page once it is up.

6.8.8 Description of Funding Needed

None.

6.9.1 Group Members

• Ethan Davis – UCAR Unidata
• Dan Gunter – Lawrence Berkeley National Lab
• Liz Jessup – University of Colorado
• Mark Miller – University of California, San Diego
• Lindsey Powers – The HDF Group
• Daniel Ziskin – NCAR Atmospheric Chemistry Observations and Modeling (ACOM) Laboratory

6.9.2 Summary of Discussion

Discussion revolved around a few questions: what is the benefit of having a “community” for software sustainability, what practices and circumstances lead to having and maintaining a community, how can funding help or hinder this process, and perhaps most importantly, how can best practices be described and distilled into a document that can help new projects.

The benefits of having a community that were brought up were considered largely obvious. In addition to having advocates for the software, and a possible source of “free” contributions to the codebase, the community becomes a good source for requirements, feedback, and metrics. The software community can also act as “cheerleaders” who convince funders or other potential users to fund/use the software, and thus help sustain the software.

Practices and circumstances that lead to a community are first, that the software offers value. But in addition to this, a community will be much more likely to form if they receive (expert) support when they have questions. Additional contributing factors are good usability (not always needed), and an open development process such as IPython developer meetings on YouTube. It was also pointed out that an evangelist for the project, not necessarily but often one of the developers, can often make a big difference.

Funding can help the process by encouraging both value to the community and high-quality user support. Only providing funding for the software development may create good software, but with less likelihood to have a real community. It was discussed that federal laboratories are a good incubator for software communities, and that a general facility like EarthCube is too dispersed to really make a community. Also, domain-specific groups within laboratories or universities might provide as an incubator for software communities.

In describing best practices, the group discussed the different modes for starting a scientific software project: building on an existing product that needs improving, recognizing an unsatisfied need of an existing community, or creating a new solution to a need not yet recognized by the community. The group also thought that the existing books on software communities would need to be evaluated in light of differences between science software projects and general open-source software (OSS) projects in terms of scale, science, acknowledgement and credit, and funding models.

6.9.3 Description of Opportunity, Challenges, and Obstacles

The main opportunity is to increase awareness among scientific software developers and project managers of the importance of developing a community around their project. While this message is fairly well understood in the open source community, the scientific community can be more focused on the science a software project is supporting rather than the software project itself.

As with many of the issues relevant to the sustainability of science software, the main challenge here will be changing the culture and expectations around scientific software.

6.9.4 Key Next Steps

The most important next steps is a “Best Practice” document, which would describe what successful projects with engaged communities look like, how to replicate this type of project, and look at end-of-life on a community project. Inputs to this document would include a software community survey of highly functioning communities such as R Open Science, Python SciPy, OPeNDAP, and Unidata, with analysis of factors that feed into their success. Also references like the “Art of Community” could be adapted and summarized for the science software community.

More specifically, the group would like to take the following steps:

• Survey successful science software projects
• Survey community members from the surveyed projects
• Distill the survey results and document best practices around community engagement
• Look for ways to raise awareness

Another next step would be increasing recognition of need for science software projects to focus on building and supporting their user communities. Good software engineering practices are not enough, and popular training like Software Carpentry does not currently address this issue head-on.

6.9.5 Plan for Future Organization

No definite plans were agreed upon for future organization. The major ideas discussed were coordinating with another group or adapting some existing text.

Collaboration within the framework of an existing organization seems a good initial path. Mozilla Science maintains a “Working Open Project Guide” [39], the introduction of which states:

Working openly with contributors enables your community to learn how to build and collaborate together. This document is a guideline on how to work openly and involve others in your projects with Mozilla. We want to help you engage your community in a way that encourages contributors and builds other leaders.

Another idea is to form a group that could adapt existing commercial-oriented guidelines for the world of scientific software and top-down funding structures. For example, to distill the “Art of Community” by Jono Bacon [40] for scientific software.

6.9.6 What Else is Needed?

The group had many points of agreement, but there is not currently a dedicated core group of people who have committed to producing the key milestones. Coordination via phone or online would be necessary to build this “community” of contributors.

6.9.7 Key Milestones and Responsible Parties

The key milestones for the group’s activities align closely with the Key Next Steps above:

• Complete and write up a survey of project members, and community members, for successful science software projects
• Distill the survey results and document best practices around community engagement

6.9.8 Description of Funding Needed

With a small amount of seed funding, it is possible that members of this group or other parties could spend the time necessary to devise a survey of existing projects and deploy this, probably by traveling to meetings and workshops for the various software communities.