Report on the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)

This report records and discusses the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). The report includes a description of the keynote presentation of the workshop, the mission and vision statements that were drafted at the workshop and finalized shortly after it, a set of idea papers, position papers, experience papers, demos, and lightning talks, and a panel discussion. The main part of the report covers the set of working groups that formed during the meeting, and for each, discusses the participants, the objective and goal, and how the objective can be reached, along with contact information for readers who may want to join the group. Finally, we present results from a survey of the workshop attendees.


I n t ro d u c t i o n
The Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)1 was held over 2 1/2 days on 12-14 September 2016 in Manchester, England.This location and date was selected so that WSSSPE4 immediately preceded the First Research Software Engineers (RSE) Conference.Previous events in the WSSSPE series include WSSSPE12 [1,2], held in conjunction with the SC13 conference; WSSSPE1.1 3 , a focused workshop organized jointly with the SciPy conference 4 ; WSSSPE25 [3,4], held in conjunction with SC14; WSSSPE2.1 6 , a focused workshop organized again jointly with SciPy7 ; and WSSSPE38 [5], held in Boulder, Colorado, USA.Note that the first WSSSPE workshop was named "Working towards Sustainable Software for Science: Practice and Experiences," which remains the meaning of the WSSSPE group, but the workshops after that were named "Workshop on Sustainable Software for Science: Practice and Experiences."Together these reflect that WSSSPE is both a community and a set of workshops.
The WSSSPE4 workshop included multiple mechanisms for participation and encouraged team building around solutions.WSSSPE4 strongly encouraged participation of early-career scientists, postdoctoral researchers, graduate students, early-stage researchers, and those from underrepresented groups, with funds provided to the conference organizers by the National Science Foundation (NSF), the Gordon and Betty Moore Foundation, the Alfred P. Sloan Foundation, and the Software Sustainability Institute (SSI) to support the travel of potential participants who would not otherwise be able to attend the workshop.These funds allowed 29 additional people to attend and participate.A subset of the organizing committee reviewed 44 applications for travel support and competitively selected the awardees, including 11 students and five early-career researchers.In addition, the travel award subcommittee tried to increase the diversity of applicants by forwarding the notice of travel support to organizations including PyLadies9 , Django Girls10 , Women Who Code 11 , and Women in HPC 12 .
WSSSPE4 also included two professional event organizers/facilitators from KnowInnovation who helped the organizing committee plan the workshop agenda, and during the workshop, they actively engaged participants with various tools, activities, and reminders.
At the workshop, we also introduced a Code of Conduct (CoC). 13The CoC was conceived for the workshop itself; however, we intend it to also be a guideline for the community of scientists that WSSSPE supports, and their personal and online interactions (e.g., on Twitter, in email lists, in the Slack team).The WSSSPE4 CoC is based on the FORCE11 conference CoC [6], in turn based on the Code4Lib CoC [7].The main guidelines of the CoC are: WSSSPE events are community events intended for networking and collaboration as well as learning.We value the participation of every member of the community and want all attendees to have an enjoyable and fulfilling experience.Accordingly, all attendees are expected to show respect and courtesy to other attendees throughout the event and in interactions online associated with the event.
The WSSSPE event organizers are dedicated to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, age, sexual orientation, disability, physical appearance, body size, race, ethnicity, religion (or lack thereof), technology choices, or other group status.
To make clear what is expected, everyone taking part in WSSSPE events and discussions-speakers, helpers, organizers, and participants-is required to conform to the following Code of Conduct.
• Communicate appropriately for a professional audience including people of many different backgrounds.Sexual language and imagery are not appropriate for any event.
• Be kind to others.Do not insult or put down other attendees.Be mindful of jargon, which can sometimes exclude others from engaging in the discussion.• Behave professionally.Remember that harassment and sexist, racist, ageist, or exclusionary behavior are not appropriate.The CoC was introduced at the beginning of WSSSPE4, with the CoC subcommittee and a general email address introduced for reporting concerns or incidents, or asking questions.There was one concern mentioned to the CoC subcommittee after the first half day, regarding how part of the workshop was being run, and we changed the workshop to address this.
This report is based on the events at the workshop and the submitted materials.These events included discussion of the call for papers ( §2), the WSSSPE mission and vision ( §3), a keynote ( §4), a set of presentations ( §5 and §6), a panel ( §7), and a number of working groups ( §8).One full day of the workshop was spent with participants in the working groups, which occurred in parallel with each other.And each of the working groups left with a plan for how they could move forward.This report also mentions the Slack channel created for further discussions ( §9), and it highlights an attendee survey ( §10) before concluding ( §11).Appendices list the organizers ( §A), attendees( §B), travel award recipients ( §C), and program committee ( §D), and detail the attendee survey ( §E).

C a l l f o r pa rt i c i pat i o n
WSSSPE4 was based on the work done at WSSSPE1, WSSSPE2, and WSSSPE3, but aimed at producing working groups that better continued working after the workshop ended.In addition, based on feedback after WSSSPE3, it became clear that WSSSPE attendees had two different motivations in participating.One motivation was to make a better future for research software, and the other was to immediately do better research software development.This led to the idea of WSSSPE4 being partially divided into two tracks: Track 1 -Building a sustainable future for open-use research software has the goals of defining a vision of the future of open-use research software, and in the workshop, initiating the activities that are needed to get there.The idea of this track is to first think about where we want to be 5 to 10 years from now, without being too concerned with where we are today, and then to determine how we can move to this future.Track 2 -Practices & experiences in sustainable scientific software has the goal of improving the quality of today's research software and the experiences of its developers by sharing best practices and experiences.This track is focused on the current state of scientific software and what we can do to improve it in the short term, starting with where we are today.The call for participation requested idea papers, which present implementable proposals related to Track 1; position papers, which are longer, not previously published papers related to Track 2 specifically discussing what we can do to improve sustainable scientific software in the short term, starting with where we are today; experience papers, longer papers related to Track 2 that discuss current practices and experiences and how they have been used to improve the quality of today's research software and/or the experiences of its developers; demos, brief papers describing a tool or process that would be demonstrated, relevant to Track 2 that improves the quality of today's research software and/or the experiences of its developers; and extended abstracts describing lightning talks, where each author could make a brief statement about work that either had been done or was needed.
The following list of contribution topics was suggested, though this was not intended to limit potential submissions: • Development and Community -Best practices for developing sustainable software -Models for funding specialist expertise in software collaborations -Software tools that aid sustainability -Academia/industry interaction -Refactoring/improving legacy scientific software -Engineering design for sustainable software -Metrics for the success of scientific software -Adaptation of mainstream software practices for scientific software -Reproducibility in conferences and journals -Best practices for code testing and code review Submissions to WSSSPE4 comprised 19 lightning talks, 4 idea papers, 3 position paper, 5 experience papers, and 3 demos.Most of the submissions were accepted, since the goal of WSSSPE is always to take in as many relevant inputs as possible, and to encourage their authors to participate in sharing and implementing their ideas.Specifically, 19 lightning talks, 4 idea papers, 3 position papers, 2 experience papers, and 3 demos were accepted.(Two of the submitted lightning talks and one submitted experience paper were rejected, while two more submitted experience papers were accepted as lightning talks.)The papers are discussed in Section 5, and the lightning talks are discussed in Section 6.The papers and lightning talks have been published as a volume in the CEUR Workshop Proceedings [8].

M i s s i o n a n d v i s i o n
Going into WSSSPE4, the WSSSPE organization had not had a formal mission or vision statement.The organizers developed a strawhorse, which was presented to the participants early in the meeting.The presentation included guidelines as well as examples from other similar communities.The guidelines were that in general, an organization's mission should stand the test of time and state what is wrong and how the organization is going to fix it; its vision should imagine what the world would look like if the organization is successful; and from which, focus areas could be used to establish the scope of the organization along with its goals.
The participants were given time after the presentation to write down their comments.Seven people volunteered to work on revising the mission and vision statements, incorporating these comments and presenting back to the community.Based on this feedback, the committee redrafted the mission and vision.The committee added focus areas to clarify the organization's breadth and to keep the mission and vision simple and long-lasting.Comments after the close of the meeting were incorporated into the statements.
Mission.WSSSPE is an international community-driven organization that promotes sustainable research software by addressing challenges related to the full lifecycle of research software through shared learning and community action.
Vision.We envision a world where research software is accessible, robust, sustained, and recognized as a scholarly research product critical to the advancement of knowledge, learning, and discovery.
Focus areas.WSSSPE promotes sustainable research software by positively impacting: • acquiring and assembling resources (including funding and people) into teams and communities • developing software • using software • recognizing contributions to and of software • maintaining software

K e y n o t e
The keynote was given by Patricia Lago and entitled "The legacy of unsustainable software".Sustainability is broadly associated with natural ecologic systems.When we translate the notion of sustainability to software solutions, however, we often confuse impact in a certain point in time with balanced and durable effects.In addition, software sustainability adds a fourth dimension to environmental, social and economic aspects: technical sustainability, and hence higher complexity [9].
Supporting technical sustainability in (scientific) software is very near to supporting properties like longevity and resilience, i.e., the ability of a software system or application to fulfill its intent [10], as planned independently from time passing or from changes in its operational context.Technical sustainability has never been easy, but it is even more of a challenge nowadays with the increasing demands on software solutions to support many aspects of our daily lives.
The keynote discussed the challenges (and some related ongoing research) of combining technical and environmental sustainability.This provided a complementary angle to the workshop discussions.It addressed software energy efficiency (i.e., how to develop software that consumes less energy, like mobile apps that should consume less device-battery) and software energy-awareness (i.e., how to develop software that can actively decide between energy efficiency and other goals such as performance).
The keynote used a few myths to elaborate on such challenges and related research.Among them: Myth #1: Energy-efficient hardware will solve the issue.At the implementation level, to make software use resources effectively, we need to first understand the impact of different coding practices on the way software exploits computing resources (hence energy to feed them).
In addition, when we have to deal with complex software or new projects, we must understand "the big picture," hence we must reason at the design or architecture level.Maybe the software is not implemented yet, or maybe we must identify which design decisions are more or less beneficial for energy consumption.To this aim, we capture best practices in reusable patterns and so-called architectural tactics.These document generic designs, specific implementations, metrics, and measures.
Myth #2: Extensive experimentation will provide the needed know how.While empirical methods to design and execute software engineering experiments are mature and well known, they are insufficient to handle the too many variables and uncertainties for modern software to be sustainable.We need new empirical methods that help us gather evidence faster [11].This would probably entail relaxing the too many constraints that govern current rules for mitigating threats to validity, and (equally importantly) collaboration-and experience-sharing to build a common knowledge base.In addition, we should exploit better (and more truthful) visualizations of experimentation results, to communicate on advances (and less promising directions.) Myth #3: With massive migration of software systems to cloud-based environments, we do not need to invest in a "greener cloud" anymore.The use of cloud-based services promises large-scale optimizations in terms of the resources used by our software applications and related investments [12].Unfortunately, cloud provisioning heavily relies on over-capacity rather than intelligent governance, and so far has invested very little in innovation for sustainability.We need to build smarter software that both manages resources more efficiently in the cloud, better understands the changes in the software operational context (including user requirements), and dynamically adapts to address sustainability intents [13] (both in the cloud and on customer's premises.)

Pa p e r s a n d d e m o s
WSSSPE4 included the presentation of 12 10-minute talks (based on four idea papers, three position papers, two experience papers, and three demos) that addressed a wide range of topics around sustainability for software in science.These talks covered three main areas: (1) The specifics of the academic environment for software development and for so-called research programmers or research software engineers; (2) The characteristics and needs of research software in general such as the lifecycle of software in science, preservation, curation, sharing, and reproducibility challenges; and (3) Lessons learned from or visions for use cases of scientific software and communities working with and/or on a software package.The following four papers belong to the first area: the academic environment.They address diverse aspects ranging from incentives for quality software to advocating a professional society for research software to roles and degrees for research software engineers.For talks with multiple authors, an * indicates the author who presented the talk.
• Michael Heroux: Idea paper: Sustainable & Productive: Improving Incentives for Quality Software [14].The author presents that improving quality of scientific software is a measure to address the increased complexity in both modeling and software.To support quality, it is essential to provide Computational Science and Engineering teams with publication venues, funding, and professional recognition.Broad improvements in sustainability and productivity can be only achieved when publishers, funding agencies and employers raise their expectations for software quality.Additionally, software community leaders are in the unique position to positively impact software quality by working on establishing incentives that will spur creative and novel approaches to improve developer productivity and software sustainability.• Gabrielle Allen: Idea Paper: Establishing a Professional Society for Research Software [15].This paper advocates to create a professional society or extend the scope of an existing organization in research software.The author presents the motivation, a variety of potential roles of such a society and diverse potential funding mechanisms to be able to establish a sustainable organization.She gives an overview on similar activities as well as existing organizations and aims at stimulating discussion among potential members and stakeholders.• Olivier Philippe * , Simon Hettrick, and Neil Chue Hong: Experience Paper: Preliminary analysis of a survey of UK Research Software Engineers [16].This paper presents results from a survey conducted on the new role of research software engineers in academia.The survey provides demographic information about the education, field, gender, job satisfaction, and career plans of the people in this community.The outcome is that members of the community were found to be highly educated, derive mainly from the hard sciences, and are predominantly male.Respondents reported satisfaction in their jobs, but indicated that career progression is both difficult and opaque.The authors urge a continued discussion about the experience of research software engineers and recommend further investigation into this important community.• Christopher Gwilliams: Demo: Using industrial engagement to create and develop research ties within academia [17].This demo focuses on the benefits of including a new software engineering degree that is influenced by, and works directly with, industry.Cardiff University introduced a new industry-focused degree, Bachelor of Science in Applied Software Engineering (ASE) in September 2015.The engagement with industry allows students to learn through live projects with real-world applications and forge links for career opportunities.Also, industry benefits from this course that provides an entry point for industry to engage with the university and to develop a network of contacts between industry and academia, which is also a benefit for the university and allows to identify potential opportunities for new research projects and collaborators.
The second main area, concerned with characteristics and needs of research software, includes the following five papers: • Anshu Dubey * and Katherine Riley: Experience Paper: Software Engineering and Community Codes Track in ATPESC [18].The Argonne Training Program in Extreme Scale Computing (ATPESC) was started by Argonne National Laboratory with the objective of expanding the ranks of better prepared users of high-performance computing (HPC) machines.One of the unique aspects of the program was inclusion of a software engineering and community codes track.The inclusion was motivated by the observation that the projects with a good software process were better able to meet their scientific goals.The paper presents the experience in running the software track, the motivation, reception, and evolution of the track over the years.The authors recommend the inclusion of similar tracks in other HPC-oriented training programs.• Neil Chue Hong: Position Paper: Why do we need to compare research software, and how should we do it?[19].This position paper sets out the reasons for why different stakeholders, from users to developers to funders, might wish to undertake the difficult task to compare research software.The author describes a proposed framework for doing so (based around measures of accessibility, usability, maintainability, and portability) which takes into account the possibility of variation between different communities about how they prioritize different aspects of research software.
• Anshu Dubey * and Lois Curfman McInnes: Idea Paper: The Lifecycle of Software for Scientific Simulation [20].The authors discuss the specifics of the lifecycle of scientific software.While the software lifecycle is a well-researched topic, one class of projects, research software, has received relatively little attention from lifecycle researchers.This paper formalizes a lifecycle model for end-to-end scientific application software, featuring loose coupling between submodels for development of infrastructure and scientific capability.The authors also invite input from stakeholders to converge on a model that captures the complexity of this development processes and provides needed lifecycle guidance to the scientific software community.• Francisco Queiroz * and Rejane Spitz: Position Paper: Collaborative Gamification Design for Scientific Software [21].The paper focuses on usability design and presents that gamification, a design trend for improving scientific software usability, for scientific software should be facilitated by an open, collaborative design process supported by conversational media.This approach is compatible with qualities often attributed to computational science community regarding openness and collaboration between members of varied professional backgrounds.They exemplify the use of conversational media for collaborative design and expect the synergy between collaborators to result in better usability, greater user acceptance, and adequacy to requirements, obtaining optimal design solutions in a sustainable way.[22].The paper presents the implications of "active curation systems" that provide executable access to artifacts and experiments behind published results and enables their reuse.These systems allow changing and repeating experiments to understand how an innovation behaves in conditions beyond the ones described in a paper and address reproducibility challenges.Active curation systems also enable accountability and accelerate research progress by giving access to complete experimental details.As these systems take hold, it is important to understand their capabilities and how digital libraries should be integrated with them.The authors describe a study underway to explore how best to do this integration.
Three contributions focused on the third main area: elucidating lessons learned from or visions for use cases of scientific software and communities working with and/or on a software package.
• Debashis Ganguly, William C. Garrison III, David Wilkinson, Bruce R. Childers, Adam J. Lee, and Daniel Mosse * : Demo: Composing, Reproducing, and Sharing Simulations [23].The authors demonstrate their approach of sharing, reproducing, and composing simulations toward accelerating research productivity while also improving accountability and credibility.Specifically, they have developed a case study in which they compose and share access control simulations in the form of shareable data store units for cloud systems.They share, compose and repeat simulations through a collaborative repository (OCCAM) and suggest a general simulation framework (SST) that can accelerate the efforts for the whole community.• Sameer Shende * and Allen Malony: Demo: Using TAU for Performance Evaluation of Scientific Software [24].This demo presents the TAU Performance System for performance evaluation of Scientific Software written in C++, C, and Fortran.The system addresses performance technology problems at three levels: instrumentation, measurement, and analysis.The TAU framework supports the configuration and integration of these layers.The authors advocate that effective exploration of performance will necessarily require prudent selection from the range of alternative methods TAU provides to assemble meaningful performance experiments.• Janos Sallai * , Christopher Iacovella, Christoph Klein, and Tengyu Ma: Idea Paper: Development of a Software Framework for Formalizing Forcefield Atom-Typing for Molecular Simulation [25].The authors present an essential and error-prone task in the area of Molecular Dynamics simulations: the efficient selection of correct parameter values for specific molecules, especially forcefields.They present a framework that aims to solve this data management issue, proposing a common format for forcefields that is selfdocumenting with machine readable, declarative usage rules.They investigate processes and tools that are commonly used today in software development (e.g., unit testing, verification and validation, continuous integration, and version control) and reason that they are, with proper infrastructure support, applicable to forcefield development, as well.The paper describes how such an infrastructure can tackle managing and evolving forcefields by the molecular dynamics community, and proposes a way to encourage and incentivize involvement by the stakeholders.

L i g h t n i n g ta l k s
There were 19 lightning talks presented at WSSSPE4, as follows (in order of presentation).For talks with multiple authors, an * again indicates the author who presented the talk.
• Shoaib Sufi: The Software Sustainability Institute Fellowship programme; supporting the social side of research software [26].Sufi, Community Lead at the Software Sustainability Institute (SSI), discussed the SSI Fellowship program and the fellows, including their demographic breakdown and domains of practice.This program came about based on the institute's desire to integrate with the research community-thus inspiring change from within-rather than preach to it.Five SSI fellows attended WSSSPE4.
• Sandra Gesing * , Maytal Dahan, Linda B. Hayden, Katherine Lawrence, Suresh Marru, Marlon Pierce, Nancy Wilkins-Diehr, and Michael Zentner: The Science Gateways Community Institute -Supporting Communities to Achieve Sustainability for Their Science Gateways [27].In this talk, Gesing introduced the concept of science gateways: end-to-end software solutions for a community that hide underlying complexity.She also posed the need for gateways, which include the increased complexity of research and associated hardware/software, a greater need for reproducibility and openness, and the opportunity to integrate research with teaching for better workforce development.• Karthik Ram * , Noam Ross, and Scott Chamberlain: A model for peer review and onboarding research software [28].This talk described the rOpenSci community's review process for research software packages.Submission of a package starts with a pre-submission inquiry for fit with the community criteria, then peer reviewers evaluate based on usability, quality, and style.Upon acceptance, packages are given an rOpenSci README badge and added to the system.Important components include a code of conduct, open and non-adversarial review, and that no submission is rejected-just out of scope.Remaining challenges include scaling, author incentives, and automation of the process.
• Frank Löffler and Steven Brandt * : A vision of computing in 10+ years [29].This talk began by describing the challenges facing science, and particularly computational science, in recent years: the amount and complexity of knowledge has increased dramatically, making it difficult for individuals or even teams to manage modern, complex problems.Brandt submitted an argument for decoupling science simulation and the solution of computational problems away from numerical methods and low-level optimization.This may be possible with domain-specific languages, or the use of modern features in existing languages.This vision also includes reuse of basic computing infrastructure, including common data structures, parameter files, and highperformance I/O.This computing future necessarily requires better recognition and reward systems for research software developers (i.e., research software engineers), and ways to make increasingly complex scientific papers more understandable.
• Daniel S. Katz, Kyle E. Niemeyer * , Arfon M. Smith, and the FORCE11 Software Citation Working Group: Software Citation: Process, Principles, and Implementation [30].This talk described the timeline of activities of the FORCE11 Software Citation Working Group and the principles of software citation established by the group.Niemeyer, one of the working group co-chairs, also discussed the paper published describing the principles [31], and the remaining steps of the working group before sunsetting it and spinning up a new Software Citation Implementation Working Group.
• Ray Idaszak  [32].HydroShare is a five-year-old online, collaborative system that supports open sharing of hydrologic data, analytical tools, and computer models.The project involves 10 collaborating institutions that produce tested and reviewed community code every two to three weeks, following software engineering best practices.Undergraduate and graduate students involved in the project not only adopt the best practices exhibited by HydroShare, but see the open, collaborative approach to research software development as the norm.The talk made recommendations to other community efforts, including the need to identify committed, hands-on PIs; evidence-based software engineering for science; continuous integration; the importance of professional software development and operations practices, even for domain scientists; and to identify technology decisions that overlap with career paths.
In this talk, Hwang first described the ASPECT (Advance Solver for Problems in Earth Convec-Tion) open-source, parallel, extensible finite element code used to simulate thermal convection.ASPECT began development in 2011 as a community-driven project under the Computational Infrastructure for Geodynamics (CIG).Community-building activities around ASPECT focused on the traditional workshops, tutorials, and webinars, but also on slightly nontraditional, multi-day hackathons-which turned out to be one of the most effective forms of fostering a community.These start with leaders establishing project conventions, demonstrating workflows and relevant technical skills, and identifying starter projects.Daily activities involve work plans and science/coding briefs, with mini-tutorials given as needed.One important requirement is that each participant must submit at least one commit.Hwang also summarized lessons learned, including that hackathons should last 7-10 days, include participants with multidisciplinary expertise and at both novice and expert levels, have at least one dedicated code reviewer per 10 participants, hold daily group updates, organize activities to promote group interactions, and enforce free time.
• Carole Goble: A Simple Profiling Framework for Software User-Producer Reciprocity Review [34].Goble first described some existing frameworks for measuring the maturity of software, including the Software Sustainability Maturity Model, OSS Watch Openness Rating, NASA Reuse Readiness Rating, Capability Maturity Model, and Qualification and Selection of Open Source software.Then, she proposed a new, simpler framework for project reciprocity, drawing on the intentions of users and producers of the software, from the viewpoint of the producer.Next steps include further definition of the model's characteristics based on both theoretical and empirical investigation.
• Stephan Druskat: A proposal for the measurement and documentation of research software sustainability in interactive metadata repositories [35].In this talk, Druskat first identified two technical barriers to software sustainability: identification of good software, and software discoverability.Some of his perspectives were based on experiences in the humanities, a lack of funding for computational infrastructure results in little experience with sustainability (although this may differ in other domains).Druskat proposed metadata repositories as a possible solution, as they can measure and document technical sustainability, and document resulting metrics and more general features of software.Technical sustainability goals include ensuring software existence, preserving potential operation of software, and creating/ retaining possibilities for further development and maintenance of software.The next steps of such a system include compiling measurable criteria for the sustainability goals (based on qualitative concepts of, e.g., "sustainability," "maintainability," and "usability"), and crowd-sourcing a weighting of these criteria.
• Mistral Contrastin, Matthew Danish, Dominic Orchard * , and Andrew Rice: Supporting software sustainability with lightweight specifications [36].This talk introduced the tool CamFort for verifying Fortran code based on lightweight specifications, which aid in software maintainability and reuse through high-level indications of programmer intent.Currently, CamFort supports specification (and verification) of units and stencil operations.Plans for future specifications include automatic test generation based on user-supplied properties, dependency specifications, and software contracts to check, e.g., expected value ranges and program behavior.
•  [37].This talk described the activities of the weeklong Dagstuhl Perspectives Workshop manifesto on "Engineering Academic Software," which produced a manifesto describing a pledge affirming the scholarly value of research software and the personal responsibility to act in accordance with that value.The draft (personal) manifesto includes 20 pledges, organized into three areas: recognition of academic software, academic software development processes, and the intellectual content of academic software.The talk concluded with a survey for participants to give feedback on the pledges and their importance.
• Neil Chue Hong * : Making it easier to understand research software impact [38].Chue Hong proposed that the WSSSPE community collaboratively develop a framework to better understand (and measure the impact of) research software.This could be based on a workflow involving mining the literature for mentions of software, identifying linkages between software, and visualizing dependencies between software or between software and other research artifacts.Many tools already exist in this ecosystem, so Chue Hong proposed that a WSSSPE working group collect these tools, summarize use cases, identify missing pieces, and find lowhanging fruit to integrate tools or build new ones.
• Willem Robert van Hage * , Jason Maassen and Rob van Nieuwpoort: Software Impact Measurement at the Netherlands eScience Center [39].This talk described eStep (eScience Technology Platform) at the Netherlands eScience Center, which includes three components: a software catalogue of eScience software, interfaces, libraries, tools workflows, applications, etc.; a knowledge base of guides, reports, and recommendations for scientific software development; and eScience research documents including such as scientific publications, book chapters, and demos.The eStep website currently presents automatically-generated information about software based on metadata, with a plan for displaying measures of impact and quality.
• Gabrielle Allen, Emily Chen * , Ray Idaszak, and Daniel S. Katz: Report on Software Metrics for Research Software [40].In this talk, Chen first summarized some of the outcomes of the WSSSPE3 Metrics Breakout Group, who discussed that metrics are important for scientific impact, discovery, reducing duplication of efforts, and evaluating software.The group also summarized barriers to widespread use of metrics: no common set of metrics collected, a need for usefulness to be obvious for projects, and no common standard for collecting/ presenting metrics.To find out more about practices related to metrics, the authors ran a survey of National Science Foundation Software Infrastructure for Sustained Innovation (SI2) program investigators, with 19 participants.Important results include the observation that most use citations and acknowledgements of software in funding proposals, while download numbers are both challenging to collect and not useful.Overall, participants generally agreed that metrics are important, but said useful metrics are difficult or tedious to collect.
• Eric L. Seidel * and Gabrielle Allen: Bringing Techniques from Software Engineering into Scientific Software [41].This talk, given by a senior graduate student in Computer Science/programming languages research, described several areas of software engineering concepts that benefit the development of research software.In particular, Seidel focused on software correctness, starting from high-level specifications.Correctness can be demonstrated by verifying or proving that the software satisfies the specification, testing against a range of inputs, or synthesis of code from the specification.However, all of these present technical challenges, in addition to the social challenges of encouraging collaboration between software engineering and research software communities.
• Iain Emsley * and David De Roure: Sustaining the social: Connecting the lives of Drupal community [42].In this talk, Emsley discussed efforts to develop the community around the Drupal content management system, which is supported through the DrupalCon series of conferences, user groups, and an association.The talk described the use of social coding and software credit to develop the Drupal software community, and social documentation to provide documentation and tutorials.By analyzing identities of committers, they found that the larger community is actually a loosely coupled set of heterogeneous communities that use (and sustain themselves) through different social mechanisms.
• Jack Dongarra, Sven Hammarling, Nicholas J. Higham, Samuel D. Relton * , Pedro Valero-Lara, and Mawussi Zounon: Creating a Standardised Set of Batched BLAS Routines [43].This talk described Batched BLAS (BBLAS), a new standard for computing multiple BLAS (Basic Linear Algebra Subproblems) in parallel.Unlike existing parallel BLAS algorithms provided by, e.g., Intel MKL, cuBLAS, or MAGMA, BBLAS is optimized for smaller matrices, which apply to a variety of problems in areas such as machine learning, multifrontal solvers, image processing, fluid dynamics, and astrophysics.BBLAS offers nearly a factor of two improvement in computing speed compared to non-batched operations.To develop a sustainable BBLAS library, the developers have focused on proposing a standard API, holding workshops between academics and vendors to reach consensus, and performing research into API design decisions on performance.
• Lukas Breitwieser * , Roman Bauer, Alberto Di Meglio, Leonard Johard, Marcus Kaiser, Marco Manca, Manuel Mazzara, Fons Rademakers, Max Talanov: The Bio-DynaMo Project [44].Breitwieser described the BioDynaMo project, a large-scale platform for biological simulations that provides access to computational resources while hiding the complexity of distributed computing.It also promotes computational reproducibility through shared open-access data.The project began as a modernization initiative porting an existing simulation software, Cx3D, from Java to C++, but also incorporated many modern software engineering practices such as hosting and version control on GitHub, automated tests and continuous integration via Travis-CI, code coverage reports, and an automated build system.In addition, over 1000 students participated in development via the Intel Modern Code Developer Challenge competition-the winner achieved a performance speedup of 320 times.The project plans to continue improving the development process with, e.g., static code checkers and Git commit hooks, continued architecture redesign, and verification.
• Aseel Aldabjan, Robert Haines, and Caroline Jay * : How should we measure the relationship between code quality and software sustainability?[45].This talk explored the relationship between code quality and the sustainability of research software; however, the latter has no concrete definition, thus making it challenging to provide guidance on how to achieve sustainability.The authors proposed a "software sustainment" metric based on the time difference between the most recent and initial commits/contributions to the codebase.Using this metric, they analyzed 3113 Java projects hosted on GitHub, created in 2009, and found that 22% and 35% had sustainment metrics of 0 and less than seven, respectively.Interestingly, the authors also found high code quality (measured using a variety of static analytic metrics) across all these projects, with little correlation to sustainability.

B e s t p r ac t i c e s f o r s c i e n t i f i c s o f t wa r e pa n e l d i s c u s s i o n
To investigate best practices for scientific software we brought together a panel of five experts with different perspectives on "Best Practices for Scientific Software."These were: Alice Allen, Editor, Astrophysics Source Code Library, who brought an understanding of the difficulties of organizing a community and curating their software; Mike Croucher from the University of Sheffield and Rob Haines from the University of Manchester who are both Research Software Engineers with decades of experience of writing code for researchers; Patricia Lago from Vrije Universiteit Amsterdam, who presented a keynote at the workshop ( §4) and brought a fresh perspective on software sustainability from the point of view of its impact on society and business; and Karthik Ram from the Berkeley Institute for Data Science, who brought his perspective on the practicalities of using scientific software to conduct his research as a data scientist.
The panel opened with a question to the floor "Who in the audience still codes in anger?" which caused some confusion as, it turns out, that this is a turn of phrase that has not crossed the Atlantic.After translating this question into the more open, "Who in the audience still writes code as part of their day-to-day work?" we were able to confirm the suspicion that the WSSSPE audience is heavily involved writing, not just using, code.Hence, the panel were going to have to deal with questions from a highly experienced audience.
The panel opened by noting that the world of scientific computing is changing rapidly.Each panelist was asked for their views on practices which had recently emerged and, if not handled correctly, would become the basis for the next generation's software problems?This quickly led to a discussion about the dangers of containerization.These technologies are pushed by some-a minority it is hoped-as a panacea for software sustainability.The panel and attendees generally agreed that containerization is useful, but should not be seen as the solution to sustainability.After all, these technologies are just as liable as any to decay.
A later question looked at the common practices from software engineering that the scientific community should be using.The discussion that followed picked up on the key elements of basic sustainable practice: version control, documentation and ensuring that communities are active and engaged.When asked whether there was resistance from the research community to adopting these practices, few negative experiences were reported.From the panel's point of view, it would appear that the problem in getting researchers to employ sustainable practices lies more in getting the message out to researchers, rather than selling them on the benefits.
The panel rounded up with a final question about "If you wanted to have the most disastrous effect on scientific software, what one practice would you advise people to keep doing?"A few good responses were discussed, but the biggest laugh went to Mike Croucher who said "Containerize everything!" 8. Wo r k i n g g ro u p s After most of the lightning talks and other presentations, WSSSPE used three areas of a large room to let attendees use sticky flip charts and sticky notes on the walls to suggest (1) A vision on any aspect of the work.
(2) A gap or challenge (3) A project idea (with one area of the room for each topic.) Once we had the walls covered in ideas, we had people organically form groups around the flip charts on the wall 14 .For those folk who worked on Vision and Gaps, after an hour, we encouraged them to finish their Vision or Gap Google Doc, and to find and join or start working on a project with a group.This led to the 12 groups that are discussed in the rest of this section, specifically: (1) Verifying best practices & metrics for sustainable research software (2) Software Sustainability Alliance (3) Scientific Software Prototyping Infrastructure (S2PI) (4) Standard metadata for software (CodeMeta) (5) White paper on developing sustainable software (6) Social science for scientific software (7) Software best practices for undergraduates (8) Meaningful metrics for sustainable software (9) Coordinating access to CI for research software (10) Software engineering processes tailored for research software (11) Open research index (12) Letters of evaluation for computational scientists The groups worked in Google docs, and eventually created Google slides to show their work back to the larger groups.The docs and slides were stored in a common Google Drive folder 15 .In each group's discussion, a number of Specific, Measurable, Achievable, Relevant, and Timely (SMART) steps were created, and ideally, at least the first one (to set up some communication mechanism) was completed.In many cases, this was done through Slack (see §9. • Ray Idaszak <rayi@renci.org>• August Muench <august.muench@aas.org>• Jonah Miller <jmiller@perimeterinstitute.ca>• Lorraine Hwang <ljhwang@ucdavis.edu>8.1.2.Working group objective.The overall objective of the Verifying Best Practices and Metrics for Sustainable Software working group is to take the outputs of the WSSSPE efforts to identify best practices for creation of sustainable software for science/academia, and also the outputs of WSSSPE efforts to identify metrics for sustainable software for science/academia and cross-reference these with current open source research software that successfully uses modern software engineering.This will allow the group to identify gaps on both sides.This approach will also allow the group to hypothesize how successful open source projects can be further improved, verify that recommended Plans.The group's plans are for each of group members to volunteer to take on one to two projects.As an incentive, the projects that each individual takes on should be ones in which they are already interested.This would lead to an estimated 5-10 projects to be evaluated within this project.In terms of increasing this sample size, the group hopes that once it documents the workflow by which these evaluations are performed, others can be self-sufficient in following this workflow and contribute their own evaluations of software they wish to evaluate.• Document workflow of getting to these results so it can be repeated for additional domains.8.1.7.More information & joining instructions.To join this group or obtain more information about it, please send an email to the Verifying best practices & metrics for sustainable research software group: <wssspe4-verify-best-practices@googlegroups.com>.8.2.Software Sustainability Alliance.The Software Sustainability Alliance working group aims to establish an alliance between organizations interested in improving the sustainability of research software.Such an alliance would ease the collaboration between member organizations to improve the sharing of expertise, resources and best practices.It is currently seeking feedback on potential member organizations, as well as the aims and scope of this alliance.An alliance of software sustainability organizations would ease inter-organization collaboration and the promotion of software sustainability.This alliance would also improve the pooling of competencies and the sharing of expertise.Furthermore, with the international scope of this alliance, it could support an organization who wants to hold an event in a country where another organization exists.

Scientific Software Prototyping Infrastructure (S2PI).
There is a productivity bottleneck-yet to be solved-in HPC from the human perspective, first identified in the 1980s [49].Of all domain-specific scientists, only a percentage of those use simulations due to perceptions mostly related to the difficulty of using and developing those tools.XSEDE and other resources have become a gateway for successfully increasing the basis of HPC scientific users by facilitating their access to infrastructure and tools [50], but the development challenge remains unaddressed.Of all scientists who use simulations, only a small percentage know how to develop prototypes or full applications that can be later scaled by computer scientists and engineers.Writing new parallel applications is unusual for domain scientists and thus has created a complex dependency on specialized scientific programmers, who are scarce [51].However, the capacity to quickly envision and prototype applications by domain experts-later to be fully implemented into by scientific programmers-is critical for acquiring maturity and proficiency in the grand challenges that Exascale attempts to solve, and moreover for allowing scientists to develop their own ideas quickly and productively [52].The latter, in addition, would increase service utilization time in HPC systems, a critical measure of efficiency.Considering the existing code base in scientific computing, software infrastructures that allow a clean transition from legacy systems to new Exascale platforms while preserving flexible prototyping towards the future are central [53].
This group attempts to tackle the latter challenge through the development of novel software prototyping infrastructure with an emphasis at a higher degree of abstraction.The group deems the ability to rapidly construct software artifacts that can be trusted in terms of the methods from their design up, easily discarded when wrong and extended when right at low human and computational cost to be one aspect of scientific software sustainability.In essence, a motto for scientific software prototyping is fail hard, fail fast.

Participants.
• Santiago Núñez-Corrales <nunezco2@illinois.edu>• Chris Sweet <chris.sweet@nd.edu>• Steven Brandt <sbrandt@cct.lsu.edu>• Dominic Orchard <d.a.orchard@kent.ac.uk> 8.3.2.Working group objective.The group's objective is development of a prototype tool that allows domain scientists to generate simulation prototypes through a simple, yet expressive declarative strongly-typed programming language close to equational expressions.The result of that specification is both an executable artifact as well as a specification for scientific programmers to later flesh out completely and adapt to particular infrastructures.This tool will start in the domain of computational physics, and (probably) extend to other domains, and the code will be open source.
The underlying hypothesis of this project is that a strongly typed, declarative problemspecification language with adequate constructs, to be implemented through stencils and algorithmic skeletons tied to numerical methods, captures a subset of useful and interesting problems in physics and other domains.8.3.3.Gap or challenge.The gap addressed by the project is described by following four statements: (1) Domain specific scientists need to strike a balance between being productive in their science areas and having some coding skills.(2) Being a research software engineer is a career in its own right which has limited depth in the science basis, especially while serving large communities.(3) Collaboration in science is made (mostly informally) through natural language and mathematics, but both present lots of opportunity for potential ambiguity towards software development.(4) An unambiguous (formal) language that sets a neutral ground is missing, capable of lifting many of the complexities of constructing useful (and correct) software artifacts that may later evolve into more elaborate codes.The latter enumeration of elements is sustained in the fact that scientific software packages are digital representations of partial knowledge architectures behind research processes.

8.3.4.
Relevant people and resources.The project is mostly self-contained and only requires development time from participants, as well as a Git repository (active).As for skills required, the following distribution between project participants was identified: Steven: parsing expression grammars, prototype stencil Santiago: functional programming, scientific programming Dominic: semantics, types, compilers, languages Chris: numerical methods In terms of practice and experience, a sample scientific community is required in order to work around the capabilities of the tool.In addition, use cases are required (e.g., micromagnetics [54]).8.3.5.Plans.The group's general plan is to develop a functional (meta-)prototype before WSSSPE5, and if useful, then develop a larger software infrastructure tied to particular cyberinfrastructure resources.

SMART steps.
(1) Define or choose the description language depending on the potential user base.
(2) Develop the language interpreter/compiler for the equational description language.
(3) Define a set of fundamental software patterns to translate to algorithmic skeletons.(4) Implement a translation mechanism from the language to the patterns.( 5 Other resources to draw on are the software metadata vocabularies of Schema.org 18and DOAP 19 , and the FORCE11 Software Citation Principles [31].Relevant documents on research software, including the Guidelines for persistently identifying software using DataCite [56] are available on the UK's Research Data Network's Research Software web page20 .8.4.5.Plans.The working group plans to engage with those already working on CodeMeta, to examine the existing CodeMeta crosswalk table to see what improvements and additions might be made, and to determine how to engage the community and provide ongoing social engagement and structure.Further, the group wants to assist in the implementation of the specifications and, by providing an outsider's view, contribute suggestions for more understandable project documentation.Finally, the WG seeks a better way or ways to present the crosswalk table to make it more easily understood and consumable by research software communities.8.4.6.SMART steps.With a comparatively large working group, the use of the established CodeMeta Github repository's issue tracking, and the quick responsiveness of CodeMeta lead Matt Jones to a barrage of questions, comments, and logged issues, several of the group's SMART steps were completed at WSSSPE; these are identified below in italics.
• Write to the managers of CodeMeta project to start dialog with the two groups  21 and include email replies in response to WG questions from one of the CodeMeta project leads.
Because of the work done at WSSSPE4, the CodeMeta project README file 22 was greatly expanded to include a description of the project that is geared to those with little or no prior knowledge of the project, a list of contributors, information on how one can get involved, a brief project history and who is managing the project, and links to additional information.Though a Google group mailing list has been established for the working group, the easiest way to engage with the CodeMeta project is through its Github repository23 .8.5.White paper on developing sustainable software.Many diverse aspects and dimensions of developing sustainable software can be investigated, such as economic, technical, environmental, and social.This group aims to write white papers that will focus on scientific environments and their implications, targeted at developers and project managers of scientific software.Given the complexity of this field, it is important to select a subset of sustainability aspects for the white papers.The idea is to create a series of papers instead of trying to tackle all important topics in one paper.For the first white paper, the group aims to set the stage with successful use cases and an analysis of why they have been successful.Further topics will include community-related practices, government and management, funding, metrics, tools, and usability.The group will collect feedback from the WSSSPE community on this first white paper and extend it to a journal paper.

Gap or challenge.
There is more and more academic research being done on topics related to software sustainability, including work on software engineering practices and management of open source projects.However, academic research in general is often siloed for many reasons, and work on topics relevant to software sustainability is no exception.There are many academic studies and projects which may be relevant for practitioners working in this area, and many projects and initiatives by practitioners that may be relevant for social scientists.However, there is a gap between these two research and practice.Furthermore, the group also recognize that academic social science and research software engineering are not monolithic, and there is a need to connect people who care about research-driven best practices inside of these two domains with each other as well.undergraduates.This working group was motivated by the perceived prevalence of so-called "hidden code" in scientific communities: code written by individual researchers in an unsustainable way that is never shared with the larger community.Participants recalled their own experience working with colleagues who write hidden code or inheriting unsustainable hidden code from collaborators.
The question is, then, how to prevent researchers from writing hidden code?The participants hypothesized that the best strategy is to catch researchers while they are still in training and teach them software best-practices.Therefore, this working group was formed with the goal of developing courses on software best practices aimed at undergraduate students studying domain science.The program might be similar to a Software Carpentry or Data Carpentry workshop but with a focus for domain scientists.8.7.1.Participants.
• Jonah Miller <jmiller@perimeterinstitute.ca>• Aleksandra Nenadic <a.nenadic@manchester.ac.uk> • Raniere Silva <raniere.silva@software.ac.uk> • Francisco Queiroz <chico@puc-rio.br>• Hans Fangohr <fangohr@soton.ac.uk> • Prabhjyot Sing <prabhjyot10@gmail.com>Aleksandra Nenadic and Raniere Silva are involved in Software and Data Carpentry.8.7.2.Working group objective.Implementation of a course aimed at each domain science is a long term goal.However, a short-term, achievable goal, is to develop a curriculum for a course aimed at a single domain science.Since the participants in the working group have expertise in physics, this is a natural target.8.7.3.Gap or challenge.The challenge is that people are writing unsustainable scientific software without ever learning that there is a better way.Here we focus on training in good software design and engineering in this context.8.7.4.Relevant people and resources.The development of a successful curriculum relies on the expertise of software engineers to describe the best practices, domain experts to describe model problems and work-flow, and instructors to formulate the pedagogy.Ideally these people will be brought together for a short workshop or hackathon with the goal of drafting the curriculum.Later, organizational partners will be required to actually implement the program.8.7.5.Plans.The working group will seek funding to organize a hackathon to develop the curriculum and seek the required expertise to invite.The hackathon will be implemented and a draft of the curriculum will be written within the next year.8.7.6.SMART steps.The working group will accomplish or has accomplished the following smart steps: (1) Create channels of communication including a mailing list, a slack channel, and a git repository (done) (2) Review literature on best and good-enough practices for scientific computing as well as literature on pedagogy for scientific and research computing (by February 1, 2017) (3) Organize a telecon to organize a workshop to write the curriculum and decide how to bring in outside experts (by February 1, 2017) (4) Write a draft of the curriculum (by April 1, 2017) (5) Invite the broader community to contribute to a more complete curriculum (by April 30, 2017) 8.7.7.More information & joining instructions.Anyone interested in contributing should get in touch via one of the following methods: • Join our Google Group/mailing list [57].
8.8.Meaningful metrics for sustainable software.Meaningful Metrics for Sustainable Scientific Software aims to increase the visibility of the quality of scientific software, facilitate the reusability of scientific software, and promote the best software practices by standardizing metrics via interviews with scientific software developers.This working group believes improving the current software metrics system will increase software sustainability.Currently, there are inefficiencies regarding software duplication, sustainability, and selection, as well as others, within the scientific software community.In order to address these inefficiencies, Meaningful Metrics for Sustainable Scientific Software aims to create a goal-oriented method to collecting productive metrics by focusing on the developer side of software. 8.8.1.Participants.
• Emily Chen <echen35@illinois.edu>• Patricia Lago <pdotlago@gmail.com>• Udit Nangia <unangi2@illinois.edu>• Tengyu Ma <tengyuma10717@gmail.com> • Aseel Aldabjan <a.dabjan@hotmail.com>8.8.2.Working group objective.In the context of the larger objective, Meaningful Metrics for Sustainable Scientific Software hopes to find an efficient solution for streamlining the process of collecting and utilizing metrics to benefit software sustainability, as well as minimize the current inefficiencies within the scientific software community.Finding meaningful metrics will improve software evaluation and comparison, thus reducing the effort spent on seeking scientific software.
8.8.3.Gap or challenge.The gap being addressed is the lack of a standard for collecting and presenting metrics.This gap delays workflow and creates a multitude of tedious tasks, including searching for the best-fit software, unknowingly duplicating software, and other related busy-work.The need for metric standardization stems from the abundance of scientific, both "dark" and open source, software and the difficulties of ensuring the software is sustainable.
8.8.4.Relevant people and resources.Relevant people and resources include scientific software developers, researchers that use scientific software, scientific software funding institutions.
8.8.5.Plans.This working group plans on interviewing scientific software developers to form metrics from the goals they have for their software.
8.8.6.SMART steps.The SMART steps Meaningful Metrics for Scientific Software proposed are: (1) Identify the population of scientific software developers who are willing to be interviewed (2) Define the interview questions and organize them into categories (3) Interview the participants and map the survey results to goals (4) Convert the goals to feasible, meaningful metrics (5) Analyze the collected metrics 8.8.7.More information & joining instructions.For more information, please contact Emily Chen at <echen35@illinois.edu>.
8.9.Coordinating access to continuous integration (CI) for research software.Each developer of software with uncommon needs (hardware, software, libraries, data sets), non-public code, or tests that exceed time limits for free plans must acquire, setup, and maintain their own continuous integration systems because their needs make them ineligible for popular free services such as Travis CI.For example, software groups that develop BIOUNO, CI4SI, or GROMACS have done this.
Note that Debian provides a testing infrastructure (https://ci.debian.net/doc/) that is mature and supports hardware and other requirement specifications.8.9.1.Participants.
• Xinlian Liu <liu@hood.edu>• Mark Abraham <mjab@kth.se>• Sameer Shende <sameer@cs.uoregon.edu>• James Hetherington <j.hetherington@ucl.ac.uk> • Dominik Kempf <dominic.kempf@iwr.uni-heidelberg.de>• Michael R. Crusoe <michael.crusoe@gmail.com>• Radovan Bast <radovan.bast@uit.no>8.9.2.Working group objective.This group is interested in reducing the burden of different projects having to build and maintain their own continuous integration systems (when publicly available CI are not a fit), by coordinating and sharing this burden across multiple projects.8.9.3.Gap or challenge.The scope of the group's interest is any type of testing, though interactive access for troubleshooting would be of particular interest beyond just automated testing.
However, there are a lot of open issues: • Some similar work has already been done.How can this group apply and/or learn from that work, rather than reinventing the wheel?• How to ensure this will work across disciplines?• Meaningful CI for large projects may need hundreds of CPU hours per day 8.9.4.Relevant people and resources.This would need to be picked up by a set of projects, potentially both small and large.
Possibly working with the RSE community would be useful.
8.9.5.Plans.Some possible goals include: • Acquire additional hardware such as GPUs, Xeon PHI, FPGAs and add/share them to e.g.Debian's testing infrastructure or to a shared Jenkins-based infrastructure • Extending Debian's scope to include published but not mature software • Implementing the same interface but with specialized hardware or available to non-public codes.
(1) Learn how to donate machines to Debian (2) Find funding -NeIC (https://neic.nordforsk.org)through the CodeRefinery project (http: //coderefinery.org) is building a CI infrastructure (in progress) that will be provided to selected projects (through an open call) (3) Build community 8.9.7.More information & joining instructions.For more information, see https://groups.google.com/forum/#!forum/continuous-integration-for-research-software8.10.Software engineering processes tailored for research software.This working group is concerned with identifying processes that are not adequately covered by general software engineering.Verification and testing is the first subtopic that it addresses.Computational science and engineering applications have many moving parts that need to interoperate with one another.The accuracy and reliability of results produced by scientific software depends not only on the individual components behaving correctly, but also on the validity of their interactions.As scientific understanding grows, the corresponding computational software models are refined, leading to more complex codes.Increasing complexity makes them more prone to defects, not only in individual code units, but also in interaction among units.Therefore, a strong verification process combined with a rigorous testing regime plays a critical role in the prevention of generating incorrect scientific results.However, most science teams struggle to find a good solution for themselves.Causes range from lack of exposure to the practices, to distrust of adopting practices because they do not meet the needs of the teams developing such software.The current focus of our effort is on testing because it is the first step towards building a software processes that can lead to provenance and reproducibility, the hallmarks of quality science.8.10.1.Participants.
• Mark Abraham <mjab@kth.se>• Anshu Dubey <adubey@anl.gov>• Hans Fangohr <fangohr@soton.ac.uk> • Dominic Kempf <dominic.kempf@iwr.uni-heidelberg.de>• Eric Seidel <eseidel@cs.ucsd.edu>8.10.2.Working group objective.Scientific computing software lags behind commercial software in adoption of software engineering practices.This gap is particularly acute in the area of software testing, verification, and validation, where the standard practices are simultaneously inadequate and overly onerous.This group aims to close the gap.8.10.3.Gap or challenge.Computational science code developers often lack of exposure to regular testing and its benefits.Good developers will test their code to verify that it operates as expected; however, they may not appreciate that without regular testing, defects can be introduced inadvertently.An even bigger challenge is that those who understand the importance of regular testing do not often find much help from software engineering literature.There is a significant gap between the testing gospel and its applicability to computational science.This gap leads to frustration and abandonment of the good with the bad.Some relevant literature exists, in particular experiences from practitioners in computational science who developed their own solutions.However, this literature is scattered among many different forums, and can be challenging to find.This working group aims to address this gap by curating the existing content and contributing content where none exists.8.10.4.Relevant people and resources.The working group will benefit from wide participation by developers of large computational science codes.The reason is that the management of such codes becomes intractable without adopting some software process and a testing regime.The experiences and customizations vary, and the community will benefit from hearing about as many as possible.The seed resources required are fairly minimal.The group has started a git repository for collecting the existing references.That, and a few volunteers reading through the references is all the group needs in the beginning.As it gathers more knowledge and pinpoint gaps, it may need more resources to reach out to a wider group of developers to collect more information.8.10.5.Plans.This working group will (1) conduct a literature survey to gauge the extent of awareness of the issue in general, (2) generate content useful for the community where needed, and (3) curate the collected and added content for the use of the community.If we identify a leader (PI), a plan could be: (1) Obtain funding for a set of initial discussions (or find a group who feel that this is important enough that they will take the time to do it anyhow) (2) Talk with potential partners (publishers, orgs, funders) (could be combined with meeting below) (3) Talk with the Google Scholar developer for better understanding of what they did and are planning in the future (4) Obtain funding for a meeting (perhaps from NSF, perhaps a Dagstuhl meeting) (5) Hold a community meeting to define a plan (which should be large enough to include representative of all stakeholder groups, and small enough that we still have a good open discussion) (6) Obtain funding to implement the plan, including contributions and commitments from the stakeholders Alternatively or additionally, the group could build a mailing list for us and discuss further, depending on how receptive others are to this idea.At WSSSPE4, there seemed to be enough interest to do this, so the group set up a Slack channel within the WSSSPE team, called #wg-open-research-idx.(See §9 for instructions on how to join the Slack WSSSPE team.) We also discussed some possible more detailed plans: (1) Look at curriculum lattes (Brazil)  • Some of these will have costs, though some may be free • Need to determine costs • Daniel S. Katz <d.katz@ieee.org>• Frank Löffler <knarf@cct.lsu.edu>• Kyle Niemeyer <Kyle.Niemeyer@oregonstate.edu>• Janos Sallai <janos.sallai@vanderbilt.edu>8.12.2.Working group objective.Scientists working on scientific software are often located in disciplinary departments, depending on whether their software originates from the mathematical, physical, chemical, or other disciplines.As a consequence, they are frequently outside the core areas of their science, and their contributions are typically to both the research activity their software enables, as well as on algorithm and implementation aspects.This presents issues when letters of evaluation for hiring, tenure, and promotion do not specifically cover how this is relevant to the discipline. 24 This group believes that the authors of scientific software provide important services to departments that are no less than strictly disciplinary research.Consequently, leveling the playing field with more disciplinary candidates for hires, tenure, and promotion requires that letter writers be aware of how their letters will be read by committees.It also requires that committees be aware that such letters often look different and may provide a different perspective of how a candidate's achievements should be assessed.For example, in mathematics a typical candidate would be evaluated on the difficulty and depth of the statements she may have proven in their papers.On the other hand, an author of mathematical software would likely be evaluated based on the impact of her software, or the number of citations of the publications that describe it.She may also be evaluated by how widely the software is used outside mathematics, a criterion that is rarely used for more disciplinary mathematicians.8.12.3.Gap or challenge.Addressing this problem likely requires that letter writers pay particular attention to who exactly the audience of such letters is, and tailor the message by specifically highlighting how the work of a candidate benefits the department and discipline that is considering a candidate.On the other hand, committee members also need to pay attention to the fact that there are areas that are important to the mission of their department and professional community in which different standards for evaluation hold.8.12.4.Relevant people and resources.Affecting letter writers and readers essentially requires raising awareness of the issue.Various organizations have attempted to do so through letters to the community, adjusting definitions of what counts in science, etc.Specifically, the group is aware of the following resources: • The Computing Research Association has produced a "Best practices memo" on the issue [60].Among other points, the memo also clarified that conference proceedings are the venue in Computer Science, and this has been used to convince deans and provosts of the value of these proceedings publications.The article may serve as a template for computational science.• The National Science Foundation has produced a number of "Dear Colleague" letter that consider the issue of software and, taken together, define a "product" for the purposes of 2-page biosketches in a way that does not include only publications, but also software. 25This definition is ultimately codified in the National Science Foundation's "Proposal and Award Policies and Procedures Guide" (often abbreviated as GPG) which in 2013 renamed the "Publications" section of 2-page biosketches to "Products" (see Chapter II.C.2.f(i)(c)) and now explicitly allows software to be referenced in this section.• There is also a 1995 report by the National Academies about Computer Science that addresses some of these issues, specifically on how writing software should be considered in comparison to more theoretical research [62]. 24The extensive use of such letters, and the problems that are associated with it, may be an issue specific to the United States. 25The most pertinent such letter is NSF 14-059, "Dear Colleague Letter -Supporting Scientific Discovery through Norms and Practices for Software and Data Citation and Attribution" [61].
8.12.5.Plans.There are essentially two important strategies that build on each other: (i) raising awareness of the problem beyond just those who are affected by it (namely, computational scientists working on scientific software), and (ii) providing letter writers, letter readers, and evaluating committees with guidance on what criteria are relevant in assessing scientific software authors.Concrete guidance-for example in the form of "Dear Colleague" letters as mentioned in the previous section-is most valuable if it comes from respected bodies such as professional societies or established and respected organizations.Getting these to act will only be possible if the problem is widely acknowledged, and so the first of the goals above should be the current focus.
The group will attempt to address it by organizing sessions at conferences that address the career problem, as well as writing editorials that can be published in the magazines of professional societies.These editorials ought to outline best practices for letter writers that specifically (i) make clear the contribution of a candidate to their interdisciplinary area, (ii) the relevance to their home discipline, and (iii) why writing software is good for the discipline itself.8.12.6.SMART steps.The group has identified a few next steps, even if they may not satisfy the exact criteria of the SMART system.Specifically: • Find and review the documents listed above, send out summary (Wolfgang Bangerth) • Gather feedback from the other members of the group • Write editorial for SIAM (Wolfgang) • Write two emails to each other group members listed above about also writing editorials in their communities.In the longer term, the group hopes to gather others in the home disciplines of its members to let professional organizations weigh in.8.12.7.More information & joining instructions.The section on resources above lists a number of letters and reports that are worth reading.Ultimately, building a community large enough to affect change is important; contact Wolfgang Bangerth at <bangerth@colostate.edu> if you are interested in helping.

S l ac k t e a m a n d c h a n n e l s
As part of the discussion at WSSSPE4, we created a Slack team for WSSSPE [58].To join it, go to https://wssspe.signup.team/.A number of the working groups also created channels within the team; see the subsections in §8 for the names of these channels.

At t e n d e e s u rv e y
All participants present on the last day of the meeting, 14 September 2016, were asked to complete the online survey administered through Qualtrics26 .44 responses were recorded, with one respondent indicating that they took the survey twice.The survey URL was also later distributed to attendees by email and advertised on twitter for attendees who were not there on the last day to have a chance to respond, and one additional response was recorded after the meeting.Hence, the total number of unique individuals responding to the survey is likely 44.The survey contained nine multiple choice and open-text answer questions.Text responses have been alphabetize to preserve anonymity.The survey had a 100% completion rate.Appendix E contains the detailed survey results.
In general, the respondents were highly satisfied with the meeting and interested in continuing and participating in WSSSPE activities.The mix of topics and types of interactions-talks, panels and discussion-was well balanced with several indicating a desire for time for questions and answers following talks as well for discussions in general.Many respondents indicated that they will remain engaged in WSSSPE working group initiatives.However, demand for a professional organization encompassing WSSSPE interests was weaker with over half interested in joining and a large percentage willing to consider the idea.Respondents were grateful for the opportunity to network, explore new collaborations and for travel support to the conference.
The survey results indicate that future WSSSPE conferences should consider the balance of the attendees.This year's meeting had many first time and early career participants who would have benefited from a review of basic concepts and terminology including what it means for software to be sustainable and the roles of research software engineers.Additional emphasis and topics to explore would be the inclusion of more case studies, focus on the decision making process in developing and using software, deeper dives into selected topics, tutorials, software training within and outside of STEM, lowering the barriers to implementing best practices in software development, and progress towards executing WSSSPE's vision.

C o n c l u s i o n s
In WSSSPE4, we heard about a number of interesting projects and ideas, and used those ideas to create and form working groups, intended to start at the workshop and then continue afterwards, to address challenges that arose from the workshop ideas.This workshop has reinforced the lesson from WSSSPE3 that it is relatively easy to get motivated people to attend a meeting and productively spend their time there both doing work and planning more work, but it is very hard to get that additional work after the meeting to take place.The main problem seems to be one of time.Once the attendees have agree to spend their time at the workshop, they put their energy into doing so productively, but they have not really committed themselves to anything more than this, so their energy and effort trails off, as all of their other commitments (particularly, those they are funded to do as part of their jobs) come back to the fore.Without a process to resolve this concern, the utility of further multi-day WSSSPE workshops is unclear.

Ac k n ow l e d g m e n t s
This material is based upon work supported by the National Science Foundation (ACI-1648293, ACI-1547611), by the Gordon and Betty Moore Foundation (GBMF#5620), and by the Alfred P. Sloan Foundation (G-2016-7214).
A p p e n d i x E. D e ta i l e d s u rv e y r e s u lt s Note: typos and spelling errors have been corrected, and spelling has been changed from UK standard to US standard in some cases, but no other changes have been made to the attendee responses.(3) What additional topics would you like to see discussed at the next meeting?
• Concrete reports about best practices • considerations of software licensing, build systems, delivery (source vs binary vs whatever) -but of course these are "later" issues... first one needs the software to not be hidden, to have version control, have a README, etc. • Container sustainability practices • education, training interns • Explicit follow-up at WSSSPE5 on all the activities that were created at WSSPE4.These should be at least lightning talks, perhaps with their own session.• financial models for sustaining software • for building software • have a short beginning presentation for each working group and a short ending presentation • How software development training can be adapted to different disciplines beyond STEM • How to low the barrier for non-tech heavy researchers to follow best software practices.
• I would like to have more discussions about research, if people can share their research or some of the interesting projects that they're working on, this will help in knowing the problems in sustaining software.• I would like to learn more about sustainability issues in industry.
• I would like to see more grounded advice and less blue-sky thinking.A lot of stuff in the talks was 'this would be nice' but there were not enough talks that were directly applicable to day-to-day programming.• Like to see more examples from the real world.
• Mechanisms of improving software reusability • More concrete projects, concrete possibilities and showcases for collaboration with results • Practical ways of achieving some of the goals rather than the discussion of more attempts on the writing of white papers • Reproducibility through containerization (I guess there is another list of best practices needed there) • RSE award, organizational questions regarding setting up RSE teams within universities • Software testing • Some theory on what makes for sustainable software, which is not the same as best practice for research software engineering.If this had been talked about before then consider that 80% of us were new.• Specific scientific software sustainability factors • Testing methods that work.Ways of promoting organizational change.
• The progress of our current project.
• The use of scientific software at different stages of research / work • We discussed scientific software.Should we also discuss software FOR scientists (like Dan's google scholar replacement)?(4) What could be improved?
• A bit less fly by night scheduling though things worked out just fine.
• A lot of the speakers just seemed to be there either to plug their own projects (e.g.Hydroshare, BioDynaMo) or spent the majority of their talk waffling (e.g.Manifesto for Personal Responsibility in the Engineering of Academic Software).Also, the icebreakers were not very useful.• A single room for the duration of the meeting.(Particular to this WSSSPE4, the "collaboratory" room with the monitors, was actually perfect for the full meeting.)• Allotting more time for questions & discussions in between talks • close to perfect workshop -if I would change one thing that give more time for discussions • consistency of participants • felt locked in to the breakout group, not sure if that was intended or not • Finding rooms a bit confusing at the meeting outset • Having more time for the demos.
• Having more time to discuss some of the talks • As a student, coming to WSSSPE4 has been very rewarding experience.Sustainable software is not my particularly sub-field of interest, but it was very interesting to me to apply the skills I had learned in my research in a different field.It definitely taught me a different perspective and expanded my original perception of where my skills can be applied.• As a student, this has helped me explore the different computational research careers and has made me more aware of scientific software best practices.I found the plenary talks particularly valuable for learning about different areas of study, and it was great how willing all of the WSSSPE participants were to share more and help each other.I also found it valuable that there were other undergraduates at this WSSSPE, and it created another small community and encouraged me to participate more.• buddy me-sheet and workshop planning and moderation was excellent • Don't start at 8.45am if you're asking people to socialize and connect in the evenings!Work on getting a better gender, geographical and discipline balance.The use of travel bursaries has helped (particularly in getting early career participants).• Great organization, enjoyed being here very much, it was surely a success!• Great to meet colleagues from the UK and Germany at this event and enjoyed this format for working, engaging, and producing with the community while at the workshop that was facilitated by KnowInnovation.• I enjoyed the brainstorming session on Tuesday morning -well organized • I had hoped to have some discussion of building the sustainable communities beyond the software writing stage.The HPC focus, though expected slightly, was too heavy and I think ignores some of the other long term issues, such as preservation and addressing first principles for new technologies, or smaller, individual projects that will also support science.Perhaps it would be good to have a student session as well?(But I do like the integration).• I think the mixing of students, staff, faculty etc in the working groups is very effective and will contribute to outcomes outside of the working group.It was great to see that we have undergraduates leading working groups here.There should be more attention on diversity.• I wish there were a project on training interns to work at academic environment as programmers • I would like to thank the committee for selecting me.As an undergraduate student this was a great experience I got to meet professors from the universities which I wanted to join.Now I exactly know how to prepare my application.I learned some really good technical skills and will definitely apply it to my research.So happy to be a part of this wonderful group.I will design a logo for WSSSPE next year.Hope I can attend this event next year also.• Is there something better than sharpies (few of running around the room with a permanent marker)?Travel Awardee: Discussions here will and have benefitted my organization and community.• It was great to connect with people doing research software engineering in various positions, contexts, and universities.I felt like I learned a lot.Moving between the two rooms was a bit confusing.• Maybe involve more researchers from low-income countries if funds are available.
• no.I think the moderator and the format are both great.
• Nothing comes to mind • Receiving a travel award was crucial for me to come to WSSSPE4 this time, especially because WSSSPE4 was held outside the US.On the other hand, I see the great benefit this had on the geographic diversity of the participants, so I do think that meetings not only in the US are of great value.And again, I would like to emphasize, that travel awards are crucial for this type of meeting, as most participants are not directly supported for this type of activity -one reason to have a WSSSP5.(on another note: have a multi-line text field for comments next time) • The issue of software sustainability is on par with the challenges of building scientific instruments.Part of what I am for my doctoral dissertation is finding better abstractions, which leads to sustainability through better mathematics.This workshop raised my awareness in several other aspects, which will be considered for this and all future projects I may be related to.• The travel award made it possible for me to come to WSSSPE-4 and broaden my horizons.
Many issues pertaining research software and sustainable software were uncovered, and WSSSPE-4 brought to light the importance of the issues.I'd encourage other travel awards to people, like me, who do not belong to the community.• There aren't lots of researchers knowing such kind of concept, sustainable software.We need to let more RSE know it.• There is a lot of discussion, but it's hard to see what actually will come of it.The discussion of the 9 working groups from last year at the beginning of the workshop clearly showed this: IIRC, not one had produced anything, all seemed to be stalled somewhere.We ought to find ways to actually make things *happen* after we talk about it (and admittedly have many good ideas).• This has been my first WSSSPE, and as someone coming from the humanities with an interest in RSE it has been an enormously rewarding experience to be able to interact with and learn from individuals and initiatives that have been part of the greater discussion around WSSSPE topics more in both quantity and quality.For me personally, it made me aware that I really need to refactor a lot of my basic assumptions for a PhD thesis, but that's a great thing because it means that I was able to draw on the exchanges and experiences from WSSSPE already.All in all, the workshop has made me more determined to become more involved with this community.And last but not least i just wanna say THANK YOU for the generous travel support I've received, which meant that my PI and dept.didn't have to have any doubts whether I'd drain their resources which are meant to be used towards achieving a completely different research goal.• This was an excellent mix of people, timed/structured activities, and untimed events.
• This was an excellent mix of people, timed/structured activities, and untimed events.
I had the opportunity to talk to numerous people about projects, issues, and ways to improve science/scientific software, and leave with many new ideas and possible collaborations/cooperative efforts with complementary projects.The working groups activities were excellent, a good mix of projects, and I saw a lot of progress on several I'm very interested in.(Please group this survey with my earlier one; I hit "end" too soon on the other!Thanks!) • We have activity on slack, github, and twitter.And working groups have their own stuff.
Can we index these in a single place?• What percentage of the working groups actually end up producing anything?It seems to me as though most come up with an idea and then find out they don't have the resources to implement it -nice as the ideas are.Also, I assume someone is going to attempt to reinvent metrics or best practices every single year, I guess that's unavoidable.• WSSPE seems to be evolving nicely from year to year.I hope that in WSSSPE5 there are lots of results to report from the outcome of WSSSPE4 working groups.• You could try unconferences instead of tracks.

) 8 . 1 .
Verifying best practices & metrics for sustainable research software.Many open source projects for research software document their best practices that contributed to the sustainability of the software.Many open source projects also document software metrics they use to define their research software project's success.This group aims to aggregate several sources of these best practices for sustainability and metrics into a consolidated list.The team then plans to create a workflow to evaluate how open source projects stand up to these lists.8.1.1.Participants.

( 1 )
Please indicate how strongly you agree or disagree with the following statements Ta b l e 5 .General questions about the conference program Question strongly somewhat agree somewhat strongly Total disagree disagree agree agree The plenary sessions had a good mix of topics and speakers.The plenary sessions had a good mix of topics and speakers.( b ) The balance was right between talks, panels, and lightning talks.(c ) There was sufficient time for discussion.

F i g u r e 1 . 2 )F i g u r e 2 .
Responses to general questions about the conference program (What part of the meeting was the most valuable?select all that apply Responses to "What part of the meeting was the most valuable?" Sustainable software has the capacity to endure such that it will continue to be available in the future, on new platforms, meeting new needs.The research software lifecycle includes: • Principles and Best Practices.Promoting best practices in sustainable software • Careers.Developing and supporting career paths in research software development and engineering • Learning.Engaging in activities to promote peer learning and interaction • Credit.Ensuring recognition of research software as an intellectual contribution equal to other research products Definitions:
[48]ware engineering for science/academia are sufficient and valid, and that metrics for software engineering for science/academia are relevant and useful.The group has a specific objective of getting a good cross-sampling of disparate software including community model codes, community cyberinfrastructure, community analytic tools, etc. 8.1.3.Gap or challenge.The gap addressed by the project is described by the following:• Are the suggested software engineering for science/academia best practices verified when evaluated against open source research software projects that currently use modern software engineering?•Are the suggested metrics for science/academia verified to be relevant when evaluated against open source research software projects that currently use modern software engineering?•Doresearchgroupshave role models to follow for successful best practices in research software?8.1.4.Relevant people and resources.The people spearheading this effort at the time of the writing of this workshop report are the working group participants listed previously along with Peter Elmer <peter.elmer@cern.ch>andHansFangohr<fangohr@soton.ac.uk>.Best Practices for Scientific Computing,Wilson et al. 2014 [47]-8 high level categories • Butterfly: a paradigm towards stable bio and neuro informatics tools development[48]InterMine: Best Practices for Open Source Software (http://www.rse.ac.uk/conf2016_ talks#T1.1)• ANNIS corpus linguistic analysis tool humanities code (http://corpus-tools.org/annis/) 8.1.5.
-Example sources: Astronomy source code library; high citation software papers; CIG software list; personal domain knowledge.• 8.2.1.Participants.•Neil Chue Hong <N.ChueHong@software.ac.uk> • Jean Salac <jeansalac@virginia.edu> • Radovan Bast <radovan.bast@uit.no>• Lorraine Hwang <ljhwang@ucdavis.edu>• Karthik Ram <karthik.ram@berkeley.edu>• Peter Elmer <Peter.Elmer@cern.ch>8.2.2.Working group objective.The overall objective of the Software Sustainability Alliance working group is to develop the steps necessary for establishing an alliance between organizations willing to engage in mutually reinforcing activities to advance the sustainability of software used in research.These organizations are funded groups or teams that aim to advance research software sustainability beyond their local university or community.More specifically, this working group aims to define the scope of this alliance and provide clear distinction with WSSSPE.The group also seeks to understand the incentives for members of this alliance and identify its key activities.8.2.3.Gap or challenge.Currently, point-to-point collaboration exists between organizations, but this inadvertently results in competition or redundancy within the sustainable software community.
8.2.4.Relevant people and resources.The working group is looking for ideas for which organizations should be consulted or invited to join the Software Sustainability Alliance.It is also looking for feedback on the aims and scope for joining the Software Sustainability Alliance.8.2.5.Plans.Throughout the scoping period, the working group will continue to refine the Software Sustainability Alliance as it collects feedback on potential members, aims and scopes.It will also update its website to reflect these changes.Next steps and further tasks will be determined based on the feedback from the scoping period.8.2.6.SMART steps.The first SMART steps of the working group are outlined below: • Create a draft text to explain context and incentives for joining the Software Sustainability Alliance • Come up with a list of potential invitees • Update www.softwaresustainability.org to provide information • Draft a letter to go out to potential alliance members • Create a timeline and framework for initial consultation 8.2.7.More information & joining instructions.If interested, visit http://softwaresustainability.org/ or email Neil Chue Hong (<N.ChueHong@software.ac.uk>) and Jean Salac (<jeansalac@virginia.edu>) ) Test the system with three known solvable problems and evaluate with a set of functional and This group, which included people who had previously been working on the CodeMeta project17, formed to address the proposed project, "Define a standard for metadata to be used in current repos including authors, dependencies, . . ., repo URL." 8.4.1.Participants.CodeMeta wants to incentivize software developers to release their software, encourage the development of metadata for it, enable credit assignment and citation of research software, increase its discoverability, more easily track dependencies, and enable reuse of software metadata, all goals that WSSSPE attendees have great interest in supporting.

•
Post questions generated by our discussion as issues on the CodeMeta repository • Look at the crosswalk table and Schema.orgdata elements for common elements • Identify text in the project documentation that is unclear and suggest changes • Define a list of services to be added to the crosswalk table and match terms • Identify two roles and information on CodeMeta that would be useful to these people for engaging with the project • Determine a better way or ways to present the crosswalk table • Create a mailing list for WSSSPE CodeMeta participants 8.4.7.More information & joining instructions.Notes for the working group were taken in real time in a Google document The working group aims to start a series of papers and supporting developers and project managers of scientific software.While there are already a few papers available on best practices and sustainability of scientific software in general, the group's goal is to create a series of papers that lead to consensus in the community, tackle many diverse aspects, and stay up-to-date with new trends.This goal is quite ambitious, and the group hopes to attract more authors over time who would like to contribute to specific aspects and take the lead on them.8.5.3.Gap or challenge.The challenge for the group is to accomplish a first version of the white paper to kick off the series.The complexity of the topic has led to a first attempt in 2016 that may have been a bit too ambitious, by trying to cover a wide range of topics.Addressing different topics over time seems to be a better strategy to finalize a paper.The series would fill also a gap on best practices to which the community can contribute.Plans.The resulting draft white paper will be distributed via the WSSSPE email list.After collecting feedback on it, the group's plans are to attract a wider community to contribute to an extended journal paper version and to investigate whether more people would like to be involved in the white paper series.More information & joining instructions.The GitHub repository for the white paper can be found at https://github.com/WSSSPE/WG-Best-Practices.For more information and requests, join the #wg-best-practices channel at WSSSPE's Slack team.

Software best practices for
Plans.The working group plans to briefly survey existing literature to get a better sense of the academic research landscape, facilitate some initial dialog between academic researchers and practitioners at WSSSPE4, then identify needed actions that would be mutually beneficial to researchers and practitioners.This plan resulted in the creation of the following SMART steps listed in the next section.
8.6.4.Relevant people and resources.People: a network of invested social scientists and research software engineers who care about research on software engineering practices Resources: a mailing list and a central repository for collecting literature and documenting lessons learned 8.6.5.

Open research index.
8.10.6.SMART steps.Our first few SMART steps are :• Create a channel in wssspe.slack.com-done•Create github repository http://github.com/wssspe/testing-in-science-done•Find and gather existing publications in repository -ongoing • Review and summarize material.Decide whether we consider this sufficient.If yes, then the group will put a brief report together and conclude the working group.•If the group does not consider the material adequate, it will start research and gather methodologies for testing in science • Write a document accessible to computational science and software engineering community, and publish document at a citable forum.The aim of this group is to investigate the building of an index of research products in an open sustainable manner.Its goal is not to eliminate commercial products, but to build on what is there and provide data and services that are missing.The Open Research Index should take in all research products (papers, software, datasets, workflows, etc.) from their publishers and recorders (journals, societies, domain repositories, government [open access] repositories, preprint servers, general repositories [e.g., figshare, zenodo]) and other services (CrossRef, ORCID).Each product should list authors and citations and allow people to search the resulting network.Users should also be able to interact with their own record and edit it, like Google Scholar allows.Robert Haines <robert.haines@manchester.ac.uk>•Caroline Jay <caroline.jay@manchester.ac.uk> • Daniel S. Katz <d.katz@ieee.org>• Robert McDonald <rhmcdona@indiana.edu>• Daniel Mosse <mosse@cs.pitt.edu>• Kyle Niemeyer <Kyle.Niemeyer@oregonstate.edu>8.11.2.Working group objective.The working group's plans are relatively simple to express, though quite complex to undertake: (1) Determine a plan to build an open research index that allows various stakeholders to satisfy their needs (2) Then determine if the plan is feasible (3) If so, then obtain resources (4) If successful, then build the index 8.11.3.Gap or challenge.Google and others provide some services now.But these services (and the underlying data) could be removed at any time.And the community cannot build new services.8.11.4.Relevant people and resources.People/organizations who might be willing to contribute (or whose expertise is needed), and how they will contribute: • ORCID, CrossRef, DataCite, ImpactStory/Depsy, altmetrics.org,Plum Analytics, GitHub, Open AIRE, CHORUS, FORCE11, COS, SHARE, CASRAI, Portico, softwareheritage.org,DBLP, eSTEP (CWO) • We could work with some initial publishers who don't have competing services (domain societies), PubMed • We need to discuss this idea with potential funders, and ask what data/services would they like to have?And are they willing to invest in this?The external resources that are needed are money and time; it is unclear how they can be brought in.8.11.5.Plans.At the time of the meeting, and today as well, it is unclear who has the time and energy to pursue this idea.
and/or Researchfish (UK).They have some problems that people do not like, the group could check what features they have and what the issues are.

•
Verify Best Practices Case Studies • White Paper WG, Open Research Index WG (9) Do