Accepted on 15 Feb 2019
Submitted on 02 Jul 2018
Science is mainly a collective activity. We can go further only by “standing on the shoulders of giants” (and of a huge number of technicians and scientists). Science is a lot about how to build new knowledge from the work of others and hence, the exchange of ideas is a fundamental aspect. In the last decades, we have lived through a revolution on how people exchange ideas. Computers of all kinds (from smartphones to HPC clusters) connected by a world wide web are used for human communication and also for many other applications. It has become nearly effortless to reproduce and exchange ideas and data. The set of intangibles that grows when shared and degrades when hoarded, such as knowledge and love, has been somehow extended. The information technology revolution open doors to fantastic opportunities for human collaboration, and of course, for science.
Web related activities alone account for at least 5% of GDP in the USA and the European Union.1 A huge amount of money (and work) is invested on the related technologies. We are familiar with the most prominent companies involved in this dynamics (Google, Facebook, etc.), but there are also several smaller and sometimes lesser known organizations. Most of these companies base part of their work on the open-source paradigm contributing to open-source languages, libraries, software and operating systems, while also using them — a win-win situation for both the corporations and the community. This has lead to deep changes in software engineering, with a massive use of open-source methods and tools, for example distributed version control systems (DVCS) and web-based source development platforms.
The computer performance continues to increase exponentially, now also with the help of Graphical Processing Units (GPU). This gave way to a big boom in practical uses of data science and machine learning, which drives a strong research on artificial intelligence. Such developments contribute to progresses in open-source software. To summarize, there is a strong dynamics in play around the use of computers (in particular with the web) and this creates very efficient tools and methods for collective work and software development.
These changes in our world also reflect in the way science is done. Software and programming in science occupy a much bigger place than before. The role of software in science has changed. In the past, coding was sometimes considered as an inferior activity by some scientists. The focus was on the theory and the mathematical demonstration, which had to be elegant as it gets included in the articles. In contrast, it was normal to write crude code and to just show the results. Nowadays, codes tend to be at the heart of research.
“Open-science” is a new trend taking advantage of these new facts. Pioneering attempts are being made to do better science, improving reproducibility and collective efficiency, by using the open-source methods and tools for science and sharing and collaborating via the world wide web.
FluidDyn is a project to foster open-science and open-source coding in Python in the field of fluid mechanics. The project envisages to provide the technical framework to allow collaborative development of tools useful for the fluid mechanics community, and to do science with open methods. We provide examples of solutions for:
We hope that such a clean framework will facilitate contributions from scientists in the field and that we can build together a nice, user-friendly and efficient ecosystem specialized in research and teaching in fluid dynamics.
FluidDyn was originally intended to be a single package to perform experiments and simulations. Since a typical user may not be involved in both experiments and simulations and also, due to increasing complexity as a virtue of rapid development cycle made the need to decentralize FluidDyn evident. Now, FluidDyn project hosts a number of specialized packages, namely:
A detailed presentation on the above packages can be found in their respective documentations on the web and for fluidfft and fluidsim in the two companion papers [4, 5]. The code base was designed to follow Python 2.7 syntax during its genesis. Now, it has been made forward compatible with Python 3 through the use of external package future. We now tend to abandon Python 2.7 support for the next releases of the package. This article will now focus on the base package fluiddyn.
All functions and classes defined in fluiddyn are pure Python elements, meaning that no extensions are implemented in fluiddyn. Thus the package fluiddyn is extremely easy to install with just a pip install command and no compilation.
The package fluiddyn is organized into five sub-packages:
Sub-packages io, util and calcul are the largest in terms of lines of code and provides the Application Programming Interfaces (API) to support fluidsim, fluidlab and fluidimage. For the sake of brevity, we shall only describe here some of the most important modules.
Sub-package paramcontainer defines class ParamContainer which is a hierarchical container for any type of parameters. As shown in this tutorial, various strengths of an object of this class include:
Thus, it makes it a much more robust implementation for saving key parameters, compared to conventional methods which rely on text or CSV files.
This module provides classes to iterate over files. It is a common task in data processing to understand how to organize a sequence of files from filenames and formats. Typically, one has to form smaller sets of arrays contained in the files. For example, we may have a sequence such as:
(im1_1.png, im1_2.png, im1_3.png, im2_1.png, im2_2.png, im2_3.png)
from which we could create different subsets like,
((im1_1.png, im1_2.png, im1_3.png), ((im1_1.png, im2_1.png),
(im2_1.png, im2_2.png, im2_3.png)) (im1_2.png, im2_2.png),
(im1_3.png, im2_3.png))
The classes of this module allows one to do it with a quite simple and general API as shown in the tutorial on the module.
This simple module makes simultaneous sequential and MPI programming a breeze by providing number of processes, nb_proc = 1 and rank = 0, when used in sequential mode, otherwise providing the appropriate values provided by mpi4py package. Use of this module thwarts coding several if-else clauses. If the program was using MPI, also defines the variable comm as an alias for the MPI.COMM_WORLD communicator.
Thin wrapper for an unified API using classes around the packages pyfftw and scipy.fftpack. It is very easy to perform forward and inverse Fast Fourier Transforms (FFT) in one-, two- and three-dimensions. The FFT can be multithreaded if the environment variable OMP_NUM_THREADS is defined.
The package fluiddyn also provides few command-line utilities to perform simple tasks useful for a scientist developing with Python and using the FluidDyn packages.
The package fluiddyn currently supplies unit tests covering around 70% of its code. These unit tests are run regularly through continuous integration on Travis CI with the most recent releases of fluiddyn’s dependencies and on Bitbucket Pipelines inside a static Docker container. The tests are run using standard Python interpreter with all supported versions.
For fluiddyn, the code coverage results are displayed at Codecov. Using third-party packages coverage and tox, it is straightforward to bootstrap the installation with dependencies, test with multiple Python versions and combine the code coverage report, ready for upload. It is also possible to run similar isolated tests using tox or coverage analysis using coverage in a local machine. Up-to-date build status and coverage status are displayed on the landing page of the Bitbucket repository. We also try to follow a consistent code style as recomended by PEP (Python enhancement proposals) 8 and 257. This is also inspected using lint checkers such as flake8 and pylint among the developers. The code is regularly cleaned up using the Python code formatter black.
Windows and any POSIX based OS, such as GNU/Linux and macOS.
Python 2.7, 3.4 or above. For the next versions, we will drop Python 2.7 support and Python >= 3.6 will be required.
We list here only the dependencies of the base package fluiddyn.
Name: PyPI
Persistent identifier:https://pypi.org/project/fluiddyn
Licence: CeCILL-B, a BSD compatible French licence.
Version published: 0.2.4
Date published: 02/07/2018
Name: Bitbucket
Persistent identifier:https://bitbucket.org/fluiddyn/fluiddyn
Licence: CeCILL-B
Date published: 2015
English
As a library, fluiddyn has been used in the project’s specialized packages. The common code base for packages with such varied applications is a proof of fluiddyn’s versatility and generality. It can be used in other packages outside the FluidDyn project easily depending on the need. The command-line tools can be useful to all scientists working with Python. Other use cases could be:
There is no formal support mechanism. However, bug reports can be submitted at the Issues page on Bitbucket. Discussions and questions can be aired on instant messaging channels in Riot (or equivalent with Matrix protocol)4 or via IRC protocol on Freenode at #fluiddyn-users. Discussions can also be exchanged via the official mailing list.5
FluidDyn is an attempt to set off collaborative dynamics based on open-source development in fluid dynamics research. We shall try, with this project, to explore the possibilities of open-source in science and fluid dynamics by fully exploiting the new open-source tools and methods.
The project is right now in a preliminary stage. Packages are actively evolving with interesting features and a framework for collaborative development, packaging, documenting and testing is now well set. However, the community around the project is currently tiny and now, we have to work on attracting users and developers since an active community is a criteria for success and sustenance of an open-source project.
Will people use the FluidDyn tools and collaborate through the project FluidDyn? There are clearly many challenges and potential barriers:
On our side, we also have positive points. The quality of the tools we use (Python and its scientific ecosystem, Mercurial, Read the Docs, Jupyter, …) is impressive. Scientific code is done to be read and to transmit ideas. To this effect, Python is among the best languages today. Python starts to be a standard tool in fluid dynamics, especially used for CFD (Dedalus, SpectralDNS, TriFlow, PyLBM, Oasis, PyFR, FEniCS, Cassiopee, pyCGNS, etc.) and data analysis (OpenPTV, PyPIV). Moreover, we can benefit from the dynamics of Python and of emerging subjects like deep learning and the Internet of Things. Finally, if we manage to gather a community of users and of developers, the collective efficiency related to open-source methods and tools can be a strong booster.
1See the report by Internet Association titled “Refreshing Our Understanding of the Internet Economy”.
3We use FluidDyn (with capital letters) to name the project and fluiddyn for the base package.
We thank the CNRS to finance the work of Pierre Augier while giving freedom in terms of scientific project. Similarly, Ashwin Vishnu Mohanan could not have been as involved in this project without the kindness of Erik Lindborg. We thank Antoine Campagne, Miguel Calpe Linares and Julien Salort for their implications in the development of some of the FluidDyn packages. We are grateful to Bitbucket for providing us with a high quality forge compatible with Mercurial, free of cost. Finally, we thank Gabriel Moreau and Joël Sommeria for their constant will to discuss and share their knowledge.
The authors have no competing interests to declare.
Campagne, A, Alfredsson, H, Chassagne, R, Micard, D, Mordant, N, Segalini, A, Sommeria, J, Viboud, S, Mohanan, A V, Lindborg, E and Augier, P 2016 First report of the MILESTONE experiment: Strongly stratified turbulence and mixing efficiency in the coriolis platform, VIIIth International Symposium on Stratified Flows (ISSF, San Diego, USA).
Leclercq, C, Partridge, J L, Augier, P, Dalziel, S B and Kerswell, R R 2016 ‘Using stratification to mitigate end effects in quasi-Keplerian TaylorCouette flow’. J. Fluid Mech, 791: 608–630. DOI: https://doi.org/10.1017/jfm.2016.44
Lindborg, E and Mohanan, A V 2017 ‘A two-dimensional toy model for geophysical turbulence’. Phys. Fluids, 29(11): 111114.
Mohanan, A V, Bonamy, C and Augier, P 2019a ‘FluidFFT: Common API (C++ and Python) for Fast Fourier Transform libraries’. J. Open Research Software (DOI).
Mohanan, A V, Bonamy, C and Augier, P 2019b ‘FluidSim: Modular, objectoriented python package for high-performance CFD simulations’. J. Open Research Software (DOI).
Salort, J, Rusaouëen, É, Robert, L, Du Puits, R, Loesch, A, Pirotte, O, Roche, P-E, Castaing, B and Chillà, F 2018 ‘A local sensor for joint temperature and velocity measurements in turbulent flows’. Review of Scientific Instruments, 89(1): 015005. DOI: https://doi.org/10.1063/1.4989430