Open Science and Open-Source Software in Industrial Ecology

Open Science is a movement toward more accessible, transparent, and collaborative research practices that can benefit the scientific community and society as a whole [3]. By sharing knowledge and data openly and promoting collaboration among researchers and other stakeholders, Open Science can accelerate the pace of scientific discovery, improve the quality of research, and promote more inclusive and equitable access to scientific information [4]. Transparency and accessibility of models and their associated data are crucial for producing higher-quality science, increasing productivity, and improving the science-policy boundary in the energy research community [5].

Widespread use of open-access software can, at the same time, increase the quality, transparency, and reproducibility of Industrial Ecology (IE) research [6]. The growing data and computation intensity in IE research has led to a greater need for the development of scientific software, but this has also created challenges related to transparency, reproducibility, reusability, and collaboration. The authors of [6] propose a fourfold response to this problem. Firstly, the authors propose the implementation of existing general principles for the development of good scientific software in IE and related fields. Secondly, they argue that collaborating on open-source software could make IE research more productive and increase its quality and provide guidelines for the development and distribution of such software. Thirdly, the authors call for stricter requirements regarding general access to the source code used to produce research results and scientific claims published in the IE literature. Finally, a set of open-source IE modelling tasks, which they hope will turn their recommendations into practice.

As a result of these efforts, a Python toolbox for industrial ecology has been introduced, which includes a range of frameworks and modules such as Brightway2 [7], ecospold2matrix [8], pySUT [9], pymrio [10], and others. Recently, other Python packages such as pycirk [11, 12], and pyLCAIO [13] have also been developed to enable modeling of Circular Economy scenarios and streamline the hybridization of process-based Life Cycle Assessment and Environmentally Extended IO databases. Recently, unfold was released, a repository that improves the packaging and sharing of data, making it easier to reproduce life-cycle databases that are based on proprietary data sources [14].

This paper introduces MARIO, a Python-based IO modelling framework that builds upon and improves many of the features of the previously cited pieces of software. MARIO provides a scale- and database-agnostic approach to IO modelling, allowing the user to adopt any kind of IO table, ranging from a single- to multi-region, monetary- or physical-units, and symmetric (or IOTs) or supply-use (SUTs). MARIO’s key features include automatic parsing of various well-known available databases, an intuitive Application Program Interface (API) for applying advanced modifications to an existing database ranging from aggregation, creation of scenarios for policies and processes economic and environmental impact assessment, and extensions to new supply chains by adopting a hybrid-LCA approach. Finally, by standardizing the mathematical complexities of IO models and being designed both for Python newcomers and expert users, MARIO represents to modelers an efficient tool to avoid recursive and time-consuming operations while focusing on their analyses.

Underlying mathematics and nomenclature

A comprehensive reference text to deepen IO analysis theory and application scopes is [1]. However, for the sake of clarity, it is necessary to synthetically provide an overview of the main concepts as well as to define the nomenclature adopted in MARIO and along the paper itself.

As mentioned, IO tables represent an insightful instrument to inspect the economic transactions that occurred during one year. Economic accounts are stored in various matrices, specifically:

• Matrix Z includes the transactions of goods and services occurring from one industrial activity to another. In the case of a supply-use table (SUT), activities are distinct from the commodities they produce, therefore two sub-matrices can be identified within Z, namely a supply matrix S, representing the production of each commodity by the supplying activities, and a use matrix U, characterizing the inputs, in terms of commodities, required by each activity.
• Matrix Y shows the consumption of goods and services by those generally denominated as consumption categories, such as households and government. Also, information about changes in valuables and investments is reported in Y.
• Matrix V complements Z by reporting the information on the value added provided by each industrial activity to its output in addition to the costs sustained due to its input consumption (i.e. capital, labor, taxes, and subsidies).

Multiregional tables introduce information on trades by specifying the geographic origin of each economic account. Moreover, IO tables are also often extended with environmental transactions, tracking the interactions of industrial activities (matrix E) and consumption categories (matrix EY) with different environmental dimensions such as emissions, primary energy, water, and land use.

Given any IO table, it is possible to calculate the total production vector X, which reports the total production of each industrial activity (and of each commodity, in the case of a SUT), as described by Eq. 1 (i, in this case, is a vector with dimension equal to the rows number of Z). It is possible to express any of the above-mentioned matrices (except for Y) per unit of total production, calculating the so-called technical coefficients matrices by dividing the desired matrix by the inverse of X. An example is provided by calculating the intersectoral transactions coefficients matrix z in Eq. 2. Please note that the “hat” symbol indicates a diagonalized vector. The same equation, applied to matrix E, leads to the derivation of matrix e which can be identified as direct environmental impact coefficients matrix, or production-based accounting impact matrix.

Following Eq. 2, Eq. 1 can be re-arranged to derive the Leontief production model (Eq. 3), expressing the total production of each activity (and/or commodity) represented in the IO table that is required to fulfil given final demand yields; I, in this case, represents an identity matrix and w is conventionally known as Leontief Inverse matrix.

IO tables can also be used for consumption-based impact accounting (or footprinting analysis), by adopting the so-called Leontief impact model (Eq. 4). Unlike e, matrix f, called specific footprint matrix, allocates to each industrial activity not only its direct but also its indirect environmental impact.

Furthermore, IO tables are often adopted to perform scenario analyses, generally implemented by modelling a perturbation (or shock) of the economic system and by evaluating the difference between the new and the original system. Eq. 5 describes how the matrices of a new shocked economic system t + 1 can be seen as the sum of the corresponding matrix of the original system t and a matrix represented only the shock to be implemented, indicated as $\Delta {\left(z,v,e,Y\right)}_{ }$M6 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{\bf \Delta} ({\bf z}, {\bf v}, {\bf e}, {\bf Y})\] \end{document} .

Code Structure

Before deepening the code structure, a preliminary glossary of the main terms and items is provided in Table 1 and adopted henceforth in the paper.

The core functionalities of MARIO are implemented through an Object-Oriented programming paradigm, specifically by means of two classes. These classes are named CoreIO and Database.

The CoreIO class encompasses general methods and functionalities for any IO model, including tasks like calculating desired matrices and checking whether a table is balanced. On the other hand, the Database class contains additional methods and functionalities specifically tailored for scenario analyses, such as shock analysis.

In MARIO, an IO table is structured as a set of matrices, represented by MultiIndex Pandas DataFrames [15]. This allows for a unified representation of various IO tables by utilizing different levels of indexing. Regarding MARIO scenario analysis, all the matrices for a particular scenario are gathered into a nested Python dictionary after they have been calculated. In practical terms, through this dictionary, it is possible to compare multiple shocked versions of the same table with the baseline one.

Each instance is additionally associated with another object called MARIOMetaData. This object serves as metadata and tracks all the changes made to the instance, as well as the analyses performed by the user. The metadata keeps a record of the instance’s history, which can be printed out as a JSON or TXT file, upon user request. An essential representation of MARIO’s structure is sketched in Figure 1.

Tables parsing

One of the main challenges associated with using IO tables is the lack of a standardized approach for storing and organizing the data. Each specific table follows its structure, both in terms of data storage and labelling conventions for elements within the database. For example, Eora [16] stores all the matrices of a single-region table in a unique TXT file, the monetary version of EXIOBASE 3 [17] stores each matrix in a distinct TXT file, while the hybrid version [18] adopts CSV files and a significantly different manner to store environmental transactions.

This challenge becomes even more complex when working with SUTs or hybrid tables that incorporate additional dimensions of information into the IO framework. Additionally, non-structured tables, such as those developed by specific groups or for particular projects, add further inconsistencies. The lack of standardized structure among different existing tables hinders the reproducibility of IO modelling exercises, as modellers constantly need to adapt to different paradigms for reading and handling various types of tables.

To overcome this issue, MARIO adopts its specific structure. Regardless of the type or origin of the data, any table read by MARIO is transformed into a standardized format according to which each matrix is characterized with specific indexing, as reported in Table 2.

MARIO incorporates a set of parsing functions tailored to differently structured tables:

• Eora, namely Eora26, single-region IOTs and single-region SUTs;
• EXIOBASE 3, including both IOTs and SUTs as well as monetary and hybrid versions;
• Eurostat SUTs.

When dealing with non-structured tables, MARIO is capable of reading datasets from Excel (XLSX), CSV, or TXT files provided that they have been firstly structured in a pre-defined shape. Furthermore, with the release of v.0.2.0, it is also possible to import data from a pymrio object [10] (IOSystem class) and transform it into a MARIO Database class. In version 0.2.2, MARIO can also exploit the pymrio table downloading functions. More about the tables available for download at pymrio documentation: https://pymrio.readthedocs.io/en/latest/notebooks/autodownload.html.

Calculating matrices

The CoreIO class includes a function called calc_all, which enables users to calculate a specified list of matrices for a given scenario within the model. If the calculation of a particular matrix relies on another matrix that is not yet present in the scenario, MARIO will determine the appropriate order of calculations for different matrices to generate the requested matrix.

In addition to the basic matrix calculations, MARIO offers a query function that allows users to perform more advanced automatic calculations across different scenarios. This includes calculations such as determining the absolute or relative changes of a matrix among different scenarios.

Furthermore, all the mathematical calculations related to IO analysis are programmed as separate functions. This enables users to utilize these functions outside of the main structure of a MARIO Database or CoreIO class, providing greater flexibility in their calculations.

Modifying an instance

The Database class in MARIO offers several capabilities to alter an existing instance, including the following functionalities:

• Aggregating (or just renaming) the instance into different dimensions of data by reducing the number of regions, sectors, and so on.
• Adding new sectors in case of an IOT, or new commodities and activities in case of a SUT.
• Adding or dropping satellite accounts to the existing instance.

Excel-based interaction

To enhance the user experience for MARIO users, several functionalities have been designed to be accessible through automatically generated Excel files which serve as an interface. These interfaces allow modellers to provide the minimum required inputs to make their changes. Furthermore, the users are supported by list validation, to minimise input errors and typos.

For instance, to aggregate an instance, users can utilize the get_aggregation_excel function, which generates an Excel file with pre-filled indices related to the instance sets such as the list of regions, sectors, satellite accounts… This Excel file enables easily defining the desired aggregation and then it is enough to read back the file to execute the aggregation process. Similarly, tasks such as adding a new sector, or satellite account, or implementing a shock can be accomplished using automatically generated Excel files within MARIO.

Surely, however, every task which is simplified through Excel interfaces can be also performed directly in Python, in case of more advanced necessities.

Example application

In this section, we show a simple yet comprehensive tutorial on the main novel features of the MARIO modelling framework. Check the “Software location: Emulator environment” section below to get access to the Jupiter Notebook related to this example, along with all the supporting Excel files.

For further information and tutorials, we developed an online video-course available at Polimi Open Knowledge platform [19]. You can also visit the Zenodo repository dedicated to our “Input-Output analysis and modelling with MARIO” online course [20].

The following example requires the user to first download the EXIOBASE hybrid supply-use table referred to the year 2011 [18]. The downloaded folder can be stored in any directory. The path to this folder is indicated henceforth with exio_path.

To start parsing the downloaded table, it is necessary to have MARIO installed (link to the installation in the documentation [21]), to import it, and to call the save a MARIO Database object called world by using the parse_exiobase function, indicating the type of table and the type of units. Also, it is possible to filter over the desired environmental extensions. Any of the sheets stored in the “MR_HSUTs_2011_v3_3_18_extensions.xlsx” provided in [18], files can be provided, as a list. In this case, we are going to use only the Emiss extensions, referring to the emissions transactions.

import mario

world = mario.parse_exiobase(path=exio_path, table=’SUT’, unit=”Hybrid”, extensions=[‘Emiss’])

To investigate the parsed table, it is often useful to get the list of labels of a desired set or to extract the list of labels containing a determined string within a set. These two features are allowed by using the get_index and the search functions respectively. Examples of getting a list of all the labels included in the “Activity” set and of getting a list of all the activities including the “gas” string among the same set are reported as follows:

world.get_index(‘Activity’)

world.search(‘Activity’,’gas’)

It is also possible to calculate new matrices, indirectly using the above-mentioned calc_all function. To get a new, or an already calculated matrix in the baseline scenario, it is enough to use the matrix property as shown for the case of the footprint coefficient matrix f (already described in Eq. 4).

world.f

The first advanced table modification we perform in this tutorial is aggregation. It is possible to export an empty Excel template by calling the get_aggregation_excel function and by providing a path aggr_path.

world.get_aggregation_excel(aggr_path)

The Excel file must be filled in at least one of its sheets. Each sheet presents the list of labels of the sets it is dedicated to and the user is required to map the original labels with new desired labels to reduce the numerosity of the desired sets.

Figure 2 shows the aggregation of the commodities of the raw EXIOBASE table into a new configuration.

It is worth noting that in the case of a hybrid-units table, like the one in this example, the user must pay attention to the homogeneity of the units relative to the commodities that are going to be aggregated. A specific error is raised in case this rule is neglected.

To aggregate the Database, it is just required to read back the filled Excel file by using the aggregate function. It is finally possible to check the new dimension of the Database by printing it. The aggregated database has now 5 regions, 50 activities, 44 commodities, and just 1 satellite account, instead of the original 48 regions, 164 activities, 200 commodities, and 66 satellite accounts.

world.aggregate(aggr_path)

print(world)

The second modification to the table would be the extension of the newly obtained aggregated table to a new supply chain. In this example, we will display how to model a new industrial activity, namely the European supply chain of battery manufacturing. A new activity called “Manufacture of batteries” as well as a new commodity called “Batteries” need to be added to the table. Also in this case, the user can be supported by an integration with dedicated Excel files. In the case of a SUT, two files are requested, one to add the “Batteries” commodity and a second to add the new “Manufacture of batteries” activity. It is first necessary to create a list of new_activities including the label of the new activity to be modelled, then the user shall provide a path add_activity_path where the template Excel file would be exported. The get_add_sectors_excel function can finally be executed, by specifying also that the item which will be modelled in this template is an activity and in which regions this activity will be implemented.

new_activities = [‘Manufacture of batteries’]

world.get_add_sectors_excel(

      new_sectors=new_activities,

      regions=world.get_index(“Region”),

      item=’Activity’,

      path=add_activity_path,

      )

To properly model a new activity, it is necessary to provide a characterization of the input structure of the new activity in the “input_from” sheet (Figure 3), the unit of measure of its output (“units” sheet), and possibly the related satellite transactions (“Satellite account” sheet). The characterization of the input structure is intended to be performed in the form of a life-cycle inventory [22] and therefore needs to be supported by appropriate data collection as well as by a harmonization process to correctly map the collected literature data with the sets of the adopted table.

Once the template to add the activity is filled, the add_sectors function shall be called to add the new activity. It is possible to notice that the Database now has 51 activities by printing the Database object.

world.add_sectors(

      new_sectors=new_activities,

      regions=world.get_index(“Region”),

      item=’Activity’,

      io=add_activity_path

      )

print(world)

The same procedure needs to be followed for the addition of the new commodity: it is required, therefore, to provide a list of new_commodities including “Batteries”, and to generate another Excel template (add_commodity_path).

new_commodities = [‘Batteries’]

world.get_add_sectors_excel(

      new_sectors=new_commodities,

      regions=world.get_index(“Region”),

      item=’Activity’,

      path=add_commodity_path,

      )

When filling the file, in this case, it is necessary to indicate that the “Batteries” commodity is produced by the new activity in the “output from” sheet (Figure 4) and that the only consumption of batteries is in the European final demand (“Final consumption” sheet). The “Batteries” commodity is measured in “kWh” (“units” sheet).

Once the template Excel to add the commodities is filled it is necessary to call the add_sectors function once more. Afterward, it is possible to notice that the Database now has 45 commodities by printing the Database object.

world.add_sectors(

      new_sectors=new_commodities,

      regions=world.get_index(“Region”),

      item=’Commodity,

      io=add_commodity_path

      )

print(world)

The tutorial analysis is finalized by implementing a scenario analysis, therefore a shock would be modelled into the table. MARIO allows exporting an empty Excel template to be filled with the information regarding the desired modifications to be applied to the original table. It is enough to call the get_shock_excel function and provide a path (shock_path). In this example, the environmental impact of the increase in European final consumption of domestic batteries up to 1 TWh is evaluated.

world.get_shock_excel(shock_path)

To implement such shock, the user would need to navigate to the Y sheet and provide the instructions to update the final consumption of the European “Batteries” commodity in Europe (Figure 5).

Once the shock template is filled, the shock_calc function implements the shock in the Database object. It is necessary to provide which matrices are intended to be modified (just Y for this particular shock) and the name of the new scenario which will be created (“increased batteries demand” in this case). It is possible to print the Database object to check that two scenarios are present in the instance after this operation.

world.shock_calc(

      io=shock_path,

      Y=True,

      scenario=’increased batteries demand,

      )

print(world)

To conclude, it is possible to visualize the results, for example, the impact caused by the implemented shock on the consumption of commodities required for the production of batteries. Such impact can be provided with respect to the baseline scenario. To do so, we can use the plot_matrix function, which is based on the Plotly library [23], and apply a set of desired filters.

world.plot_matrix(

      matrix = ‘U’,

      item = ‘Commodity’,

      facet_col=’Commodity_from’,

      x = ‘Activity_to’,

      color=’Region_from’,

      path= plot_path,

      base_scenario = “baseline”,

      filter_Activity_to= [‘Manufacture of batteries’],

      filter_Region_from=[‘China’,’EU27+UK’,’RoW’],

      filter_Region_to=[‘EU27+UK’],

      filter_Commodity_from=[‘Chemicals’,’Non-ferrous metal ores’,’Electricity’, ‘Natural gas’],

      )

The resulting plot is an interactive HTML file, as shown in Figure 6, which reports the consumption of the new “Manufacture of batteries” activity of selected commodities (one per subplot, each expressed in its specific unit of measure) coming from different regions (legend colors). The values are expressed as a variation with respect to the baseline scenario.

Archive

Name: Zenodo

Persistent identifier:https://doi.org/10.5281/zenodo.5879382

Licence: GPL v3

Publisher: Mohammad Amin Tahavori

Version published: 0.2.2 and earlier versions.

Date published: 16/09/23 (version 0.2.2)

Code repository

Name: Github (also hosted on PyPI)

Identifier:https://github.com/it-is-me-mario/MARIO

Licence: GPL v3

Date published: 16/09/23 (version 0.2.2)

Emulation environment

Name: Jupiter Notebook of the displayed example application

Identifier:https://github.com/it-is-me-mario/Tutorials/blob/main/JORS%20paper/Tutorial.ipynb

Licence: GPL v2

Date published: 22/05/23