A. User-Centred Approach

COMPLEX-IT increases the usability of computational modelling by distilling these methods into their essential features and streamlining their integration– which is accomplished in two ways: functionality and interface design. COMPLEX-IT’s functionality is unique because it runs a specific suite of techniques that support case-based data exploration, modelling, forecasting and scenario simulation. In turn, COMPLEX-IT’s tab-driven interface provides users a seamless, concise and visually intuitive platform. Also, advanced users can examine, download or modify COMPLEX-IT’s algorithms, code, and outputs.

Currently (circa 2020), COMPLEX-IT’s suite includes (1) k-means cluster analysis, (2) the Kohonen topographical neural net, (3) a series of data visualization techniques, (4) a machine intelligence algorithm for data forecasting, and (5) a tab for simulating and exploring future scenarios. The simulating scenarios tab is a major methodological advance, as it provides an alternative to agent-based modelling and microsimulation for exploring how to influence, change or alter the course of complex data/systems.

Overview of K-means and COMPLEX-IT output

K-means (which is a type of semi-supervised learning) operates by separating cases into groups by minimizing the within group differences between a cluster’s associated cases [12]. K-means then iteratively moves cases between groups to achieve this goal. A final cluster is represented by a centroid, which contains the average values for each element of the case profiles in each cluster. Thus, distributions across clusters ideally are tightly packed around the centroid and unique from other clusters in the set.

Upon running the k-means algorithm, COMPLEX-IT will present the resulting cluster profiles and size of the clusters. Additionally, the user can request evaluations on the quality of the clustering: pseudo f and silhouette plots, two commonly used metrics for evaluating clusters. Pseudo f is an overall measure of how tightly cases are grouped within clusters and how separate or non-overlapping clusters are, with higher values indicating better performance. The silhouette plot allows for visual and quantitative inspection of cases fit within their clusters, see Figure 3. Through inspection of the clusters vis-à-vis domain knowledge or relevant theory and these quality measures, users can identify the best arrangement for the major and minor trends in their set of cases.

Overview of the SOM and its output

The SOM is a topographical neural network designed for unsupervised learning (i.e., machine intelligence), data visualization, and clustering [11, 15, 21]. This value of this approach is that, while users are required to set up some of the initial parameters, no set number of clusters are prechosen. Instead, the SOM is left to identify the key clusters and, for temporal data, the major and minor trends.

For example, while k-means requires users to look for a specific number of clusters amongst a dataset of COVID-19 cases, the SOM identifies the number of clusters that best fit the data based on the algorithm. It is because of this difference in approach that experts in the field recommend the SOM solution compared to the k-means solution to corroborate results or suggest further analysis [11, 15].

We chose SOM over other neural network techniques because it is a well-established approach with unique features for supporting human analysis of results [11, 15]. Unlike other artificial neural network techniques, the SOM classification model can be directly visualized and analysed to understand how cases were assigned to a neuron (cluster) outcome – which is the focus of the following tab. The classification model is directly interpretable because the SOM projects high-dimensional (multiple variable) input data onto a 2-dimensional grid solution (See Figure 4 below) that can be visualized in a variety of ways – which we discuss in the SOM analysis section. Thus, the SOM provides users, including those less familiar with machine learning techniques, a more direct way to compare their k-means and SOM results.

Options for running the SOM training Tab

The options on this tab concern SOM algorithm setup, including setting the size of the grid, weights initialization, the number of algorithm iterations, data scaling, and a seed for preserving the initialization and the learning rate. After training the SOM, ANOVA is run to provide information on which (if any) of the profile elements differed significantly across the neurons. This lends insight into what may be distinguishing factors across the neurons. Additionally, two quality measures are displayed, the quantization error and the topographical error. The first indicates how much, on average, cases diverge from their assigned neuron; and the second addresses the rate at which surrounding neurons are a good fit for a neuron cases across the grid. Preferably these have lower values close to zero, as they are measures of error.

Understanding the SOM Map

The central object of study for this tab is the SOM map, shown in Figures 4 and 5. As mentioned earlier, the SOM ‘maps’ complex data by creating a grid of n × n neurons. On this map, each neuron has a set of weights equal to the number of configurational variables in the case profile (i.e., causal conditions, factors, measures, etc.), which makes each neuron somewhat similar to a k-means cluster centroid. For example, looking at Figures 4 and 5, one sees a 5 × 5 SOM Map, with a total of 25 neurons (i.e., cluster solutions). We happen to call these neurons quadrants, which came from our work with various users, who favoured this terminology.

One also sees in Figure 4 that some of the quadrants do not have a bar-plot configuration. The same with Figure 5: some of the quadrants do not have cases. The reason is that these quadrants did not constitute viable cluster solutions for the data. And it is because of these “empty” quadrants, in part, that the SOM is referred to as unsupervised learning, as it lets a dataset of cases settle (map) onto the ‘best’ cluster solution across all quadrants, with cases iteratively finding the ‘right’ quadrant on which to reside. Unsupervised learning takes place by the SOM setting default weights for all the quadrants and then associating cases with the quadrants they most closely resemble, usually through a distance measure. The weights of the quadrants and their neighbours are then updated based on an adjustable learning factor, which also leads to similar cases ‘settling’ into comparable regions of the map. The degree of ‘learning’ for each quadrant (and its capacity to attract similar cases in its region) decreases over time. The SOM is assessed, as mentioned earlier, by the two validity measures quantization error and the topographical error.

In short, the SOM’s ‘topographical’ mapping of the data leads to quadrants in proximity having similar weights for their configuration of variables (i.e., factors, causal conditions, measures, etc.) and those further away having progressively larger differences in weights. In other words, quadrants (and hence cases) that are near each other on the map are more alike configurationally than those further apart. Hence why this approach is called the self-organizing topographical map.

Options for viewing the SOM map Tab

As suggested above, the strength of this tab for the SOM map is the variety of options it provides for visualizing and analysing the SOM’s results. They are broken down into two types: prototypes (i.e., the configurational factors/variables in one’s study) and observations (i.e., cases in the dataset).

Figure 4, for example, is a bar-plot of a study exploring four factors. The bar-plot displays the entire SOM grid and each quadrant will contain a series of bars or no bars if no cases were assigned to that particular quadrant. Each bar in a quadrant represents one of the elements of the cases’ profiles and should be read from left to right. Similar to k-means, the bars for each quadrant represent the average values for cases within it. Bars are centred on the global mean for any given case variable; therefore, if a bar is only a line within the quadrant, it is at the global mean, whereas bars higher or lower than the centre line indicate it is higher or lower than the global mean, respectively. In this way the bar-plot provides a view into the major and minor trends for the cases through quadrant profiles. As some quadrants may have similar profiles, full numeric data for each quadrant can be retrieved from the tab for generating your report. Quadrant profile data can then be used to group quadrants based on similarity and thereby simplify the set of major and minor trends.

Corroborating the SOM and k-means solutions

For some analysis, users may wish to compare the SOM solution to the k-means solution or even use one to corroborate the other. When corroborating, we recommend the two-step procedure advocated by [15], which found that a combination of k-means and SOM is often more useful (particularly for reducing misclassification) than the conventional approach of using just one technique or combining hierarchical clustering and k-means.

In terms of statistical validation, ‘goodness of fit’ for k-means is measured by pseudo f and silhouette plots; and for the SOM it is quantization error and topographical error. The first step in corroboration is to make sure both solutions fit the data.

Next is visual inspection of the SOM map. Using the names plot, users can explore how the k-means clusters are distributed across the map and the SOM cluster profiles they are associated with. Plotted cases follow the convention of displaying cluster ID and case ID overlaid on the SOM map. Two things need to be considered when making such comparisons. First, if one runs a large SOM map or chooses a sizable k-means solution, the resulting map will show a more fine-grain distribution of cases. In turn, smaller maps and k-means solutions will give the impression of things fitting better. The real comparison comes from using the bar-plot option to look for similarities in profiles: one is looking to see if the k-means profile for a case is similar to the SOM profile for the quadrant it is in. Second, visual inspection, while useful, has its limits. The report generation tab creates Excel files with the cluster and quadrant IDs for all cases. These files can be statistically explored further, which is particularly useful when using large datasets.

The goal of scenario simulation

The main purpose of this tab is to enable users to simulate different scenarios that the case trends in their dataset could experience, as well as identifying any associated contingencies or responses they may express or failures to change. Here we define ‘interventions’ as a particular type of simulated (but potentially real) scenario reflecting some proactive attempt to change one or more of the configurational factors for a cluster or trend (i.e., variables, measurements, causal conditions). Examples of interventions could be wider changes in the settings or systems in which the clusters are situated (i.e., economic or environmental changes); or they could be specific policy changes (i.e., improve schooling or access to health care) or changes within the cluster by its own cases (i.e., a community-level shift in thinking or behaviour). Other scenarios a user may wish to explore include reactive events originating from environmental forces or outside of proactive action, such as out-migration in a community. Better understanding from scenario simulations can also lead to more informed planning or strategic action, which may prove useful for areas like policy analysis and program evaluation.

Understanding the scenario simulation map

To begin, scenario simulation combines results from the previously trained SOM and k-means analysis, presenting the same SOM map used in the SOM analysis tab. However, unlike the previous SOM map, which plots all the cases in the study, this map plots the k-means clusters identified previously.

Scenario simulation takes this focus because the k-means solution offers a more concise user-identified summary. For example, if one had a study with N = 100 cases, simulating interventions into each and every one of them may prove unmanageably complex and time consuming. Still, given that the SOM map from the SOM analysis tab is preserved, the cases for any given cluster and their respective differences can always be re-examined. In other words, while exploring the scenario map, one can also explore the prior SOM Map to gain a more granular understanding of how interventions or changes made on the scenario map might differentially “play out” at the case level.

Running a simulation

By pressing the Model Setup and Run Clusters buttons, the scenario tab will be initialized, as shown in Figure 6. Below the SOM plot an editable table that allows the user to update one or more elements of the k-means case trends. Different scenarios can be explored by changing the relevant elements in the table for a given scenario and pressing Run Clusters. The updated case trend will be re-examined against the SOM quadrants and the impact of the scenario, if any, will be shown through which quadrant it is now most closely associated with. The SOM bar-plot is also plotted here as a reference for the quadrant profiles.

Exact estimation of the change a scenario causes a case trend may not be possible. Therefore, the scenario tab also offers sensitivity analysis to account for uncertainty in the efficacy of scenario influencing a case trend. The user can specify how much a change to one or more element in a case trend may deviate by, from zero to one hundred percent of the submitted change, both in the positive and negative direction. This can be accessed by selecting the case trend/cluster to test and pressing the Sensitivity button on the left panel. After inputting the deviation for changed element(s), a Monte Carlo simulation will randomly sample from across this range, testing and providing a summary of which quadrants the case trend would be associated with – which allows a key insight into how ‘sensitive’ the effect of the scenario is to projected deviations.