Dr. Nicholas Danks

Summary

A framework for reproducible, reliable, and valid computational analysis in business analytics and research.

Person Months

6

Summary

Construct-based models have become a mainstay of management and information systems research. However, we believe these models are very likely overfit to the data samples they are estimated on, which makes them risky to use in prescriptive or predictive ways. We propose a novel methodological framework for these models that can highlight risks to out-of-sample generalizability in theoretically useful ways using a mixed-methods, explanatory-predictive approach. The proposed Composite Overfit Analysis framework: (1) gauges predictive performance of focal constructs, (2) identifies individual cases that exacerbate overfit, (3) identifies structural relationships between constructs that may not generalize well out-of-sample, and (4) guides qualitative analysis to explore the deeper reasons for such conflicts. Along the way, we seek to distinguish conflated terms in predictive and inferential literatures, and resolve methodological issues that prevent straightforward integration of predictive and inferential mechanics. We demonstrate the practical utility of our analytical framework on a technology adoption model in a new context.

Project Type

Research Article

Person Months

24

Summary

Structural Equation Models (SEM)​ are commonly used in MIS, marketing, management and behavioral sciences as a graphical way to represent a causal model between constructs, and as a statistical tool to test causal hypotheses, especially when multiple items are used to measure constructs. SEMs are becoming more complex, as knowledge of mechanisms advances and as more nuanced data can be collected. SEMs used in IS, marketing and behavioral sciences now contain complex relationships between independent and dependent variables (IVs and DVs), including multiple mediators, moderators, and control variables. We find models with moderated mediation, mediated moderation, and even moderated moderation. Researchers use SEM models instead of separate regression models because they aim at capturing the global relationship1 rather than a collection of single direct effects. Methods such as CB-SEM and PLS-SEM estimate the entire system simultaneously, yielding estimates for each of the direct causal effects (arrows in a SEM diagram). However, in such complex diagrams, it is not straightforward to identify which causal effects can be estimated, and moreover, which parts must be conditioned upon (and how) in order to estimate those effects. Moreover, in complex models contradicting theories sometimes posit different causal sequences, and selecting the best sequence among several competing alternatives can be challenging.