An underpinning theory and approach to applicability testing of constructive computational mechanisms

Abstract

Applicability testing of constructive computational mechanisms (CCMs) is a new challenge for both the academia and the industry. The overwhelming majority of the existing validation approaches focuses on the internal validity of CCMs (e.g. consistency, bias), while there is a shortage of efficient approaches for assessing the external validity (e.g. applicability, reusability). The objective of this paper is to clarify the concepts and criteria, and to develop an approach for a systematic evaluation of the applicability of a given CCM to cases that were not considered at design time. The approach is adapted from the validation square approach (VSA). The adapted methodology (A-VSA) makes it possible to evaluate CCMs from (a) theoretical structural, (b) empirical structural, (c) theoretical performance, and (d) empirical performance dimensions. Altogether eight indicators are introduced that support the evaluation process. The effectiveness of the A-VSA was confirmed through a case study, in which a specific CCM is considered and the strategy of the A-VSA was operationalized with three completely different application cases. As evidenced by the results, the proposed A-VSA establishes a tight coupling among the enablers embraced by a CCM and the aspects of theoretical and empirical validation, which approves the approach to be an efficient tool for defining the range and/or the extent of applicability. The advantage of the A-VSA is that it offers a way to transfer qualitative applicability evaluation into quantitative applicability assessment, which allows the use of both subjective statements and mathematical modeling in applicability testing. The results of the assessment can guide the adaptation work of a CCM when applied to an out-of-domain application.