Source: Unveiling the Mystery: Understanding Black Box AI and Its Real-World Implications -
Ezeiatech The recent proliferation of deep neural networks has introduced a conflict between black box properties and the need for interpretability in critical domains. Modern artificial neural networks, including multi-layer perceptrons, convolutional networks, and foundational transformer models, are distributed non-linear systems of hundreds of millions of parameters. While such complexity is necessary to obtain performance on large-scale datasets, it makes the inner workings of the model difficult to understand. However, in the highly regulated domains of healthcare and finance, opaque decision-making processes often lead to model misuse, discrimination, or faulty diagnoses with fatal outcomes. This essay discusses the application of interpretable machine learning methods to satisfy the regulatory requirements for the explainability of deep learning models. Across various jurisdictions, regulatory agencies have imposed requirements for model explainability on the finance and healthcare industries. In the United States, the Equal Credit Opportunity Act and the Fair Credit Reporting Act require lenders to supply adverse action notices that explain the rationale behind credit rejections or unfavorable terms. At the same time, the Federal Reserve’s model risk management guidance mandates strict validation of banking models, including conceptual checks of inputs and outputs. As applied to deep learning models, these requirements take the form of interpretability: to meet the legal expectations, one needs to be able to explain how particular features influence the final score and whether latent variables encode any protected characteristics. In healthcare, the General Data Protection Regulation (GDPR) of the EU grants the right to an explanation for all individuals subject to automated decision-making. Moreover, the recently proposed AI Act designates medical diagnosis and credit risk assessment systems as high-risk, which imposes even stricter transparency requirements. When a doctor receives an automated recommendation for a particular treatment, the physician should be able to review the supporting evidence to ascertain the recommendation’s validity. Thus, meeting the regulatory requirements for model explainability is a precondition for adopting deep learning in healthcare and finance. Researchers have proposed a number of model-agnostic methods that enable post-hoc interpretation of the behavior of a deep neural network. Given a particular instance that the model has predicted, such methods seek to identify a set of inputs that were relevant to the prediction. One of the most popular approaches to the interpretation of deep learning models is Local Interpretable Model-agnostic Explanations (LIME). The method, proposed by Ribeiro et al. in 2016, approximates the behavior of a complex model as a local linear function around the instance that needs interpretation. To explain the prediction for one example, the LIME algorithm samples points around it, predicts their labels with the complicated model, and trains a sparse linear model that approximates the behavior of the neural network in this small region. This linear approximation is obtained by minimizing an objective function that consists of local accuracy and simplicity terms. When applied to healthcare, for example, LIME highlights the terms in the clinical note that have contributed to the risk prediction. Another approach to model interpretation is Shapley Additive exPlanations (SHAP), which applies game theory to estimate the contribution of each input feature to the final output. The method is grounded in the concept of Shapley values, a solution concept in cooperative game theory that estimates the average contribution of each feature to the prediction. By calculating SHAP values, one can obtain an explanation of what inputs have contributed to the model output. SHAP is more theoretically grounded than LIME, as it satisfies the properties of local accuracy, missingness, and consistency. The model-agnostic methods discussed above can be useful in practice, as they can be applied to any model. Nevertheless, black box neural networks frequently fail to satisfy the interpretability requirements even after post-hoc processing. For instance, the LIME algorithm can generate explanations for inputs that lie far from the training distribution. Moreover, an adversary could train a pathological model that fools the explanation algorithm. Finally, as both LIME and SHAP only approximate the behavior of the model, they can be used to hide potential biases encoded in the weights of the neural network. To address these limitations, researchers have turned to mechanistic interpretability, which analyzes the inner workings of a neural network. Mechanistic interpretability aims to discover how a deep neural network accomplishes its tasks by inspecting the weights and activations. The recent line of work on mechanistic interpretation is grounded in the concept of activation patching, also known as causal tracing. This approach seeks to identify the specific components of a model that participate in processing some information or performing a computation. For instance, the causal tracing procedure can be used to determine what attention heads in a transformer-based language model carry information about the answer to a question. The activation patching procedure usually involves three steps: clean run, corrupted run, and patched run. In the clean run, the model processes some input and saves the activations that occur in all operations. Then, the model processes a modified input in the corrupted run. Finally, the researchers perform a patched run, where they replace some activations from the corrupted input with activations from the clean input. If the patched input results in the output close to the clean input, this implies that the replaced part of the model is responsible for some computations. Activation patching can be applied to find circuits, sets of operations that contribute to some aspect of the model behavior. For instance, in a medical language model, certain circuits carry the information about the patient’s medical history to the output. In finance, activation patching can help diagnose what operations turn the numbers in a structured report into a liquidity risk score. The circuits discovered with activation patching can then be used to evaluate the model for regulatory compliance. For instance, in the context of fairness, the model may appear to make decisions based on protected attributes such as race or age, when in fact, it uses proxies encoded deep in the model weights. In the field of oncology, for instance, SHAP values can be used to explain to the patient why the risk score for metastases is high. At the same time, activation patching can be used to ensure that the language model encoder does not utilize any protected attributes, such as race or gender. When applied to a financial model, SHAP values can be used to generate the adverse action notice in accordance with the Equal Credit Opportunity Act. The values would highlight the inputs that have contributed to the adverse action, such as a high utilization of a revolving line of credit. On the other hand, activation patching can verify that the model does not encode any protected attributes in its latent representations. Thus, in both fields, activation patching can be combined with post-hoc explanation methods to satisfy the regulatory requirements for model transparency. Sources: https://arxiv.org/abs/1705.07874 https://arxiv.org/abs/1606.05386 https://arxiv.org/abs/2202.05262