Source: https://blogs.icrc.org/law-and-policy/2024/09/03/the-problem-of-algorithmic-bias-in-ai-based-milit ary-decision-support-systems/ The use of machine learning and autonomous decision-making systems in critical recruiting, credit approval, and healthcare triage functions may introduce new forms of discrimination that are not easily apparent. Such systems rely on extensive data about past events to construct predictive models that help select the best options among thousands of available ones. Yet, often such systems may actually embody historical prejudices, thus embedding them in the way decisions are made by the system. The phenomenon is particularly acute in such areas as recruiting, credit rating, and hospital treatment allocation where the potential for discrimination is particularly large. One of the ways such systems may discriminate is through using proxies for forbidden attributes. For the purposes of this discussion, proxies are features that may be used by the algorithm to indirectly encode the forbidden attribute. Forbidden attributes are those which cannot be used for fair decision-making, such as race, gender, or other protected characteristics. When building decision-making algorithms, it is crucial to use features that are independent of the protected characteristics. Yet, for such characteristics, there often are proxies in the form of other features that can be used to encode the same information. A wide array of features may be used as proxies for forbidden attributes. Some examples may include location, college affiliation, the company one works for, specific words used in phrases or applications, and many others. Thus, each of the protected characteristics may have dozens or hundreds of proxies that allow the algorithm to indirectly encode the protected characteristic. Algorithms used in automated recruitment functions may acquire such proxies when learning to score applicants’ resumes or video responses. When learning to rank, such algorithms often use the historical data about which applications were successful to determine which features can be used to differentiate between successful and non-successful applicants. However, when the historical data contains discriminatory biases, the algorithms may learn those to further improve their capacity to differentiate between the groups. The problem becomes particularly acute when the same protected characteristic is not directly used in the data but can be inferred from other available features. In practice, such proxies can include attendance of women’s college or having a name that is common for a particular ethnic group. In the context of credit approval algorithms, proxies would allow the algorithm to infer the protected characteristics from the consumer behavior of the applicants, thus potentially denying credit to minorities simply on the basis of their race or ethnicity. Healthcare triage algorithms may use proxies to deny care to minority groups by determining that they are less likely to need chronic care. Chronic care management algorithms are used by hospitals to determine which of the hospitalized patients should be accepted for specialized chronic care treatment. Such algorithms use the number of times the patient has been hospitalized to determine how much care the individual needs. Yet, the chronic care algorithm was shown to systematically underestimate the level of care needed by Black and Hispanic patients because fewer of them had been hospitalized before compared to White patients. The latter had more access to healthcare, and therefore were more likely to be hospitalized when an incident occurred. By using past hospitalization as a proxy for likelihood to need chronic care, the algorithm discriminated between White and Black patients, failing to recommend the treatment for the latter even though they needed it more. There are multiple ways to deal with the phenomenon, each of which has its own drawbacks. The simplest way to deal with the presence of proxies is to ensure that none of the features that are statistically significantly correlated with the protected characteristics are present in the dataset. Such features can either be removed or modified in the process of pre-processing the data before training the machine learning classifier. However, such a procedure often leads to worse generalization performance due to the removal of useful information. Another approach is to modify the learning process itself so that the learned weights are independent of the protected characteristics. In practice, it means adding additional terms to the loss function that penalize the weights that encode the relationships between the protected characteristics and the outcome. However, each of the fairness notions described earlier has to be converted to specific mathematical terms, and the resulting objective may become significantly more difficult to optimize. Moreover, for most notions except statistical parity, the modified objective function has to compete for the same weights that determine the prediction function, thus introducing additional challenges. Auditing and modifying such algorithms to ensure that they do not use proxies for forbidden attributes requires significant effort and is not guaranteed to produce satisfactory results. In the context of recruiting, credit approval, and hospitalization algorithms, it is of paramount importance to ensure that the algorithms used are not using proxies for protected characteristics to make decisions. Yet, even when such proxies are removed or adjusted for, there are still multiple fairness notions that the modified algorithm should satisfy, and the adjustments required to achieve that may be complicated and suboptimal.
Sources
https://www.nist.gov/publications/proposal-identifying-managing-bias-artificial-intelligence https://www.brookings.edu/topics/artificial-intelligence/