{"id":198,"date":"2024-11-05T19:09:54","date_gmt":"2024-11-05T19:09:54","guid":{"rendered":"https:\/\/pacific.ai\/staging\/3667\/?p=198"},"modified":"2026-02-19T11:31:15","modified_gmt":"2026-02-19T11:31:15","slug":"building-responsible-language-models-with-the-langtest-library","status":"publish","type":"post","link":"https:\/\/pacific.ai\/staging\/3667\/building-responsible-language-models-with-the-langtest-library\/","title":{"rendered":"Building Responsible Language Models with the LangTest Library"},"content":{"rendered":"
<\/div>

Automatically generate test<\/a> cases, run tests, and augment training datasets with the open-source, easy-to-use, cross-library LangTest package<\/em><\/p>\n

\"\"<\/figure>\n

If your goal is to deliver NLP systems for production systems, you are responsible for delivering models that are robust, safe, fair, unbiased, and private – in addition to being highly accurate. This requires having the tools<\/a> & processes to test for these requiremenst in practice – as part of your day-to-day work, your team\u2019s work, and on every new version of a model.<\/p>\n

The LangTest library<\/strong><\/a> is designed to help you do that, by providing comprehensive testing capabilities for both models and data. It allows you to easily generate, run, and customize tests to ensure your NLP systems are production-ready. With support for popular NLP libraries like transformers, Spark NLP, OpenAI, and spacy, LangTest is an extensible and flexible solution for any NLP project.<\/p>\n

In this article, we\u2019ll dive into three main tasks that the LangTest library helps you automate: Generating tests, running tests, and augmenting data.<\/p>\n

Automatically Generate Tests<\/h2>\n

Unlike the testing libraries of the past, LangTest allows for the automatic generation of tests – to an extent. Each TestFactory<\/code> can specify multiple test types and implement a test case generator and runner for each one.<\/p>\n

The generated tests are presented as a table with ‘test case’ and ‘expected result’ columns that correspond to the specific test. These columns are designed to be easily understood by business analysts who can manually review, modify, add, or remove test cases as needed. For instance, consider the test cases generated by the RobustnessTestFactory<\/code> for an NER task on the phrase \u201cI live in Berlin.\u201d:<\/p>\n\n\n\n\n\n\n\n\n
Test type<\/strong><\/th>\nTest case<\/strong><\/th>\nExpected result<\/strong><\/th>\n<\/tr>\n<\/thead>\n
remove_punctuation<\/td>\nI live in Berlin<\/td>\nBerlin: Location<\/td>\n<\/tr>\n
lowercase<\/td>\ni live in berlin.<\/td>\nberlin: Location<\/td>\n<\/tr>\n
add_typos<\/td>\nI liive in Berlin.<\/td>\nBerlin: Location<\/td>\n<\/tr>\n
add_context<\/td>\nI live in Berlin. #citylife<\/td>\nBerlin: Location<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Starting from the text “John Smith is responsible”, the BiasTestFactory<\/code> has generated test cases for a text classification task using US ethnicity-based name replacement.<\/p>\n\n\n\n\n\n\n\n\n
Test type<\/strong><\/th>\nTest case<\/strong><\/th>\nExpected result<\/strong><\/th>\n<\/tr>\n<\/thead>\n
replace_to_asian_name<\/td>\nWang Li is responsible<\/td>\npositive_sentiment<\/td>\n<\/tr>\n
replace_to_black_name<\/td>\nDarnell Johnson is responsible<\/td>\nnegative_sentiment<\/td>\n<\/tr>\n
replace_to_native_american_name<\/td>\nDakota Begay is responsible<\/td>\nneutral_sentiment<\/td>\n<\/tr>\n
replace_to_hispanic_name<\/td>\nJuan Moreno is responsible<\/td>\nnegative_sentiment<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Generated by the FairnessTestFactory<\/code> and RepresentationTestFactory<\/code> classes, here are test cases that can ensure representation and fairness in the model’s evaluation. For instance, representation testing might require a test dataset with a minimum of 30 samples of male, female, and unspecified genders each. Meanwhile, fairness testing can set a minimum F1 score of 0.85 for the tested model when evaluated on data subsets with individuals from each of these gender categories.<\/p>\n\n\n\n\n\n\n\n\n\n\n
Test type<\/strong><\/th>\nTest case<\/strong><\/th>\nExpected result<\/strong><\/th>\n<\/tr>\n<\/thead>\n
min_gender_representation<\/td>\nMale<\/td>\n30<\/td>\n<\/tr>\n
min_gender_representation<\/td>\nFemale<\/td>\n30<\/td>\n<\/tr>\n
min_gender_representation<\/td>\nUnknown<\/td>\n30<\/td>\n<\/tr>\n
min_gender_f1_score<\/td>\nMale<\/td>\n0.85<\/td>\n<\/tr>\n
min_gender_f1_score<\/td>\nFemale<\/td>\n0.85<\/td>\n<\/tr>\n
min_gender_f1_score<\/td>\nUnknown<\/td>\n0.85<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The following are important points to take note of regarding test cases:<\/p>\n