Evaluating large language models (LLMs) is crucial to ensure their effectiveness, reliability, and alignment with human expectations.
In the era of LLMOps (Large Language Model Operations), robust evaluation metrics are crucial for ensuring the scalability, reliability, and ethical deployment of large language models in real-world applications. These metrics enable continuous improvement, effective monitoring, and proactive maintenance of models, ensuring high-quality outputs and aligning with user expectations. They also support compliance with evolving regulations, promote transparency and accountability, and optimise resource usage, thereby enhancing user trust and satisfaction while preventing biases and ensuring responsible AI usage.
Context Recall
Context Recall is an evaluation metric used to measure how well a generative AI model captures and utilises contextual information when generating text.
Why use Context Recall?
It is particularly important for tasks like dialogue systems, story generation, and text completion, where maintaining coherence and relevance to the preceding context is crucial.
How can I use Context Recall?
Calculating context recall doesn’t need to be a sophisticated process. You can leverage libraries such as Spacy and NLTK, libraries that will already be in every data scientists toolkit. Then after a bit or preprocessing, you can use a semantic similarity model to match contextual units. Recall is computed as the proportion of relevant context units in the reference text that are correctly identified in the generated text.
ROUGE
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is actually a set of metrics used to evaluate the quality of summaries and translations by comparing the generated text to reference texts. It is particularly popular in the domains of automatic text summarisation and machine translation.
ROUGE measures the overlap between the generated text and a set of reference texts, focusing on aspects like recall, precision, and F1-score for various n-gram levels, sequences, and words.
Why use ROUGE?
ROUGE effectively evaluates both recall and precision, ensuring that the generated summaries or translations retain the essential information and are contextually relevant, making it an essential metric for text summarisation tasks.
How can I use ROUGE?
There really is a python library for everything and ROUGE is no exception. The rouge-score library should be all you need for this metric. But even better than that, MLFlow has a rouge metric stack so you are able to incorporate this evaluation into you usual ML workflows.
Perplexity
Perplexity is a measurement used to evaluate the quality of probabilistic models, particularly in natural language processing tasks such as language modeling. It quantifies how well a probability distribution or model predicts a sample.
This metric assesses the model's ability to anticipate the next word in a sequence, reflecting its overall performance in understanding and generating human-like language. A lower perplexity score indicates that the model has a better grasp of the language, making fewer mistakes in its predictions
Why use Perplexity?
Perplexity is crucial for tasks like text generation, translation, and conversational AI. It serves as a reliable metric for comparing different models and guiding improvements in language model training.
How can I use Perplexity?
Perplexity is one of the most used LLM evaluation metric. This means there are several ways we can tackle this problem. Hugging face, for example, offers a perplexity metric in their evaluation library.
BLEU
BLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of text that has been machine-translated from one language to another. It is one of the most popular and widely used metrics for evaluating machine translation systems
BLEU measures the similarity between a machine-generated text and one or more reference texts provided by humans.
Why use BLEU?
BLEU assesses the precision of n-grams in the generated text, capturing both the accuracy and fluency of translations by comparing them to one or more reference translations. This makes it a widely adopted and reliable metric for benchmarking the performance of different translation models, guiding their improvement, and ensuring that the generated translations maintain high quality and relevance.
How can I use BLEU?
The trusty NLTK (Natural Language Toolkit) offers many different flavours of the BLEU evaluation metric so just select which one is right for you use case and off you go.
Toxicity
Toxicity is blanket term for a metric that aims to assess the quality of generated text in terms of its potential to produce harmful or offensive content. While toxicity is primarily associated with human behavior and interactions, it can also manifest in the output of language models, especially if they are trained on datasets containing biased or inappropriate content. Toxicity can be calculated in several ways.
Why use Toxicity?
Assessing toxicity in language models is essential for ensuring user safety and trust, especially in applications where generated text is publicly visible or interacts with users.
How can I use Toxicity?
MLFlow also has built in functionality for assessing toxicity using the model roberta-hate-speech-dynabench-r4, the industry standard, which defines hate as “abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender, or sexual orientation.” This score ranges from 0 to 1, where scores closer to 1 are more toxic.
Honorable mentions
LLMs as a Judge
Large Language Models can be employed to assess generated text quality based on criteria like fluency, coherence, relevance, and grammaticality. LLMs offer scalable, objective, and data-driven evaluation methods, addressing the limitations of human judgment. They enable large-scale assessments across various text generation tasks and provide insights for system improvement. It is important to remember that LLMs may perpetuate bias so aren’t always a good system to regulate each other.
Precision, Accuracy and F1
Just because you are using a foundational model to analyse your data, doesn't mean classic machine learning no longer applies. Perhaps you are using a LLM for Sentiment Analysis, perhaps to score written essays. These are both still classification problems! For these use cases, you can use the tried and true Precision, Accuracy and F1 metrics.
Human Evaluation
Human judgment has been the gold standard for assessing the quality of generated text and for good reason. However, this approach is labor-intensive, subjective, and often impractical for large-scale evaluations. Human evaluation in Generative AI is essential for providing nuanced, context-aware assessments of text quality and ensuring alignment with human needs and ethical standards, while also serving as a benchmark for validating automated evaluation metrics.
Final Remarks
In the realm of LLMOps/FMOps, robust evaluation metrics are paramount for ensuring the scalability, reliability, and ethical deployment of large language models (LLMs) in real-world applications. Metrics like Context Recall, ROUGE, Perplexity, BLEU, and Toxicity serve as vital tools for continuous improvement, effective monitoring, and proactive maintenance of LLMs, ensuring high-quality outputs while aligning with user expectations and ethical standards.
