When reports emerge about new or improved LLMs (Large Language Models), as has been the case recently with OpenChat or Mistral, or even when it’s just about the capabilities of ChatGPT, benchmark values are repeatedly presented and brought into consideration.
These benchmarks are intended, on one hand, to represent the capabilities of the LLMs, and on the other, to enable comparability between the systems. However, these benchmarks are often only conceptually understood by a truly knowledgeable audience.
In this article, we will introduce some of the methods and benchmarks for evaluating generative AI models. These tools are crucial for objective performance assessment, as they provide standardized and comparable testing scenarios.
The diversity of the tasks in the test methods ensures that models are effective in various real-world application areas, ranging from language comprehension to image recognition. Their use promotes transparency and trust in AI systems and facilitates interdisciplinary collaboration. The methods are indispensable for developing, evaluating, and improving AI models, enabling continuous measurement of progress and expansion of the boundaries of AI technology.
1. MMLU (Massive Multitask Language Understanding)
Explanation: MMLU is an extensive benchmark aimed at assessing the general language understanding of AI systems. It encompasses a wide range of topics, from history and science to art and literature, in the form of multiple-choice questions.
Scope of Application: MMLU is a comprehensive benchmark for linguistic understanding, covering a multitude of topics.
Current Usage: MMLU is frequently used to evaluate the ability of AI models to understand complex textual content across a broad spectrum of subjects.
Examples:
Question 1: “Who wrote ‘War and Peace’?”
Options: A) Leo Tolstoy B) Fyodor Dostoevsky C) Anton Chekhov D) Ivan Turgenev
Answer: A) Leo Tolstoy
Question 2: “At what temperature does water freeze?”
Options: A) 0 degrees Celsius B) 100 degrees Celsius C) 32 degrees Fahrenheit D) 212 degrees Fahrenheit
Answer: A) 0 degrees Celsius
2. HellaSwag
Explanation: HellaSwag is a benchmark that tests AI models’ ability to generate plausible continuations for incomplete scenarios based on common-sense understanding.
Scope of Application: Tests comprehension of common-sense scenarios and predictive capabilities.
Current Usage: Used to assess how well models can select plausible scenarios from a range of options.
Examples:
Statement 1: “A child builds a sandcastle.”
Continuations: A) “It adds water.” B) “It starts to rain.” C) “The child eats the sand.” D) “The child goes to sleep.”
Answer: A) “It adds water.”
Statement 2: “Someone lights a candle.”
Continuations: A) “The flame goes out.” B) “The candle begins to melt.” C) “The candle turns into a flower.” D) “The candle flies away.”
Answer: B) “The candle begins to melt.”
3. ARC Challenge (AI2 Reasoning Challenge)
Explanation: The ARC Challenge consists of science-based questions that require deep understanding and complex reasoning skills. The focus is on assessing the AI’s analytical thinking ability.
Scope of Application: Focuses on scientific understanding and logical thinking.
Current Usage: Used to evaluate the ability of models to answer complex, science-based questions.
Examples:
Question 1: “What is typically needed to drive a nail into wood?”
Options: A) Screwdriver B) Hammer C) Drill D) Pliers
Answer: B) Hammer
Question 2: “Which planet is known as the ‘Red Planet’?”
Options: A) Venus B) Mars C) Jupiter D) Saturn
Answer: B) Mars
4. WinoGrande
Explanation: WinoGrande is a test for understanding Winograd Schema sentences, based on common-sense logic. It challenges AI models to recognize and resolve ambiguities in sentences.
Scope of Application: A test for understanding Winograd Schema sentences, which rely on common-sense logic.
Current Use: Used to test AI models’ ability to understand subtle linguistic nuances and implicit relationships.
Examples:
Sentence 1: “The city could not build the bridge because it was too large.”
Question: What does “it” refer to?
Answer: The bridge
Sentence 2: “The hunter aimed at the bear because it was dangerous.”
Question: Who does “it” refer to?
Answer: The bear
5. MBPP (MassiveBank of Python Problems)
Explanation: MBPP is a benchmark for assessing the ability of AI models to solve programming tasks in Python. It includes a variety of practical programming problems.
Scope of Application: Evaluates AI models in the field of programming.
Current Use: Used to test how well models can understand and solve programming tasks.
Examples:
Write a function in Python that checks if a number is even.
def is_even(num):
return num % 2 == 0Write a function in Python that takes a list of strings and returns the longest string.
def longest_string(strings):
return max(strings, key=len)6. GSM-8K (Grade School Math 8k)
Explanation: GSM-8K is a benchmark for evaluating the ability of AI models to solve elementary-level mathematical problems.
Scope of Application: Assesses AI models’ ability to solve mathematical problems.
Current Use: Useful for evaluating AI in terms of mathematical understanding and problem-solving skills.
Examples:
Problem Description: “Jerome had 4 friends who visited him on a specific day. The first friend rang the doorbell 20 times before Jerome opened, the second friend rang the doorbell 1/4 times more than Jerome’s first friend. The third friend rang the doorbell 10 times more than the fourth friend. If the fourth friend rang the doorbell 60 times, how many total doorbell rings were there?”
Solution: “The second friend rang the doorbell 1/4 times more than Jerome’s first friend, a total of 1/4*20=5 times. So, Jerome’s second friend rang the doorbell 20+5=25 times. The first two friends rang the doorbell a total of 25+20=45 times before Jerome could open. Jerome’s third friend rang the doorbell 60+10=70 times before he could open. In total, Jerome’s third and fourth friends rang the doorbell 70+60=130 times before he could open. The total number of doorbell rings made by Jerome’s friends before he could open is 130+45=175 times.”
Problem Description: “Cody eats three times as many cookies as Amir. If Amir eats 5 cookies, how many cookies do they eat together?”
Solution: “Cody eats 5*3 = 15 cookies. Cody and Amir eat a total of 15+5 = 20 cookies together.”
7. MT Bench (Machine Translation Benchmark)
Explanation: MT Bench evaluates the ability of AI models to perform machine translation between different languages.
Scope of Application: Measures the ability of AI models in machine translation.
Current Use: Important for assessing the quality of translation models.
Examples:
English: “Life is what happens when you’re busy making other plans.”
German Translation: “Das Leben ist das, was passiert, während du beschäftigt bist, andere Pläne zu machen.”
English: “To be or not to be, that is the question.”
German Translation: “Sein oder Nichtsein, das ist hier die Frage.”
8. GLUE and SuperGLUE
Explanation: GLUE and SuperGLUE are collections of benchmark methods aimed at testing the understanding of natural language and complex text processing capabilities.
Scope of Application: A series of tests for assessing general language understanding.
Current Use: Serve as standard benchmarks in AI research to measure the language processing capabilities of models.
Examples of benchmarks and methods included in GLUE and SuperGLUE:
CoLA (Corpus of Linguistic Acceptability) from GLUE
Task: Assess whether a given sentence is grammatical. Here, “grammatical” refers to the correctness of a sentence according to the rules of the respective language, in this case, German. CoLA is a dataset consisting of sentences classified as either grammatically correct or incorrect.
Example 1:
Sentence: “The cats are sleeping”
Evaluation: Grammatical
Example 2:
Sentence: “The cats sleeps.”
Evaluation: Ungrammatical
SST-2 (Stanford Sentiment Treebank) from GLUE
Task: Determine the sentiment (positive/negative) of a sentence.
Example:
Sentence: “This film was a visual masterpiece.”
Evaluation: Positive
MNLI (Multi-Genre Natural Language Inference) from GLUE
Task: Decide whether a hypothesis follows from a premise, contradicts it, or neither.
Example:
Premise: “A dog sleeps on the sofa.”
Hypothesis: “The animal is tired.”
Evaluation: Follows
QNLI (Question Natural Language Inference) from GLUE
Task: Determine whether the answer to a question is contained in the given text.
Example:
Question: “Where do koalas live?”
Text: “Koalas are a symbol of Australia and live in eucalyptus forests.”
Evaluation: Yes
RTE (Recognizing Textual Entailment) from GLUE
Task: Decide whether one text passage implies another.
Example:
Text: “NASA has launched a new satellite.”
Hypothesis: “A satellite was sent into space.”
Evaluation: True
BoolQ (Boolean Questions) from SuperGLUE
Task: Answering yes/no questions based on a paragraph.
Example:
Paragraph: “Pandas mainly live in China.”
Question: “Do pandas live in Africa?”
Answer: No
CB (CommitmentBank) from SuperGLUE
Task: Determine whether a hypothesis follows from a given statement, contradicts it, or if it’s undecidable.
Example:
Statement: “Tom said he would come to the party.”
Hypothesis: “Tom will be at the party.”
Evaluation: Follows
WiC (Words in Context) from SuperGLUE
Task: Determine whether a word has the same meaning in two sentences.
Example:
Sentence 1: “He pulled his hat down over his face.”
Sentence 2: “She won a prize in the lottery.”
Word: “pulled”
Evaluation: Different
9. SQuAD (Stanford Question Answering Dataset)
Explanation: SQuAD is a benchmark for machine understanding of text, where questions must be answered based on a given text segment.
Examples:
Text: “Albert Einstein was born in 1879 in Ulm.” Question: “Where was Albert Einstein born?” Answer: “In Ulm”
Text: “The Earth revolves around the Sun.” Question: “Around what does the Earth revolve?” Answer: “The Sun”
10. COCO (Common Objects in Context)
Explanation: COCO focuses on image recognition and description. It involves tasks for object recognition, segmentation, and generating image descriptions.
Examples:
Image: A picture of a dog catching a frisbee.
Description: “A dog jumps to catch a frisbee.”
Image: A shot of a busy street intersection.
Description: “People and vehicles at a busy intersection.”
Significance of each AI testing method:
MMLU and HellaSwag: Highly rated for testing general language understanding and common-sense logic. These benchmarks are particularly valuable for assessing AI models that must be capable of understanding complex textual content and generating plausible scenarios in various contexts.
ARC Challenge and WinoGrande: Essential for examining analytical thinking and problem-solving abilities. These tests are crucial for models used in areas such as education, research, and situations requiring a deep understanding of language nuances.
MBPP and GSM-8K: Very relevant for testing specific skills like programming and mathematical understanding. These benchmarks are indispensable for assessing AI models employed in technical and educational applications where precise and correct solutions are of high importance.
MT Bench: Important for assessing capabilities in machine translation. This benchmark is central for evaluating AI models that must be able to accurately and coherently translate texts into different languages, a key component in the globalized world.
GLUE/SuperGLUE and SQuAD: Central for assessing comprehensive language understanding and the ability to provide precise answers to specific questions. These benchmarks are critical for evaluating AI models used in a variety of applications, from automated customer service systems to assistants that handle complex, language-based tasks.
COCO: Highly important for testing skills in image recognition and description. This benchmark is crucial for AI models in the field of computer vision, a vital area in modern AI, ranging from autonomous vehicles to medical image analysis.
General Note: This text was generated with the assistance of ChatGPT, and technical terms were explained by a GPT.