How do you evaluate the quality of a natural language generation system?

1 NLG tasks and challenges

NLG is the process of producing natural language text or speech from non-linguistic data, such as structured data, images, or knowledge graphs. This technology can be used for a variety of tasks, such as generating captions for images or videos, writing headlines or summaries for articles or reports, creating product descriptions or reviews, generating responses or dialogues for chatbots or voice assistants, and writing stories or narratives from prompts or keywords. However, NLG is not a simple task and presents many challenges, such as capturing the meaning, context, and relevance of the input data, producing fluent, coherent, and grammatical output, avoiding repetition, inconsistency, or contradiction, adapting the style, tone, and register to the audience and purpose, and generating novel, creative, and engaging output.

2 NLG evaluation metrics

To measure the quality of an NLG system, we need to define criteria and metrics that can capture the various aspects of the output. Common metrics include automatic metrics, which are quantitative measures that compare the output of the system with a reference text, usually provided by a human. These can be based on word overlap, semantic similarity, or syntactic structure. Automatic metrics are fast, scalable, and objective, but they may not reflect the human perception of quality. Human metrics, on the other hand, are qualitative measures that rely on human judgments or feedback. These can be based on ratings, preferences, or comments. Human metrics are more reliable, valid, and comprehensive, but they are also costly, time-consuming, and subjective, and they may vary depending on the task, the domain, and the evaluator.

3 NLG evaluation methods

To apply the metrics, we need to design some methods or protocols that can collect and analyze the data. Intrinsic evaluation is the evaluation of the output in isolation, without considering the context, the goal, or the impact of the system. It focuses on the internal properties of the output, such as fluency, coherence, grammar, or accuracy. This type of evaluation is useful for debugging, comparing, or improving the system, but it may not capture the real-world performance or usefulness of the system. On the other hand, extrinsic evaluation is the evaluation of the output in relation to the context, the goal, or the impact of the system. It focuses on the external effects of the output, such as usability, functionality, satisfaction, or engagement. This type of evaluation is useful for testing, validating, or deploying the system, but it may require more resources, expertise, or scenarios.

4 NLG evaluation best practices

To ensure a reliable and meaningful evaluation of an NLG system, it is important to define the task and objective clearly and explicitly. Additionally, the metrics and methods used should be relevant and appropriate for the task and objective. To capture the different dimensions of quality, multiple metrics and methods should be used. Furthermore, a large and diverse sample of input data and output texts, as well as a representative and unbiased sample of human evaluators, should be employed. Finally, the results and limitations should be reported transparently and honestly.

5 NLG evaluation examples

To illustrate some of the concepts and techniques we discussed, let's look at some examples of NLG evaluation for different tasks and domains. For instance, image captioning can be evaluated using automatic metrics such as BLEU, ROUGE, or CIDEr to measure the word overlap or the syntactic structure of the output with a reference caption. Additionally, human metrics such as ratings or preferences can be used to measure the fluency, relevance, or informativeness of the output. Similarly, text summarization can be evaluated using automatic metrics such as ROUGE, BERTScore, or MoverScore to measure the semantic similarity or the content coverage of the output with a reference summary. Human metrics such as ratings or preferences can also be used to measure the readability, coherence, or salience of the output. Lastly, chatbot dialogue can be evaluated using automatic metrics such as BLEU, METEOR, or ADEM to measure the word overlap or the dialogue quality of the output with a reference response. Human metrics such as ratings or preferences can be used to measure the appropriateness, naturalness, or engagement of the output.