{"revision":"c10276e953456835cdbb1f3a60d58ae1980be353","document":{"id":23953,"path":"intelligent-experiences/mv-conversational-evaluator.md","title":"Conversational evaluator","bundle":null,"doc_type":null,"release":"australia","body":"\n# Conversational evaluator\n\nConversational evaluator measures the quality of Virtual Agent conversations in AI Control Tower.\n\n## Conversational evaluator overview\n\nConversational Evaluator gives you a measurable view of conversation quality so you can track Virtual Agent performance over time.\n\nA conversation that saves time for a live agent is treated as a successful conversation. Conversational Evaluator combines quality scores with value metrics, such as estimated time savings, so you can connect conversation quality to business impact.\n\nYou access Conversational Evaluator from the **Evaluation** tab in AI Control Tower.\n\n**Note:** Generative AI may produce inaccurate or incomplete information. Always validate AI-generated content.\n\n## Benefits\n\nConversational evaluator provides the following benefits:\n\n-   Track Virtual Agent performance across a defined set of quality metrics.\n-   Define your own quality benchmark by rating conversations.\n-   Estimate the hours and cost that Virtual Agent conversations save.\n\n## How it works\n\nConversational evaluator uses an auto-evaluation framework to score Virtual Agent conversations. It works as follows:\n\n1.  The system identifies qualified conversations by excluding conversation types that aren't suitable for evaluation.\n2.  The system selects a random sample of qualified conversations for auto-evaluation.\n3.  The auto-evaluation framework scores each sampled conversation against a set of quality metrics and rolls the metric scores up into a user satisfaction score.\n4.  The system calculates value metrics, such as estimated time savings and estimated Virtual Agent efficiency, based on the evaluated conversations.\n\n## Conversations excluded from evaluation\n\nThe following conversation types are excluded from auto-evaluation by default:\n\n-   Conversations that use an inaccessible or empty knowledge base\n-   Conversations that transfer immediately to a live agent\n-   Short conversations\n-   Conversations started by custom triggers\n\nA script include controls these exclusion rules. You can extend the script include to exclude other types of conversations, such as conversations from a specific geographic location.\n\n## Evaluation metrics\n\nThe auto-evaluation framework scores conversations on the following metrics, which roll up into the **User Satisfaction** score:\n\n-   Request completion\n-   Intent accuracy\n-   Slot filling\n-   Truthfulness\n-   Context retention\n-   Conciseness\n-   Smoothness — deadlock avoidance\n-   Coherence\n\n## Value metrics\n\nThe **Value** view shows metrics that estimate the business impact of Virtual Agent conversations, including **Total evaluations**, **Average auto-evaluation score**, and **Evaluation score trend**.\n\nConversations are grouped into small, medium, and large sizes based on the number of words in each conversation. For example, a conversation of about 200 words or fewer is a small conversation.\n\nYou can view conversation efficiency and time savings by conversation size or by content type, such as knowledge and catalog.\n\nThe value metrics also include a user acceptance score and an estimated cost saving calculation that converts hours saved into a currency value.\n\n**Related topics**  \n\n\n[Review conversation evaluation scores](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/intelligent-experiences/mv-review-conversation-evaluation-scores.md)\n\n[mv-label-a-conversation-evaluation]\n\n[Calculate Virtual Agent cost savings](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/intelligent-experiences/mv-calculate-virtual-agent-cost-savings.md)\n\n"}}