{"revision":"c10276e953456835cdbb1f3a60d58ae1980be353","document":{"id":22729,"path":"intelligent-experiences/ai-control-tower/ai-eval-flow.md","title":"Evaluation flow","bundle":null,"doc_type":null,"release":"australia","body":"\n# Evaluation flow\n\nThe workflow for evaluation execution, which performs evaluations when conversations are completed.\n\nConversations are evaluated using the following logic:\n\n1.  Conversation capture:\n\n    All end-user interactions with the virtual agent are logged in the Conversation table \\[sys\\_cs\\_conversation\\]. When a user ends the conversation, the record's state is updated to Complete.\n\n2.  Automated flow evaluation trigger:\n\n    Flow name: Execute Evaluation.\n\n    Trigger condition:\n\n    -   Table: Conversation table \\[sys\\_cs\\_conversation\\]\n    -   State: Complete\n    -   Device type: Web Client, Slack, Teams, Bot to Bot, Messenger\n\n**Sequence of execution**:\n\nAction 0: Check evaluations count for today\n\n-   Perform a query on evaluation table and to get record count.\n-   If record count is less than Max Number of evaluations per day, continue to Action 1, else end flow.\n\nAction 1: evalExecuteCondition\n\n-   Invokes the evalExecuteCondition.executeEvaluation script Include with conversation reference.\n-   Generates a random number \\(1–100\\). Proceeds only if ≤10 \\(10% random sampling\\).\n-   Outcome: Returns true or false for further processing.\n\nAction 2: Conditional Branch\n\n-   If true: Proceed to the next action.\n-   If false: Evaluation stops.\n\nAction 3: Lookup Interaction Table:\n\nMatches the conversation's channel metadata with the interaction table to fetch related records.\n\nAction 4: Application Scope Filter:\n\nIf the interaction's application scope doesn’t include `hr`, continue.\n\nAction 5: buildTranscript:\n\nDetailed Transcript Construction:\n\n-   Tags: \\[User\\]: For user messages, \\[Virtual Agent\\]: For virtual agent messages.\n-   For any referenced Knowledge article:\n    -   Pulls the complete article body to replace genius result, tagged with `[Virtual Agent]: Help articles for user query:` and delimited by Article\\_Start/Article\\_End.\n    -   If the Knowledge article is in HR scope/inaccessible, skip evaluation.\n    -   If the Knowledge article content is &gt;10,000 words: Truncate at 10,000.\n    -   Attached files \\(PDF/Word/Txt\\): Use genius result instead.\n-   For referenced Catalog Items:\n\n    Extracts name, short description, description, annotated as `[Virtual Agent]: Please choose one of the below options:` with citation number.\n\n-   If the first message is to the live agent, or the live agent is invoked within the first 120 words: skip evaluation.\n\nOutputs:\n\n-   ExecuteEvaluation \\(true/false\\)\n-   Chat Transcript\n-   Knowledge articles or catalog items referred\n-   Sys\\_id of first live agent invocation \\(if any\\)\n-   List of skills to invoke \\(all evaluation skills for Evaluation dashboard\\)\n-   Additional evaluation logs\n\nAction 6: Conditional Branch:\n\nIf ExecuteEvaluation is true: Continue to Action 7.\n\nAction 7: Chat Classifier Eval\n\n-   Builds the initial transcript from **sys\\_cs\\_message**.\n-   Uses Chat topic classifier to determine:\n    -   Should the conversation be evaluated? \\(ExecuteEvaluation: true/false\\)\n    -   Topic Name\n    -   Category \\(IT/HR\\)\n-   If ExecuteEvaluation is true: Proceed to Action 6.\n\nAction 8: Create or Update Evaluation Record:\n\nCreate a record on Evaluation \\[sn\\_na\\_conv\\_eval\\_evaluation\\] table with:\n\n-   Document Conversation: Conversation reference\n-   State: Processing\n-   Topic, Category, Knowledge article or catalog references, first live agent sys\\_id, type, user who initiated, message log\n\nAction 9: For each skill:\n\nRepeats for each skill flagged in Action 6.\n\nAction 10: invokeApiDefinition\n\n-   Inputs: Skill name, conversation, transcript, evaluation id\n-   Calls Now Assist Skill API asynchronously.\n-   Post processing available in sys\\_generative\\_ai\\_response\\_validator, performs the following parsing:\n    -   Score\n    -   Reason for Score\n    -   Examples for the reasoning\n-   Parsed data is created on the Evaluation Metrics \\[sn\\_na\\_conv\\_eval\\_evaluation\\_metrics\\] table \\(Score, Reasons, Examples, and the entire reasoning for scoring \\[Scratchpad\\]\\).\n\nAction 11: Waits 7 seconds before continuing to the next skill.\n\nSpecial behavior and edge case handling:\n\n-   Sampling: Only 10% of conversations \\(randomly chosen\\) are evaluated.\n-   Channel Filter: Only Web, Slack, Teams, Bot to Bot, Messenger.\n-   Application Scope: Excludes records with `_hr_` in the scope.\n-   Knowledge article controls: No evaluation for HR or inaccessible. Knowledge articles, limits on Knowledge article size, and file handling.\n-   First live agent invocation: Excludes conversations routed to the live agent at the start or within 120 words.\n-   The Request Completion skill is added as part of a business rule where the score is tagged as the lowest between Slot filling and Intent.\n-   The reason on the record is added as follows:\n\n    ```\n    if (Slot filling score > Intent score) {\n    Intent reason is used\n    } else if (Slot filling score < Intent score) {\n    Slot filling reason is used\n    } else {\n    Both are used\n    }\n    \n    ```\n\n\n"}}