Evaluate Feedback

To evaluate the feedback, a basic technical understanding of large language models, the Genow application, and existing and future features and limitations of Genow is necessary. The feedback received can be evaluated particularly well if it is of high quality.

We would recommend encouraging users to provide continuous, comprehensive feedback via the feedback function in order to further develop the Agent in terms of quality. This feedback can then be viewed via the admin panel. For a structured analysis of the feedback, we recommend processing it via CSV. To do this, you can use the ‘Download Feedback’ function in the Admin Panel.

Show introduction to the feedback dashboard in the admin panel

Navigate to the admin panel and select your Agent. You will find yourself in the Agent setting. On the right, you will find the “Feedback Dashboard” in the ‘Evaluate Agent section”.On the top of the page, you will find the total amount of feedbacks and their distribution into positive and negative ratings over different time horizons, as well as the star rating in each case.On the button of the page, you can unfold the list of feedbacks received for the Agent. By clicking “View”, you will be able to see all the relevant information provided with the feedback (filled-out fields of the feedback form as well as the user questions and answers from Genow).

Formation of Clusters

Clusters are useful for evaluating feedback. Similar feedback is grouped together. The following table provides an overview of common clusters and how to address them. First, it’s a good idea to cluster negative and positive feedback. Positive user feedback describes how well users have adopted the technology and how well they have been trained. Negative feedback provides insight into potential for improvement. Please note that the following table refers to the evaluation of negative feedback:

Cluster Name	Description	Responsibility	Takeaway
Correct	Genow’s response was actually correct based on the connected data and the technical capabilities of language models, but feedback was provided.	Agent	User training on expectation management or on the capabilities and limitations of language models may be necessary.
Information missing	The information is not available to the language model, but should be added.	Agent	Update information
Missing Feature / will be fixed by updates	The issue will be corrected by Genow (Roadmap!)	Agent / Genow	Update information	Agent Admin contacts Genow to submit a feature request. If there is a high degree of urgency, a project contract can be concluded for implementation. Sometimes the feature is already on Genow’s roadmap or it is a known problem that will be resolved with a short-term release.
Information inaccurate	Information in existing documents inaccurate / insufficiently formulated, which has a negative impact on the quality of the response.	Agent	Update document, make sure the guidelines and recommendations for your data was matched (see Data Maintanace article).
Imprecise question / interaction in need of improvement	User input is sometimes crucial when complex information is to be extracted from documents.	Agent	User training necessary
Information not output correctly	Information not extracted correctly, although it is available.	Agent / Genow	Depending on the individual case. The files and their names should always be checked. In complex cases, pipeline optimisation is necessary. This can be carried out by Genow.
Available feature or optimisation of an existing feature would provide improvement	Issue or process during use could be fixed or improved by an available feature.	Agent / Genow	Depending on the individual case.
Bug	Error message, demonstrably incorrect behaviour by Genow; ideally traceable.	Genow	Error message, time of the error message and description as well as screenshot to support@genow.ai for error correction.
In individual cases there are further clusters …

Review Feedback

All feedback should be reviewed. For this purpose, either a spreadsheet (e.g., an Excel file) or the existing feedback dashboard in the Genow admin panel can be used. For the evaluation of feedback, the query, Genow’s response, and the user comment should be recorded. Subsequently, the question should be assigned to one, or at most two, clusters. Additionally, a comment can be added to justify the cluster assignment. A possible structure results from the following example table:

Question	Answer from Genow	Genow thread history	Target answer / comment by test user	Positive / negative	Comment Cluster	Cluster 1	Cluster 2
The user request sent to Genow that was evaluated	Genow’s answer to the graded question	History of previous communication before the graded response	Comment by the evaluator; contains information about what answer would have been expected or whether information is missing		Comment from test evaluation

Create a summarized Evaluation

The clusters created and the frequency with which they occur should be placed in an overall context with the total number of feedbacks, positive and negative feedbacks and the total number of requests. The summarized evaluation can be shown in the following table:

Evaluation	Takeaway	Optimisation steps
Total number of requests (see Analytics Dashboard in the Admin Panel)	I.e. used more or less than expected	I.e.: motivate user to increase usage
Positive Feedbacks	I.e. comparison to total number of requests	I.e: Go live or next project phase possible?
Negative Feedback	I.e. comparison to total number of requests	I.e.: Optimisation necessary?
Responsibility Agent	Sum of clusters that can be solved internally by the Agent (e.g. updating information). Indicator of how much work still needs to be done internally.
Responsibility Genow	Sum of Clusters that will be solved by Genow (i.e. Bugs)
Optimisation necessery	Clusters that can be improved through optimizations in the data pipeline or through updates and features.	In consultation with Genow - depending on the individual case. E.g. glossaries, fallback strategy, individual base prompt, metadata, …
Cluster 1
Cluster 2…

The table can be supplemented with any other relevant clusters. For the sake of clarity, a graphical representation is recommended at the end.

Getting Started

Manage Your Knowledge

Use Case Settings

Analyse Your Use Case

Individualise Your Use Case

Formation of Clusters

Review Feedback

Create a summarized Evaluation

​Formation of Clusters

​Review Feedback

​Create a summarized Evaluation

Formation of Clusters

Review Feedback

Create a summarized Evaluation