Documentation Index
Fetch the complete documentation index at: https://docs.genow.ai/llms.txt
Use this file to discover all available pages before exploring further.
To evaluate the feedback, a basic technical understanding of large language models, the Genow application, and existing and future features and limitations of Genow is necessary. The feedback received can be evaluated particularly well if it is of high quality.
Formation of Clusters
Clusters are useful for evaluating feedback. Similar feedback is grouped together. The following table provides an overview of common clusters and how to address them. First, it’s a good idea to cluster negative and positive feedback. Positive user feedback describes how well users have adopted the technology and how well they have been trained. Negative feedback provides insight into potential for improvement. Please note that the following table refers to the evaluation of negative feedback:| Cluster Name | Description | Responsibility | Takeaway | |
|---|---|---|---|---|
| Correct | Genow’s response was actually correct based on the connected data and the technical capabilities of language models, but feedback was provided. | Use Case | User training on expectation management or on the capabilities and limitations of language models may be necessary. | |
| Information missing | The information is not available to the language model, but should be added. | Use Case | Update information | |
| Missing Feature / will be fixed by updates | The issue will be corrected by Genow (Roadmap!) | Use Case / Genow | Update information | Use Case contacts Genow to submit a feature request. If there is a high degree of urgency, a project contract can be concluded for implementation. Sometimes the feature is already on Genow’s roadmap or it is a known problem that will be resolved with a short-term release. |
| Information inaccurate | Information in existing documents inaccurate / insufficiently formulated, which has a negative impact on the quality of the response. | Use Case | Update document, make sure the guidelines and recommendations for your data was matched (see Data Maintanace article). | |
| Imprecise question / interaction in need of improvement | User input is sometimes crucial when complex information is to be extracted from documents. | Use Case | User training necessary | |
| Information not output correctly | Information not extracted correctly, although it is available. | Use Case / Genow | Depending on the individual case. The files and their names should always be checked. In complex cases, pipeline optimisation is necessary. This can be carried out by Genow. | |
| Available feature or optimisation of an existing feature would provide improvement | Issue or process during use could be fixed or improved by an available feature. | Use Case / Genow | Depending on the individual case. | |
| Bug | Error message, demonstrably incorrect behaviour by Genow; ideally traceable. | Genow | Error message, time of the error message and description as well as screenshot to support@genow.ai for error correction. | |
| In individual cases there are further clusters … |
Review Feedback
All feedback should be reviewed. For this purpose, either a spreadsheet (e.g., an Excel file) or the existing feedback dashboard in the Genow admin panel can be used. For the evaluation of feedback, the query, Genow’s response, and the user comment should be recorded. Subsequently, the question should be assigned to one, or at most two, clusters. Additionally, a comment can be added to justify the cluster assignment. A possible structure results from the following example table:| Question | Answer from Genow | Genow thread history | Target answer / comment by test user | Positive / negative | Comment Cluster | Cluster 1 | Cluster 2 |
|---|---|---|---|---|---|---|---|
| The user request sent to Genow that was evaluated | Genow’s answer to the graded question | History of previous communication before the graded response | Comment by the evaluator; contains information about what answer would have been expected or whether information is missing | Comment from test evaluation |
Create a summarized Evaluation
The clusters created and the frequency with which they occur should be placed in an overall context with the total number of feedbacks, positive and negative feedbacks and the total number of requests. The summarized evaluation can be shown in the following table:| Evaluation | Amount | Takeaway | Optimisation steps |
|---|---|---|---|
| Total number of requests (see Analytics Dashboard in the Admin Panel) | I.e. used more or less than expected | I.e.: motivate user to increase usage | |
| Positive Feedbacks | I.e. comparison to total number of requests | I.e: Go live or next project phase possible? | |
| Negative Feedback | I.e. comparison to total number of requests | I.e.: Optimisation necessary? | |
| Responsibility Use Case | Sum of clusters that can be solved internally by the use case (e.g. updating information). Indicator of how much work still needs to be done internally. | ||
| Responsibility Genow | Sum of Clusters that will be solved by Genow (i.e. Bugs) | ||
| Optimisation necessery | Clusters that can be improved through optimizations in the data pipeline or through updates and features. | In consultation with Genow - depending on the individual case. E.g. glossaries, fallback strategy, individual base prompt, metadata, … | |
| Cluster 1 | |||
| Cluster 2… |