Feedback
With every trace
called, we keep a snapshot of user interaction on client side so user can rewind and get more context when providing feedback.
Here is how you can collect user feedback on client side with Agiflow sdk.
Inline feedback
You can use actionId
generated with track
or trackAsync
to identify user feedback such as thumb up / thumb down directly on client with below snippet. Additionally, you can associate a backend trace with your unique identifier (workflow Id, message Id, etc...) and use that to provide feedback:
agiflowClient.reportScore(
actionId, // ActionId is generated from track
0.6 // Normalised score
);
We use normalised score to make it easier for fine-tune. Here is a reference for conversion:
Thumbs up
= 1;thumbs down
= 0;- Rank
0 -> 10
is converted to[0, 1]
.
Feedback widget
Feedback widget is a powerful tool to provide feedback on the AI output and increase the transparency of your AI application. There are two modes you can display feedback widget at this stage.
Simple chat feedback
const agiflowClient = new Agiflow();
agiflowClient.init(
process.env.CLIENT_KEY, // Agiflow client key (please don't use API key here)
feedbackPreviews: ['screen'],
);
agiflowClient.openWidget();
When user agree to provide feedback, this will replay
the screen state when the AI interaction happens.
Complex workflow feedback
const agiflowClient = new Agiflow();
agiflowClient.init(
process.env.CLIENT_KEY, // Agiflow client key (please don't use API key here)
feedbackPreviews: ['workflow'],
);
agiflowClient.openWidget();
Unlike simple chat feedback when there is only one AI output corresponding to single user input, a complex workflow involes multiple AI agents. Simply provide correction on the output is not helpful as it can't be used for evaluating AI prompts or fine-tuning.
This mode displays a workflow chart where your end-user can click on particular component of AI workflow to review output and provide feedback. This works best for internal tools or B2B businesses.
Subscribe to issues detected by guardrails
If you wish to only open feedback widget when there are issues with LLM, you can subscribe to new issues and decide to call openWidget
at any time liked below:
agiflowClient.subscribeToNewIssue((issue) => {
...
});