Get Started with Function Calling Evaluation
Agents interact with the external world via function calling, i.e. generating valid API calls that are passed to an execution environment. With Okareo's function calling evaluations, you can measure the accuracy of your agent's function calls.
What do you need?
You will need an environment for running Okareo. Typescript and Python are both available. Please see the SDK sections for more on how to setup each.
Cookbook examples for this guide are available:
- Colab Notebook (Command-R)
- Colab Notebook (Generic)
- Typescript Cookbook (Coming soon!)
Scenarios for Function Calling
In a function calling scenario, you will need to specify the expected function call in each scenario row's result
field. This should resemble the following:
{
"name": str, # the name of the function to be called
"parameter_definitions": {
"parameter_1": {
"value": ...,
"type": str | bool | int | float | dict,
"required": bool,
},
...
}
}