Prompts
Prompt Management and Versioning
Okareo generally views prompts and models from the developer's perspective. However, not every prompt is part of a SaaS application or Agent network. It is often convenient to have a place where you can store and test prompts directly.
We are also embracing the changing ownership of the prompt. Many subject matter experts and product managers are being asked to write and improve prompts due to their facility with language and understanding of the desired business outcome. There are even modern tools (like Okareo) that allow users to write prompts as a native aspect of the application. Those prompts often do not have a home in a traditional code repository.
Okareo prompt managment solves these problems with a place to store, version, and run batch experiments with prompts.
Prompt Management API
The developer APIs for prompt versioning are part of model registration in python or typescript.
Prompt Versioning and Management
Okareo prompts are tied to models. You can experiment with new prompt-model pairings in the evaluation studio. Editing and managing prompts is the same process as registering a model.
Evaluation Studio
You can use the evaluatio studio to debug traces that had errors in them or experiment with new prompts.
Scenarios give you the flexibility to experiment with a few or even hundreds+ of prompt-and-input pairs. Tying variations of prompts and inputs gives you a clear understanding of your prompt's characteristics and likelihood to meet your expectations.
Powerful to the Okareo Evaluation Studio are the use of substitutions. You can reference data from your scenarios within the prompt you are experimenting with.
For example, if you have a "thank you" letter writing prompt that includes address, you might reference the address using moustache syntax (curly brances) in your prompt.
System Prompt:Write a short letter thanking the provided person for their charitable donation.
If their address is incomplete, do not include it. Otherwise, include the address
as part of the signature.
First Name: {scenario_input.first_name}
Last Name: {scenario_input.last_name}
Donation: {scenario_input.donation.amount}
Address: {scenario_input.address}
Using the Okareo Scenario Copilot, you could then create numerous examples of potential donors with properties like first_name, last_name, and address. By combining the versioned prompt, the scenario of example people, and the evaluation studio, you can see how your prompt will behave in a wide range of situations.
When you are done experimenting, you may discover that your prompt has formatting issues or even makes mistakes when certain information is unusually long, missing, or unique. These would be embarassing letters if you did not have a tool for batch evaluation. That's ok. You can make a few tweaks to the prompt or use our suggestions and save a new version. Crisis averted.
