Skip to main content

Okareo CLI

The Okareo CLI provides a means to run scenario/model workflows in python or typescript without modifying your larger project/repo. You can use the CLI locally or as a coordinator of scenario/model validation in CI.

tip

The SDK requires an API Token. Refer to the Okareo API Key guide for more information.

Overview​

The Okareo CLI assumes a specific config structure that incldues a config.yml for context and a flows folder with the flows you want to run. [Project] .okareo config.yml flows [your_flow_script].[py|ts]

Install​

Download the correct version of the okareo CLI (okareo) for your development environment.

curl -O -L https://github.com/okareo-ai/okareo-cli/releases/download/v0.0.19/okareo_0.0.19_darwin_arm64.tar.gz

Once you've downloaded and unpacked the release, add the location $OKAREO_PATH/bin to your path.

export OKAREO_PATH="[YOUR_LOCAL_PATH_TO_OKAREO]" 
export PATH="$PATH:$OKAREO_PATH/bin"

If okareo -v correctly returns a version, then you are ready to bootstrap a local Okareo environment.

If you want to use Okareo, outside of a broader unit test framework, you can use the okareo init command to bootstrap in Python or Typescript. Learn more

okareo init --language typescript

Config Structure​

To use the Okareo CLI, you will need to establish a .okareo folder with a config.yml file inside.

The ./.okareo/config.yml file can be configured directly or for scripts written in python, javascript, or typescript

name: Project Name 
api-key: ${OKAREO_API_KEY}
project-id: ${OKAREO_PROJECT_ID}
run:
  flows:
    config:
      - name: "Your Evaluation Name"
        model-id: "<MODEL-UUID>"
        scenario-id: "<SCENARIO-UUID>"    
        type: "NL_GENERATION" | "INFORMATION_RETRIEVAL" | "MULTI_CLASS_CLASSIFICATION"
        checks:
          - "<CHECK-NAME-1>"
          - "<CHECK-NAME-2>"
          ...
          - "<CHECK-NAME-N>"
tip

On the roadmap is a facility to split and run flows in parallel. Although a seemingly simple feature, there are interesting consequences - like model warming which causes answers to change if topical questions are asked in sequence. If this is of interest, please reach out to us product@okareo.com

CLI Commands​

-h, --help​

The Okareo CLI includes help. You can get help via --help at the CLI root or on a per command basis.

machine-dev % okareo --help
The Okareo CLI is a tool to help you evaluate your use of AI/ML in your application:
To use the CLI, refer to the docs: https://okareo.com/docs/sdk/cli

Usage:
  okareo [flags]
  okareo [command]

Available Commands:
  clean       Okareo CLI command to clean .okareo directory
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  init        Creates a default .okareo structure in the current directory
  proxy       Start a proxy server using litellm
  run         Okareo CLI command to run workflows.

Flags:
  -h, --help      help for okareo
  -v, --version   The current version of the Okareo CLI (default true)

-c, --clean​

The Okareo CLI bootstraps the Typescript and Javascript environment within ./.okareo.

To avoid accidentally checking in this generated code, we suggest adding a few entries to .gitignore per the Typescript and Javascript install section.

Clean specifically removes the following items:

  • .okareo/package.json
  • .okareo/package-lock.json
  • .okareo/tsconfig.json if using typescript
  • .okareo/node_modules
  • .okareo/dist if using typescript

    This mechanism is not needed if you are using Python.

init​

Creates a default .okareo structure in the current directory

Usage:
  okareo init [flags]

Flags:
  -d, --debug             See additional stdout to debug the init process.
  -f, --force             Forces the exiting configuration to be overwritten.
  -h, --help              help for init
  -l, --language string   The language you want to configure: Python, Javascript, or Typescript. (default "python")

Global Flags:
  -v, --version   The current version of the Okareo CLI (default true)

init -d, --debug​

Provides additional information for the init runtime.

okareo init -d

init -f, --force​

This will replace the existing configuration with a new configuration file. This will NOT remove existing flows.

okareo init --force

init -l, --language​

This initializes the .okareo environment with the specified language : Python, Javascript, or Typescript.

okareo init -l ts
tip

If you are interested in an additional language, let us know: product@okareo.com


proxy​

Starts a proxy server that can handle LLM requests using litellm's proxy functionality

Usage:
  okareo proxy [flags]

Flags:
  -c, --config string   Path to config file (default "./cmd/proxy_config.yaml")
  -d, --debug           Enable debug mode
  -h, --help            help for proxy
  -H, --host string     Host to run the proxy server on (default "0.0.0.0")
  -m, --model string    Model to use (e.g., gpt-3.5-turbo, claude-2)
  -p, --port string     Port to run the proxy server on (default "4000")

Global Flags:
  -v, --version   The current version of the Okareo CLI (default true)

proxy -c, --config string​

To easily support tracing, the CLI provides a proxy. To run the proxy, you will need to provide your OKAREO_API_KEY, and any other standard model keys you wan to use.

The Okareo proxy uses LiteLLM. As a result, you can pass in any standard LiteLLM config format you wish.

You will need to point your model base_url to the proxy endpoint. The proxy runs on port 4000 by default. You can change this with -p 4000 

okareo proxy -c ./config_proxy.yaml

Default config_proxy.yaml file

model_list:
  - model_name: "*"
    litellm_params:
      model: "*"
litellm_settings:
  callbacks: ["otel"]

Example of using OpenAI with the proxy base_url in place.

client = openai.OpenAI(
  base_url="http://localhost:4000"
)


run​

The run command is the primary feature of the Okareo CLI. The run command will call each script that matches the file pattern in .okareo/config.yml.

This command is useful locally during development and can be used directly or through the Okareo GitHub Action to drive flows in CI/CD

If the evaluation config is in the yml or the SDK uses the JSONReporter.log(), run will automatically ouput evaluations results to ./.okareo/reports

machine-dev % okareo run --help
Okareo CLI 'runs' can include multiple flows that perform a variety of tasks from scenario generation to model evaluation.

Usage:
  okareo run [flags]

Flags:
  -c, --config string   The Okareo configuration file for the evaluation run. (default "./.okareo/config.yml")
  -d, --debug           See additional stdout to debug your flows.
  -f, --file string     The Okareo flow script you want to run. (default "ALL")
  -h, --help            help for run
  -r, --reports string  The folder where eval results are made available. Defaults to ./.okareo/reports/ (default "reports")

Global Flags:
  -v, --version   The current version of the Okareo CLI (default true)

run -c, --config​

With this you can create bespoke config files with different matching patterns.

okareo run -c ./.okareo/retrieval.config.yml

A few example configurations that you could create include:

- all.config.yml could macth all flows - intent.config.yml could match flows for evaulating intent - ci.config.yml could match flows you want to run in CI/CD - prod.config.yml could match flows necessary for automated validation prior to production push


run -d, --debug​

If you are concerned that flows are being skipped or would like more information, --debug provides more detailed output.

This facility is primarily used by Okareo during development of the CLI but is perfectly safe for daily use as well.

okareo run -d

run -f, --file​

There are times during development when you will want to run a specific flow to drive and evaluate finetuning. The file flag allows you to specify the flow you want to run.

tip

The --file command takes the name of the flow you want to run without the file type designation. For example projects vs projects.py. This is a conscious decision that abstracts the language from the flow. Because Okareo allows more than one config file to exist, it is possible to have more than one language available in a single .okareo environment.


run -h, --help​

Provides help output on run.

okareo run -h

run -r, --reports string​

The CLI/SDK default to output evaluation reports to ./.okareo/reports. However if an alternative location is preferred, this flag can set the location to use for the output.

okareo run -r analytics