Skip to main content

Jupyter Notebook

There are as many flavors of models as there are problems that need to be solved. This is one of the challenges in both authoring models and evaluating their stability for real-world use. Okareo provides open-ended tools for evaluation. We also provide more focused tools for some of the most common categories of models - Classification, Retrieval, and Generation. Used as a trio, these are also the building blocks for evaluating a RAG.

Getting Started with Jupyter​

If you want to get started quickly, we would suggest using Jupyter. However any python environment will do. Here are a few quick resources for getting started with a Jupyter environment

  1. The Jupyter project. You can download and learn all about Jupyter from the community itself.
  2. IDEs such as VSCode have plugins for Jupyter. Check with your IDE to see if it supports *.ipynb files.
  3. Google Colab is ready to go with minimal work.

Your first notebook​

Now that you have a token and you have setup Jupyter, you are ready to run your first evaluation.
The following Guides are a good place to start if you are learning about Okareo or Model evaluation for the first time.

  • Classification - learn about the error matrix and evaluating classifiers
  • Retrieval - Retrieval is critical to the RAG architecture. Learn how to test your embeddings and VectorDB.
  • Generation - Generating summaries from documents and facts is one of the most powerful and difficult to learn facets of AI. Experiment with an evaluation notebook for a simple generator to learn more.
  • RAG - RAG(Retrieval Generated Augmentation) requires all three of the above processes. When testing the full RAG, linking, tagging, and metrics over time are far more important.