{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Scenarios in Okareo: An Introduction\n", "\n", "\n", " \"Open\n", "\n", "\n", "## 🎯 Goals\n", "\n", "After using this notebook, you will be able to:\n", "- Upload your test cases to Okareo as a Seed scenario by:\n", " 1. Uploading a file\n", " 2. Defining input data statically\n", "- Generate synthetic new test cases using Scenario generators\n", "- Chain generators together to make more complex test cases" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, import the Okareo library and use your [API key](https://okareo.com/docs/using-okareo/api_key) to authenticate." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "from okareo import Okareo\n", "\n", "OKAREO_API_KEY = os.environ[\"OKAREO_API_KEY\"]\n", "okareo = Okareo(OKAREO_API_KEY)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Uploading a Seed Scenario\n", "\n", "Here we use an existing `.jsonl` file to create a seed scenario with the `upload_scenario_set` method. The data here includes short articles about a fictitious company called \"WebBizz.\"" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WebBizz Articles: https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/3a904bd0-3dfa-42e5-a10a-d42135b9077e\n" ] } ], "source": [ "import os\n", "import requests\n", "\n", "file_path_articles = \"webbizz_10_articles.jsonl\"\n", "scenario_name_articles = \"WebBizz Articles\"\n", "\n", "def load_or_download_file(file_path, scenario_name):\n", " try:\n", " # load the file to okareo\n", " source_scenario = okareo.upload_scenario_set(file_path=file_path, scenario_name=scenario_name)\n", " except:\n", " print(f\"- Loading file {file_path} to Okareo failed. Temporarily download the file from GitHub...\") \n", "\n", " # if the file doesn't exist, download it\n", " file_url = f\"https://raw.githubusercontent.com/okareo-ai/okareo-python-sdk/main/examples/{file_path}\"\n", " response = requests.get(file_url)\n", " with open(file_path, \"wb\") as f:\n", " f.write(response.content)\n", "\n", " # load the file to okareo\n", " source_scenario = okareo.upload_scenario_set(file_path=file_path, scenario_name=scenario_name)\n", "\n", " # delete the file\n", " os.remove(file_path)\n", " return source_scenario\n", "\n", "source_scenario = load_or_download_file(file_path_articles, scenario_name_articles)\n", "print(f\"{scenario_name_articles}: {source_scenario.app_link}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With a seed scenario defined, let's use Okareo to generate some synthetic scenarios." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Rephrasing generator\n", "\n", "Suppose we want to generate a scenario where the same `result`s should be retrieved under `input`s with minor changes to word order. We can achieve this using the Rephrasing generator, which attempts to change the wording of each sentence in a given `input`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ScenarioSetResponse(scenario_id='4972d1d2-d33d-4512-b16a-de997c780b25', project_id='21b8a05b-f5b6-4578-a36a-fc264036d9d3', time_created=datetime.datetime(2024, 3, 8, 21, 20, 24, 90865), type='REPHRASE_INVARIANT', tags=['seed:3a904bd0-3dfa-42e5-a10a-d42135b9077e'], name='Retrieval Articles Scenario: Rephrased', seed_data=[], scenario_count=0, scenario_input=[], app_link='https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/4972d1d2-d33d-4512-b16a-de997c780b25', additional_properties={})\n", "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/4972d1d2-d33d-4512-b16a-de997c780b25\n" ] } ], "source": [ "from okareo_api_client.models import ScenarioType\n", "\n", "generated_scenario = okareo.generate_scenarios(\n", " source_scenario=source_scenario.scenario_id,\n", " name=\"Retrieval Articles Scenario: Rephrased\",\n", " number_examples=1, # number of examples to generate per row in seed scenario\n", " generation_type=ScenarioType.REPHRASE_INVARIANT\n", ")\n", "\n", "print(generated_scenario)\n", "print(generated_scenario.app_link)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's compare the generated scenarios to their corresponding seed scenarios." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# helper methods to compare the seed/generated articles\n", "def make_points_to_dict(points):\n", " \"\"\"\n", " note: using the `result`s as keys will not generalize to any scenario,\n", " but it works for the scenarios in this notebook.\n", " \"\"\"\n", " d = {}\n", " for p in points:\n", " res = p.result if type(p.result) == str else p.result[0]\n", " if res not in d.keys():\n", " d[res] = [p.input_]\n", " else:\n", " d[res].append(p.input_)\n", " return d\n", "\n", "def compare_seed_and_generated(seed_scenario, generated_scenario):\n", " gen_points = okareo.get_scenario_data_points(generated_scenario.scenario_id)\n", " seed_points = okareo.get_scenario_data_points(seed_scenario.scenario_id)\n", " gen_d = make_points_to_dict(gen_points)\n", " seed_d = make_points_to_dict(seed_points)\n", " N = len(gen_d)\n", " for i, key in enumerate(seed_d.keys()):\n", " print(\"-\"*8 + f\"Seed #{i}\" + \"-\"*8)\n", " print(seed_d[key][0])\n", " for j in range(len(gen_d[key])):\n", " print(\"-\"*5 + f\"Generated #{j}\" + \"-\"*6)\n", " print(gen_d[key][j])\n", " print(\"-\"*4 + f\"End of Seed #{i}\" + \"-\"*5)\n", " " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------Seed #0--------\n", "WebBizz is dedicated to providing our customers with a seamless online shopping experience. Our platform is designed with user-friendly interfaces to help you browse and select the best products suitable for your needs. We offer a wide range of products from top brands and new entrants, ensuring diversity and quality in our offerings. Our 24/7 customer support is ready to assist you with any queries, from product details, shipping timelines, to payment methods. We also have a dedicated FAQ section addressing common concerns. Always ensure you are logged in to enjoy personalized product recommendations and faster checkout processes.\n", "-----Generated #0------\n", "WebBizz prioritizes a smooth digital shopping journey for our customers. Our platform is tailored with straightforward interfaces for easier product browsing and selection. We present a broad selection of items from well-recognized brands and emerging ones, keeping diversity and high standards. We have round-the-clock customer service prepared to help with inquiries, from product specifics, dispatch timeframes, right down to payment procedures. We also feature a comprehensive FAQ section for usual questions. Make sure you're signed in to receive custom product suggestions and expedited checkout procedures.\n", "----End of Seed #0-----\n", "--------Seed #1--------\n", "Safety and security of your data is our top priority at WebBizz. Our platform employs state-of-the-art encryption methods ensuring your personal and financial information remains confidential. Our two-factor authentication at checkout provides an added layer of security. We understand the importance of timely deliveries, hence we've partnered with reliable logistics partners ensuring your products reach you in pristine condition. In case of any delays or issues, our tracking tool can provide real-time updates on your product's location. We believe in transparency and guarantee no hidden fees or charges during your purchase journey.\n", "-----Generated #0------\n", "WebBizz prioritizes the protection and privacy of your data by using advanced encryption techniques, maintaining the confidentially of your personal and financial information. For additional safety, we have implemented a two-factor authentication system at checkout. We acknowledge the significance of prompt deliveries and to ensure this, we've associated with dependable logistics partners. This guarantees the delivery of your products in perfect condition. Our tracking tool can keep you updated on your product's whereabouts in real-time in case of any delay or issues. Also, we practice transparency and assure you there will not be any concealed costs or charges during the process of your purchase.\n", "----End of Seed #1-----\n", "--------Seed #2--------\n", "WebBizz places immense value on its dedicated clientele, recognizing their loyalty through the exclusive 'Premium Club' membership. This special program is designed to enrich the shopping experience, providing a suite of benefits tailored to our valued members. Among the advantages, members enjoy complimentary shipping, granting them a seamless and cost-effective way to receive their purchases. Additionally, the 'Premium Club' offers early access to sales, allowing members to avail themselves of promotional offers before they are opened to the general public.\n", "-----Generated #0------\n", "WebBizz deeply appreciates its loyal customers and expresses this through the 'Premium Club' membership. This unique program enhances the shopping journey, extending customized benefits for our cherished members. Perks include free delivery, ensuring a smooth and wallet-friendly acquisition of purchases. 'Premium Club' also provides priority access to sales, enabling members to capitalize on promotional deals prior to public availability.\n", "----End of Seed #2-----\n", "--------Seed #3--------\n", "At WebBizz, we recognize that your shopping preferences can change over time. That's why we've introduced the 'Wishlist' feature, allowing you to save products that catch your eye, for easy access whenever you're ready to make a purchase. This not only saves time but ensures that the items you're interested in don't get lost in the shuffle of our extensive inventory. The Wishlist goes beyond simple convenience. When the time comes to buy, your selected items are just a click away, streamlining the shopping process. Furthermore, we offer special promotions on wishlisted items, providing an opportunity for savings on products you've shown interest in.\n", "-----Generated #0------\n", "With the evolving shopping needs at WebBizz, we offer the 'Wishlist' function, enabling customers to keep track of items they are interested in and purchase them when ready, eliminating the risk of forgetting about them amid our broad product range. The wishlist makes the shopping journey more efficient, with items ready to be purchased in just one click. Plus, we offer exclusive deals on products saved in your wishlist, granting a chance for additional discounts on items you are keen on.\n", "----End of Seed #3-----\n", "--------Seed #4--------\n", "At WebBizz, we value our customer's feedback and are always striving to improve. Our product review section allows customers to share their experiences and helps others make informed decisions. If unsatisfied with a purchase, our easy return and refund policy ensures hassle-free returns within 30 days of purchase. We also offer a loyalty program, WebBizz Rewards, where customers can earn points with each purchase and avail exclusive discounts. For any further assistance, our 'Live Chat' feature connects you instantly with a customer support representative. We thank you for choosing WebBizz and look forward to serving you again!\n", "-----Generated #0------\n", "At WebBizz, customer satisfaction is our top priority. We have put in place a product review section for customers to share their experiences and help others make well-informed choices. If a customer isn't happy with a purchase, they can take advantage of our straightforward return and refund policy for hassle-free returns within a 30-day period. We also run a loyalty program, WebBizz Rewards, that allows customers to collect points with every purchase and receive unique discounts. If there's anything else you need, feel free to use our 'Live Chat' system to speak to a customer service representative directly. We appreciate your patronage at WebBizz and can't wait to assist you again!\n", "----End of Seed #4-----\n", "--------Seed #5--------\n", "Are you facing hurdles with technical glitches? At WebBizz, we understand how frustrating technical problems can be, which is why our 'Help Center' is meticulously designed to assist you through any difficulties you might encounter. Our Help Center is a treasure trove of resources, offering detailed step-by-step guides aimed at resolving common issues that our users face. Whether you're struggling to reset your password, having trouble logging in, or looking to track your recent orders, our comprehensive guides are here to navigate you through the process smoothly.\n", "-----Generated #0------\n", "Having trouble with technical issues? At WebBizz, we know how vexing these problems can be. That's why our 'Help Center' is carefully built to guide you when you hit any bumps. It is a reservoir of useful information, providing thorough walkthroughs tailored to help resolve frequent issues encountered by our users. Whether you are unable to reset your password, encountering login difficulties, or seeking to follow your latest orders, our exhaustive guides will help you sail through the procedure without any hitch.\n", "----End of Seed #5-----\n", "--------Seed #6--------\n", "Navigating WebBizz is a breeze with our advanced search functionalities. Whether you're looking for a specific product or exploring our latest collections, our dynamic filters and sorting options will guide you effortlessly. You can sort products based on popularity, price, and ratings to find the perfect fit for your needs.\n", "-----Generated #0------\n", "You'll find it easy to navigate WebBizz thanks to our superior search features. Looking for a particular product or checking out our newest collections becomes simple with our flexible filtering and sorting mechanisms. Arrange items in order of popularity, cost, or ratings to discover what's ideal for you.\n", "----End of Seed #6-----\n", "--------Seed #7--------\n", "We're proud of our diverse product catalog that caters to customers globally. From eco-friendly products to exclusive designer collaborations, WebBizz curates its collections with care. We also frequently feature limited-time offers and flash sales, so keep an eye out for these exclusive deals!\n", "-----Generated #0------\n", "At WebBizz, we take pride in our wide-ranging product catalog which is designed to serve customers all over the world. We pay special attention to curating ecologically mindful items along with exclusive designer partnerships. You should not miss our occasional limited-availability deals and quick sales, these are the deals you may want to watch out for!\n", "----End of Seed #7-----\n", "--------Seed #8--------\n", "Subscribing to our newsletter gives you a front-row seat to all things WebBizz. Be the first to know about upcoming sales, new product launches, and exciting company updates. Plus, subscribers get an exclusive monthly discount code as a thank you for being part of our community.\n", "-----Generated #0------\n", "By signing up for our newsletter you gain direct access to everything happening at WebBizz. Stay up to date with our latest sales, newly launched products, and notable news about our company. Moreover, as a token of our gratitude, we provide an exclusive monthly discount code to our subscribers.\n", "----End of Seed #8-----\n", "--------Seed #9--------\n", "WebBizz believes in sustainability. We've integrated eco-friendly packaging and support brands that follow ethical production practices. By shopping with us, you're making a choice to support a greener planet. To learn more, visit our 'Sustainability Initiatives' page.\n", "-----Generated #0------\n", "WebBizz is committed to sustainability. We use environmentally friendly packaging and back companies that uphold ethical manufacturing. Choosing to shop with us implies you're aiding a more sustainable planet. For further details, check out our 'Sustainability Initiatives' page.\n", "----End of Seed #9-----\n" ] } ], "source": [ "compare_seed_and_generated(source_scenario, generated_scenario)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Term Relevance generator\n", "\n", "Suppose we want to generate a scenario based on keywords from the same `input`s. We can generate such a scenario using the Term Relevance generator, which extracts the most relevant terms based on [term frequency-inverse document frequency (tf-idf)](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/6a500f0f-9483-47dd-a9a6-99bfe0fd77fd\n" ] } ], "source": [ "generated_scenario_tr = okareo.generate_scenarios(\n", " source_scenario=source_scenario.scenario_id,\n", " name=\"Retrieval Articles Scenario: Term Relevance\",\n", " number_examples=1, # number of examples to generate per seed data\n", " generation_type=ScenarioType.TERM_RELEVANCE_INVARIANT\n", ")\n", "\n", "print(generated_scenario_tr.app_link)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------Seed #0--------\n", "WebBizz is dedicated to providing our customers with a seamless online shopping experience. Our platform is designed with user-friendly interfaces to help you browse and select the best products suitable for your needs. We offer a wide range of products from top brands and new entrants, ensuring diversity and quality in our offerings. Our 24/7 customer support is ready to assist you with any queries, from product details, shipping timelines, to payment methods. We also have a dedicated FAQ section addressing common concerns. Always ensure you are logged in to enjoy personalized product recommendations and faster checkout processes.\n", "-----Generated #0------\n", "dedicated products product\n", "----End of Seed #0-----\n", "--------Seed #1--------\n", "Safety and security of your data is our top priority at WebBizz. Our platform employs state-of-the-art encryption methods ensuring your personal and financial information remains confidential. Our two-factor authentication at checkout provides an added layer of security. We understand the importance of timely deliveries, hence we've partnered with reliable logistics partners ensuring your products reach you in pristine condition. In case of any delays or issues, our tracking tool can provide real-time updates on your product's location. We believe in transparency and guarantee no hidden fees or charges during your purchase journey.\n", "-----Generated #0------\n", "ensuring security art\n", "-----Generated #1------\n", "confidential security art\n", "----End of Seed #1-----\n", "--------Seed #2--------\n", "WebBizz places immense value on its dedicated clientele, recognizing their loyalty through the exclusive 'Premium Club' membership. This special program is designed to enrich the shopping experience, providing a suite of benefits tailored to our valued members. Among the advantages, members enjoy complimentary shipping, granting them a seamless and cost-effective way to receive their purchases. Additionally, the 'Premium Club' offers early access to sales, allowing members to avail themselves of promotional offers before they are opened to the general public.\n", "-----Generated #0------\n", "offers members club\n", "-----Generated #1------\n", "complimentary members club\n", "----End of Seed #2-----\n", "--------Seed #3--------\n", "At WebBizz, we recognize that your shopping preferences can change over time. That's why we've introduced the 'Wishlist' feature, allowing you to save products that catch your eye, for easy access whenever you're ready to make a purchase. This not only saves time but ensures that the items you're interested in don't get lost in the shuffle of our extensive inventory. The Wishlist goes beyond simple convenience. When the time comes to buy, your selected items are just a click away, streamlining the shopping process. Furthermore, we offer special promotions on wishlisted items, providing an opportunity for savings on products you've shown interest in.\n", "-----Generated #0------\n", "ve items time\n", "----End of Seed #3-----\n", "--------Seed #4--------\n", "At WebBizz, we value our customer's feedback and are always striving to improve. Our product review section allows customers to share their experiences and helps others make informed decisions. If unsatisfied with a purchase, our easy return and refund policy ensures hassle-free returns within 30 days of purchase. We also offer a loyalty program, WebBizz Rewards, where customers can earn points with each purchase and avail exclusive discounts. For any further assistance, our 'Live Chat' feature connects you instantly with a customer support representative. We thank you for choosing WebBizz and look forward to serving you again!\n", "-----Generated #0------\n", "purchase customer customers\n", "----End of Seed #4-----\n", "--------Seed #5--------\n", "Are you facing hurdles with technical glitches? At WebBizz, we understand how frustrating technical problems can be, which is why our 'Help Center' is meticulously designed to assist you through any difficulties you might encounter. Our Help Center is a treasure trove of resources, offering detailed step-by-step guides aimed at resolving common issues that our users face. Whether you're struggling to reset your password, having trouble logging in, or looking to track your recent orders, our comprehensive guides are here to navigate you through the process smoothly.\n", "-----Generated #0------\n", "step center guides\n", "-----Generated #1------\n", "technical center guides\n", "----End of Seed #5-----\n", "--------Seed #6--------\n", "Navigating WebBizz is a breeze with our advanced search functionalities. Whether you're looking for a specific product or exploring our latest collections, our dynamic filters and sorting options will guide you effortlessly. You can sort products based on popularity, price, and ratings to find the perfect fit for your needs.\n", "-----Generated #0------\n", "advanced breeze filters\n", "-----Generated #1------\n", "dynamic breeze filters\n", "----End of Seed #6-----\n", "--------Seed #7--------\n", "We're proud of our diverse product catalog that caters to customers globally. From eco-friendly products to exclusive designer collaborations, WebBizz curates its collections with care. We also frequently feature limited-time offers and flash sales, so keep an eye out for these exclusive deals!\n", "-----Generated #0------\n", "care catalog caters\n", "-----Generated #1------\n", "exclusive catalog caters\n", "----End of Seed #7-----\n", "--------Seed #8--------\n", "Subscribing to our newsletter gives you a front-row seat to all things WebBizz. Be the first to know about upcoming sales, new product launches, and exciting company updates. Plus, subscribers get an exclusive monthly discount code as a thank you for being part of our community.\n", "-----Generated #0------\n", "gives code community\n", "-----Generated #1------\n", "exciting code community\n", "----End of Seed #8-----\n", "--------Seed #9--------\n", "WebBizz believes in sustainability. We've integrated eco-friendly packaging and support brands that follow ethical production practices. By shopping with us, you're making a choice to support a greener planet. To learn more, visit our 'Sustainability Initiatives' page.\n", "-----Generated #0------\n", "support sustainability choice\n", "-----Generated #1------\n", "ethical sustainability choice\n", "----End of Seed #9-----\n" ] } ], "source": [ "compare_seed_and_generated(source_scenario, generated_scenario_tr)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Seed Scenario Creation via Static Definition \n", "\n", "In addition to uploading a list of json objects from a `.jsonl` file, we can also statically define a scenario by explicitly defining `SeedData` objects. The following cell shows you the required imports and arguments to do this." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/64056783-d6f9-471e-b978-dfb25a6baa35\n" ] } ], "source": [ "from okareo_api_client.models import ScenarioSetCreate, SeedData\n", "\n", "# list of statically defined seed data\n", "seed_data=[\n", " SeedData(input_=\"The quick brown fox jumps over the lazy dog\", result=\"result1\"),\n", " SeedData(input_=\"The rain in Spain falls mainly on the plain\", result=\"result2\"),\n", " SeedData(input_=\"Lorem ipsum dolor sit amet, consectetur adipiscing elit\", result=\"result3\")\n", "]\n", "\n", "# request for scenario set creation \n", "scenario_set_create = ScenarioSetCreate(\n", " name=\"Statically Defined Scenario: Seed\",\n", " generation_type=ScenarioType.SEED,\n", " number_examples=len(seed_data), # number of examples to generate per seed data\n", " seed_data=seed_data\n", ")\n", "\n", "source_scenario_static = okareo.create_scenario_set(scenario_set_create)\n", "\n", "print(source_scenario_static.app_link)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mispelling generator\n", "\n", "Using this statically defined scenario, suppose we want to introduce errors into the `input`s. The Misspelling generator does this by emulating human typing error, i.e. by editing random characters in the input to different, adjacent keys on a typical QWERTY keyboard." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ScenarioSetResponse(scenario_id='bb29a3b3-d69d-46ac-9693-ae7fa99c444d', project_id='21b8a05b-f5b6-4578-a36a-fc264036d9d3', time_created=datetime.datetime(2024, 3, 8, 21, 23, 27, 187544), type='COMMON_MISSPELLINGS', tags=['seed:0c29d21a-e91c-4950-b6e9-818a0635c552'], name='Statically Defined Scenario: Misspelling', seed_data=[], scenario_count=0, scenario_input=[], app_link='https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/bb29a3b3-d69d-46ac-9693-ae7fa99c444d', additional_properties={})\n", "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/bb29a3b3-d69d-46ac-9693-ae7fa99c444d\n" ] } ], "source": [ "generated_scenario_mis = okareo.generate_scenarios(\n", " source_scenario=source_scenario_static.scenario_id,\n", " name=\"Statically Defined Scenario: Misspelling\",\n", " number_examples=2, # number of examples to generate per seed data\n", " generation_type=ScenarioType.COMMON_MISSPELLINGS\n", ")\n", "\n", "print(generated_scenario_mis)\n", "print(generated_scenario_mis.app_link)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------Seed #0--------\n", "The quick brown fox jumps over the lazy dog\n", "-----Generated #0------\n", "The quick brown fox jumps over the lazt dog\n", "-----Generated #1------\n", "The quick brown fox humps over the lazy dog\n", "----End of Seed #0-----\n", "--------Seed #1--------\n", "The rain in Spain falls mainly on the plain\n", "-----Generated #0------\n", "Rhe rain in Spain falls mainly on the plain\n", "-----Generated #1------\n", "The rain in Spain falls mainly on tge plain\n", "----End of Seed #1-----\n", "--------Seed #2--------\n", "Lorem ipsum dolor sit amet, consectetur adipiscing elit\n", "-----Generated #0------\n", "Lorem ipsum dolor sit amet, consrctetur adipiscing elit\n", "-----Generated #1------\n", "Loeem ipsum dolor sit amet, consectetur adipiscing elit\n", "----End of Seed #2-----\n" ] } ], "source": [ "compare_seed_and_generated(source_scenario_static, generated_scenario_mis)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Contraction generator\n", "\n", "Contractions and abbreviations can occur commonly in human-written documents. The Contraction generator lets you generate a scenario that tries to find common abbreviations of strings in the `input`s." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/ecb0e6f8-e1bf-4527-ae23-de7d5d9b10c3\n" ] } ], "source": [ "generated_scenario_con = okareo.generate_scenarios(\n", " source_scenario=source_scenario_static.scenario_id,\n", " name=\"Statically Defined Scenario: Contractions\",\n", " number_examples=1, # number of examples to generate per seed data\n", " generation_type=ScenarioType.COMMON_CONTRACTIONS\n", ")\n", "\n", "print(generated_scenario_con.app_link)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------Seed #0--------\n", "The quick brown fox jumps over the lazy dog\n", "-----Generated #0------\n", "The quick brwn fox jumps over the lazy dog\n", "----End of Seed #0-----\n", "--------Seed #1--------\n", "The rain in Spain falls mainly on the plain\n", "-----Generated #0------\n", "The rain in Spain falls manly on the plain\n", "----End of Seed #1-----\n", "--------Seed #2--------\n", "Lorem ipsum dolor sit amet, consectetur adipiscing elit\n", "-----Generated #0------\n", "Lorem ipsum dolor sit amet, consctetur adipiscing elit\n", "----End of Seed #2-----\n" ] } ], "source": [ "compare_seed_and_generated(source_scenario_static, generated_scenario_con)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reverse Question generator\n", "\n", "Suppose we would like to generate some common user queries that are based on the original `input`s. The Reverse Question generator enables this by generating questions where the answer is contained in the original input." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ScenarioSetResponse(scenario_id='b5b990d0-8a00-4c83-8973-76d8ddc51202', project_id='21b8a05b-f5b6-4578-a36a-fc264036d9d3', time_created=datetime.datetime(2024, 3, 8, 21, 24, 45, 358913), type='TEXT_REVERSE_QUESTION', tags=['seed:3a904bd0-3dfa-42e5-a10a-d42135b9077e'], name='Retrieval Articles Scenario: Reverse Question', seed_data=[], scenario_count=0, scenario_input=[], app_link='https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/b5b990d0-8a00-4c83-8973-76d8ddc51202', additional_properties={})\n", "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/b5b990d0-8a00-4c83-8973-76d8ddc51202\n" ] } ], "source": [ "generated_scenario_question = okareo.generate_scenarios(\n", " source_scenario=source_scenario.scenario_id,\n", " name=\"Retrieval Articles Scenario: Reverse Question\",\n", " number_examples=1, # number of examples to generate per seed data\n", " generation_type=ScenarioType.TEXT_REVERSE_QUESTION\n", ")\n", "\n", "print(generated_scenario_question)\n", "print(generated_scenario_question.app_link)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------Seed #0--------\n", "WebBizz is dedicated to providing our customers with a seamless online shopping experience. Our platform is designed with user-friendly interfaces to help you browse and select the best products suitable for your needs. We offer a wide range of products from top brands and new entrants, ensuring diversity and quality in our offerings. Our 24/7 customer support is ready to assist you with any queries, from product details, shipping timelines, to payment methods. We also have a dedicated FAQ section addressing common concerns. Always ensure you are logged in to enjoy personalized product recommendations and faster checkout processes.\n", "-----Generated #0------\n", "What features does WebBizz offer to enhance the customer's online shopping experience?\n", "----End of Seed #0-----\n", "--------Seed #1--------\n", "Safety and security of your data is our top priority at WebBizz. Our platform employs state-of-the-art encryption methods ensuring your personal and financial information remains confidential. Our two-factor authentication at checkout provides an added layer of security. We understand the importance of timely deliveries, hence we've partnered with reliable logistics partners ensuring your products reach you in pristine condition. In case of any delays or issues, our tracking tool can provide real-time updates on your product's location. We believe in transparency and guarantee no hidden fees or charges during your purchase journey.\n", "-----Generated #0------\n", "What measures does WebBizz take to ensure the security of personal and financial data?\n", "----End of Seed #1-----\n", "--------Seed #2--------\n", "WebBizz places immense value on its dedicated clientele, recognizing their loyalty through the exclusive 'Premium Club' membership. This special program is designed to enrich the shopping experience, providing a suite of benefits tailored to our valued members. Among the advantages, members enjoy complimentary shipping, granting them a seamless and cost-effective way to receive their purchases. Additionally, the 'Premium Club' offers early access to sales, allowing members to avail themselves of promotional offers before they are opened to the general public.\n", "-----Generated #0------\n", "What are some benefits of being a 'Premium Club' member at WebBizz?\n", "----End of Seed #2-----\n", "--------Seed #3--------\n", "At WebBizz, we recognize that your shopping preferences can change over time. That's why we've introduced the 'Wishlist' feature, allowing you to save products that catch your eye, for easy access whenever you're ready to make a purchase. This not only saves time but ensures that the items you're interested in don't get lost in the shuffle of our extensive inventory. The Wishlist goes beyond simple convenience. When the time comes to buy, your selected items are just a click away, streamlining the shopping process. Furthermore, we offer special promotions on wishlisted items, providing an opportunity for savings on products you've shown interest in.\n", "-----Generated #0------\n", "How does the 'Wishlist' feature on WebBizz aid in enhancing the shopping experience?\n", "----End of Seed #3-----\n", "--------Seed #4--------\n", "At WebBizz, we value our customer's feedback and are always striving to improve. Our product review section allows customers to share their experiences and helps others make informed decisions. If unsatisfied with a purchase, our easy return and refund policy ensures hassle-free returns within 30 days of purchase. We also offer a loyalty program, WebBizz Rewards, where customers can earn points with each purchase and avail exclusive discounts. For any further assistance, our 'Live Chat' feature connects you instantly with a customer support representative. We thank you for choosing WebBizz and look forward to serving you again!\n", "-----Generated #0------\n", "What is the primary benefit of joining the WebBizz Rewards program?\n", "----End of Seed #4-----\n", "--------Seed #5--------\n", "Are you facing hurdles with technical glitches? At WebBizz, we understand how frustrating technical problems can be, which is why our 'Help Center' is meticulously designed to assist you through any difficulties you might encounter. Our Help Center is a treasure trove of resources, offering detailed step-by-step guides aimed at resolving common issues that our users face. Whether you're struggling to reset your password, having trouble logging in, or looking to track your recent orders, our comprehensive guides are here to navigate you through the process smoothly.\n", "-----Generated #0------\n", "What resources does WebBizz's Help Center offer to assist users with common issues?\n", "----End of Seed #5-----\n", "--------Seed #6--------\n", "Navigating WebBizz is a breeze with our advanced search functionalities. Whether you're looking for a specific product or exploring our latest collections, our dynamic filters and sorting options will guide you effortlessly. You can sort products based on popularity, price, and ratings to find the perfect fit for your needs.\n", "-----Generated #0------\n", "What features does WebBizz provide to enhance your search experience?\n", "----End of Seed #6-----\n", "--------Seed #7--------\n", "We're proud of our diverse product catalog that caters to customers globally. From eco-friendly products to exclusive designer collaborations, WebBizz curates its collections with care. We also frequently feature limited-time offers and flash sales, so keep an eye out for these exclusive deals!\n", "-----Generated #0------\n", "What types of products does WebBizz offer and how do they manage their special deals?\n", "----End of Seed #7-----\n", "--------Seed #8--------\n", "Subscribing to our newsletter gives you a front-row seat to all things WebBizz. Be the first to know about upcoming sales, new product launches, and exciting company updates. Plus, subscribers get an exclusive monthly discount code as a thank you for being part of our community.\n", "-----Generated #0------\n", "What special perk do subscribers to the WebBizz newsletter enjoy every month?\n", "----End of Seed #8-----\n", "--------Seed #9--------\n", "WebBizz believes in sustainability. We've integrated eco-friendly packaging and support brands that follow ethical production practices. By shopping with us, you're making a choice to support a greener planet. To learn more, visit our 'Sustainability Initiatives' page.\n", "-----Generated #0------\n", "What kind of packaging methods does WebBizz utilize to promote environmental sustainability?\n", "----End of Seed #9-----\n" ] } ], "source": [ "compare_seed_and_generated(source_scenario, generated_scenario_question)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conditional generator + Chaining generators\n", "\n", "Now suppose we would like to rephrase the questions we generated by the Reverse Question generator. The Conditional generator tries to do this by adding a qualifying phrase to the beginning of each question. \n", "\n", "Observe that the `source_scenario` argument below uses the output of the previous generation (`generated_scenario_question`) meaning the generated Reverse Question scenario is the input to the Conditional generator. This is one method for *chaining* generators together, which can let you layer different generators on one another and create composite generations." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://app.okareo.com/project/21b8a05b-f5b6-4578-a36a-fc264036d9d3/scenario/cf7d3acf-df7f-40ee-b059-29bb6d3457a3\n" ] } ], "source": [ "# generate a Conditional scenario with the generated Reverse Question scenario as a source\n", "generated_scenario_conditional = okareo.generate_scenarios(\n", " source_scenario=generated_scenario_question.scenario_id,\n", " name=\"Retrieval Articles Scenario: Conditional\",\n", " number_examples=1, # number of examples to generate per seed data\n", " generation_type=ScenarioType.CONDITIONAL\n", ")\n", "\n", "print(generated_scenario_conditional.app_link)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------Seed #0--------\n", "What features does WebBizz offer to enhance the customer's online shopping experience?\n", "-----Generated #0------\n", "Considering WebBizz, what features are offered to improve the online shopping experience for customers?\n", "----End of Seed #0-----\n", "--------Seed #1--------\n", "What measures does WebBizz take to ensure the security of personal and financial data?\n", "-----Generated #0------\n", "Given your stake in WebBizz, what measures does the company take to ensure the safety of personal and financial information?\n", "----End of Seed #1-----\n", "--------Seed #2--------\n", "What are some benefits of being a 'Premium Club' member at WebBizz?\n", "-----Generated #0------\n", "Given your status as a 'Premium Club' member at WebBizz, what are the potential benefits?\n", "----End of Seed #2-----\n", "--------Seed #3--------\n", "How does the 'Wishlist' feature on WebBizz aid in enhancing the shopping experience?\n", "-----Generated #0------\n", "Considering the 'Wishlist' feature on WebBizz, how might it enhance the shopping experience?\n", "----End of Seed #3-----\n", "--------Seed #4--------\n", "What is the primary benefit of joining the WebBizz Rewards program?\n", "-----Generated #0------\n", "Should you decide to join the WebBizz Rewards program, what would be the primary benefit?\n", "----End of Seed #4-----\n", "--------Seed #5--------\n", "What resources does WebBizz's Help Center offer to assist users with common issues?\n", "-----Generated #0------\n", "Given your need for assistance, what resources might you expect to find in WebBizz's Help Center to help with common issues?\n", "----End of Seed #5-----\n", "--------Seed #6--------\n", "What features does WebBizz provide to enhance your search experience?\n", "-----Generated #0------\n", "Considering your use of WebBizz, what features might it provide to enhance your browsing experience?\n", "----End of Seed #6-----\n", "--------Seed #7--------\n", "What types of products does WebBizz offer and how do they manage their special deals?\n", "-----Generated #0------\n", "Given the variety of products WebBizz offers, how do they manage their special deals?\n", "----End of Seed #7-----\n", "--------Seed #8--------\n", "What special perk do subscribers to the WebBizz newsletter enjoy every month?\n", "-----Generated #0------\n", "Having a subscription to the WebBizz newsletter, what exclusive advantage could one expect to enjoy every month?\n", "----End of Seed #8-----\n", "--------Seed #9--------\n", "What kind of packaging methods does WebBizz utilize to promote environmental sustainability?\n", "-----Generated #0------\n", "Dealing with sustainability goals, what packaging methods does WebBizz utilize to promote an eco-friendly environment?\n", "----End of Seed #9-----\n" ] } ], "source": [ "compare_seed_and_generated(generated_scenario_question, generated_scenario_conditional)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "dev", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 2 }