# Welcome In today's rapidly evolving AI landscape, building production-ready solutions can be challenging. Ownlayer empowers your team with a full suite of tools and services designed to streamline your AI development process, from MVP to production. {% embed url="" fullWidth="true" %} ## Overview: Core platform features ### Iteration * **Prompt**: Test, manage, and deploy AI prompts with version control. The Ownlayer interface allows for prompt optimization. * **Dataset**: Compile and interact with datasets for regression testing, POC validation and potential fine-tuning projects. ### Monitoring * **Evaluators**: Create evaluation criteria specific to your product requirements. Monitor system performance in real-time. * **Analyzers**: Examine AI interactions through various metrics, including sentiment analysis, topic categorization, and emotional content assessment. ### Integrations * **Data Export**: Send AI performance data to analytics platforms: * Posthog * Amplitude * Segment ## Get Started


Quick Start	Setup your Ownlayer account in minutes	Getting Started	1.png
Iterate	All you need to iterate on your app	Iterate	2.png
Monitor	All you need to keep track of and gain insights from your production logs	Monitor	4.png

# Quickstart This quick start helps you to integrate your AI application with Ownlayer in minitues ## Create An Account ### **Create an individual account** Sign up through . Each team member needs to create their accounts separately. ### **Create Organization** * Find your registered email address on the top right of the home screen. * Click on your email address to access the dropdown menu.

* Select 'Organizations'. This will bring you to your personal organization dashboard. If you'd like to create a separate organization to invite your team members, this is where you can do it

## Add LLM Provider Credentials To power any evals that use LLM as a judge or test prompts, you need to add the key from your LLM providers. Currently, we support OpenAI, Athorpic and Replicate. {% hint style="info" %} Provider Credentials aren't shared among members of the organization. For members to use the key, they will need to add them separately {% endhint %} If you have multiple organizations, choose the one you'd like to create an API key for from the left panel.

## **Create and Install Ownlayer API Key** Choose '**API Keys**' from the top panel and click '**Add API Key**' to generate the key. {% hint style="info" %} For security reasons, API keys are not synced among different members within the organization. {% endhint %}

### Python SDK The [**Ownlayer SDK**](https://github.com/OwnLayer/ownlayer_py_sdk) is our recommended way to integrate with Ownlayer. #### Install SDK ``` pip install ownlayer ``` Add add `OWNLAYER_API_KEY` to your `.env` file: ``` OWNLAYER_API_KEY=ey...xxx ``` Change OpenAI or Anthropic imports to use Ownlayer wrapper ``` - import openai + from ownlayer.openai import openai ``` ### Add API Key to Your Header You can also integrate with Ownlayer directly through API * Copy the generated API key and add it to the header of your API requests. * Follow our full [**API documentation**](https://app.ownlayer.com/docs) for detailed instructions on how to use the API key. Example: {% code overflow="wrap" %} ```python import json # Prepare the data data = { "input": """SOLAR FARM LAND LEASE AGREEMENT This Solar Farm Land Lease Agreement ("Agreement") is made and entered into this 12 day of September, 2024, by and between: Landowner: Hava Chen, with a principal address at 1231 Broadway, San Francisco, CA 91211("Landowner"), and Lessee: Jack Wang, with a principal address at 117 Market Street, San Francisco, CA 91211("Lessee"). RECITALS WHEREAS, Landowner owns certain real property located at [Property Address or Legal Description], consisting of a 150-acre parcel of vacant land ("Premises"); and WHEREAS, Lessee desires to lease the Premises for the purpose of constructing, installing, operating, and maintaining a state-of-the-art solar panel farm and related facilities (collectively, the "Solar Facility"); and WHEREAS, Landowner agrees to lease the Premises to Lessee under the terms and conditions set forth herein. NOW, THEREFORE, in consideration of the mutual covenants and agreements herein contained, the parties hereto agree as follows: 1. LEASE OF PREMISES 1.1 Lease Grant. Landowner hereby leases to Lessee, and Lessee hereby leases from Landowner, the Premises, for the purpose of constructing, installing, operating, and maintaining the Solar Facility. 1.2 Description of Premises. The Premises subject to this lease is described in Exhibit A, attached hereto and made a part hereof. 2. TERM 2.1 Initial Term. The initial term of this Agreement shall be for a period of twenty (20) years, commencing on [Commencement Date] and expiring on [Expiration Date], unless sooner terminated as provided herein. 3. RENT 3.1 Base Rent. Lessee shall pay to Landowner an annual base rent of Five Hundred Thousand Dollars ($500,000.00) ("Base Rent") for the use of the Premises. The Base Rent shall be payable in advance in equal monthly installments on the first day of each month. 3.2 Annual Increase. The Base Rent shall increase annually by two and one-half percent (2.5%) to account for inflation and market adjustments. 3.3 Percentage Rent. In addition to the Base Rent, Lessee shall pay to Landowner an amount equal to one percent (1%) of the gross revenues generated from the Solar Facility ("Percentage Rent") each year. The Percentage Rent shall be calculated and paid annually, based on the Lessee's fiscal year, within sixty (60) days after the end of each such fiscal year. 3.4 Payment Method. All rent payments shall be made payable to Landowner and delivered to [Landowner's Payment Address], or to such other place as Landowner may designate in writing from time to time. 4. USE OF PREMISES 4.1 Permitted Use. Lessee shall use the Premises solely for the purpose of constructing, installing, operating, and maintaining the Solar Facility and for no other purpose without the prior written consent of Landowner. 5. MAINTENANCE AND REPAIRS 5.1 Lessee's Obligations. Lessee shall, at its own expense, maintain the Premises and the Solar Facility in good order, condition, and repair, reasonable wear and tear excepted. 6. INSURANCE AND INDEMNITY 6.1 Insurance. Lessee shall obtain and maintain in full force and effect during the term of this Agreement insurance coverage as specified in Exhibit B, attached hereto and made a part hereof. 6.2 Indemnity. Lessee agrees to indemnify, defend, and hold harmless Landowner from and against any and all claims, liabilities, losses, damages, costs, and expenses (including reasonable attorneys' fees) arising out of or in connection with Lessee's use of the Premises or the Solar Facility. 7. DEFAULT AND TERMINATION 7.1 Events of Default. The occurrence of any of the following shall constitute an event of default by Lessee under this Agreement: (a) Failure to pay any rent or other amount due hereunder within ten (10) days after written notice from Landowner; (b) Failure to perform any other covenant or condition of this Agreement within thirty (30) days after written notice from Landowner. 7.2 Landowner's Remedies. Upon the occurrence of an event of default by Lessee, Landowner may, at its option, terminate this Agreement and recover possession of the Premises. 8. MISCELLANEOUS 8.1 Governing Law. This Agreement shall be governed by and construed in accordance with the laws of the state of [State]. 8.2 Entire Agreement. This Agreement constitutes the entire agreement between the parties and supersedes all prior and contemporaneous agreements, representations, and understandings. 8.3 Amendments. This Agreement may be amended only by a written instrument signed by both parties. IN WITNESS WHEREOF, the parties hereto have executed this Solar Farm Land Lease Agreement as of the day and year first above written. """, "output": """This Solar Farm Land Lease Agreement is between a Landowner and a Lessee for leasing a 150-acre parcel of land to construct and operate a solar panel farm. The key terms include: 1. Lease Terms: A 20-year lease, with the Lessee paying an annual base rent of $500,000, increasing by 2.5% annually, plus 1% of the solar farm's gross revenues. 2. Use of Premises: The Lessee can only use the land for the solar facility. 3. Maintenance: The Lessee is responsible for maintaining the property. 4. Insurance and Indemnity: The Lessee must maintain insurance and indemnify the Landowner against any claims. 5. Default and Termination: Specifies default conditions and remedies for the Landowner. 6. Miscellaneous: The agreement is governed by state law, constitutes the entire agreement, and can be amended in writing. The agreement ends with signatures from both parties.""", "tags": ["Summary"], "additional_metadata": { "model":"Claude 3.5 Sonnet", "user_id":"9c073ed8-123d-4dc8-bd25-b875009b3681", "analytics":"https://app.amplitude.com/analytics/solarFarm/7165sd", "production":"true", "session_id":"7165sd-2478d-58d0-bd25-c8do0098d298", "prompt_version":"draft_v7" }, "settings":{"vendor":"anthropic","max_generation_tokens":500}, "prompt_tokens": 54, "completion_tokens": 14, "total_tokens": 68, } # Make the POST request response = requests.post( "https://app.ownlayer.com/api/v1/inferences/", json=data, headers={ "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoiMzJmY2YyMjItNTFlNC00Y2RhLTg3YmUtZjc3OWMxOTY5NWRiIiwia2V5X2lkIjoiN2VlMjhkN2YtY2Y1OS00ZDViLWJkMTktMmRhNGJkZmM3NDE0IiwiZXhwIjoxNzUyODY0NjA3fQ.Zt-B5hBo63P4ueUJdUewqGXUZMcYLO_lfJF7EOPStPE" }, ) # Print the response print(response.status_code) print(response.json()) ``` {% endcode %} # Prompt Use Ownlayer to systematically version manage, optimize and iterate on the prompt you use in LLM application ## **Automated Prompt Generation**

* **One-Click Prompt Generation:** Effortlessly create prompts with a single click using proven and validated methodologies. * **Supported Techniques:** * **Chain-of-Thought Reasoning:** Generate prompts that encourage logical, step-by-step thinking to improve task accuracy. * **Few-Shot Sentiment Classification:** Design prompts with a few labeled examples to guide the model for sentiment analysis tasks. * **Additional Established Approaches:** Leverage other widely-recognized prompt engineering techniques (complete list available [here](https://www.promptingguide.ai/)). ## **Automated Prompt Optimization**

* **Describe the Task:** Clearly state the task or objective you aim to optimize for. * **Provide Prompt Examples and Assign Scores:** Include several example prompts and assign a score to each to establish a baseline for optimization. * **Define Generative Settings:** * **Max Steps:** Specify the maximum number of iterations the optimizer should perform. * **Prompts per Step:** Indicate the number of prompts to be generated or evaluated in each iteration. * **Top K:** Select the top *K* prompts to serve as the baseline for the next iteration. * **Choose Your Scoring Agent:** * **Supervised Mode:** Select a golden dataset to optimize prompt performance. Ensure the dataset includes at least two columns: *inputs* and *references* that represent your ideal output. * **Unsupervised Mode:** Provide a text description that clearly defines the goal or desired outcome for the prompts. ## **Seamless Deployment** * Deploy your chosen prompt version directly to production These features are designed to enhance your prompt engineering workflow, from initial creation to final deployment. # Dataset Datasets are crucial for AI product development, fine-tuning, and prompt optimization. Ownlayer provides tools to efficiently create, evaluate, and utilize datasets tailored to your needs. **Dataset Creation:** Compile datasets for various purposes (e.g., regression tests, production corner cases). **Dataset Evaluation:** Assess the quality of an entire dataset. **Data point Evaluation:** Experiment with new prompts and model configurations directly on your curated datasets. # Dataset Creation ## **Create Dataset Schema** ### **Access Datasets** * Navigate to the '**Datasets**' section from the left panel. ### **Create Dataset** * Click on 'Create dataset'. * Provide the following information: * **Name**: Give your dataset a unique name. * **Description**: Describe the dataset for future reference. * **Evaluation Variables**: This will be your data schema and variables used for evaluators. If you are not sure what variable means, check [here](/monitor/evaluators). * Click on 'Submit' to save the dataset schema. {% hint style="info" %} To evaluate the dataset as a whole, select "Prediction, Reference" from the dropdown menu. {% endhint %}

## **Populate Dataset** ### **Select the Dataset to Populate** * Choose the dataset you'd like to populate from your list of datasets. ### **Import Data** * Click the 'Import' button at the top right corner. * Upload your CSV file. * Map your desired columns from the CSV file to the evaluation variables. * Note that the maximum number of rows is currently capped at 1000. If you need extra rows, please contact us. ### **Other Methods to Populate Dataset** You can also populate dataset by * Manually add new datapoint * Save data from the playground * Save data from inferences # Data point Evaluation ## **Generate Predictions** You can generate predictions directly from the Dataset. This can be particularly helpful if you don't have predictions or would like to generate new predictions by passing your input data to specified LLM or custom endpoints.

## **Evaluate Individual Datapoint** * Select data points via the checkbox on the left * Select the desired evaluators Evaluation results will be populated as they finish, and detailed information is available when you click on them.

# Dataset Evaluation To measure the performance of a model on a dataset, it can be done by comparing the model's predictions to some ground truth references. We currently offer BLEU, ROUGE, WER, METEOR, and Accuracy. ## **Evaluate Dataset As A Whole** ### **Evaluate Dataset** * Click on 'Evaluate dataset'. * Choose the desired dataset evaluators from the list. ### **Review Evaluator Description** * Upon selection, a description of the evaluator will appear. Review this to ensure it matches your needs. * If everything looks right, click 'Create'.

# Evaluators Evaluation is a critical to score the performance of AI applications and curat well-labeled datasets Ownlayer offers 17 different kinds of evaluators covering basic search, JSON validation, and custom evaluators that leverage LLM as the judge. If you're not sure about what is the best, we recommend you start from **Playground** where you can test for the most suitable ones based on your data format. ### Variables

Attribute	Description
Input	Model input
Prediction	Model output
Prediction_b	A different model output from the same input
Reference	Ground truth

# Create Evaluators To create a new Evaluator, click on the 'Evaluators' from the left panel and click on the 'Create evaluator' button. Fill out the form to create the desired evaluator. This includes: * Name of the evaluator. * Description of what the evaluator does. * Select the evaluator type * Link the criterion you created earlier. * Select the LLM configuration to power the evaluator. ### Evaluator Type

Name	Description	Variable	Use LLM	Use Criterion	Use Prompt
Criterion	Evaluates a model based on a custom criterion. Use this evaluator when you have a custom criterion to evaluate the LLM output without needing a ground truth reference.	Input Prediction	true	true	true
Labeled Criterion	Evaluates a model based on a custom criterion, with a reference label. Use this evaluator when you have a custom criterion and would like to evaluate an output against a reference label.	Input Prediction Reference	true	true	true
Question Answering	Evaluates if the prediction answers the question posed in the input correctly, compared against a reference label.	Input Prediction Reference	true	false	true
Chain of Thought Question Answering	Given an input question, this evaluator determines if the prediction is correct using step-by-step reasoning process, compared against a reference label.	Input Prediction Reference	true	false	true
Context Question Answering	Evaluates if the prediction answers the question posed in the input correctly, using the context provided by the reference.	Input Prediction Reference	true	false	true
Score String	Evaluates the output on a scale of 1 to 10 based on a custom criterion. Use this evaluator when you have a custom criterion to evaluate the LLM output without needing a ground truth reference.	Input Prediction	true	true	true
Labeled Score String	Gives a score between 1 and 10 to a prediction based on a ground truth reference label. Use this evaluator when you have a custom criterion to evaluate the LLM output compared to a ground truth reference.	Input Prediction Reference	true	true	true
Pairwise String	Predicts the preferred prediction from between two models. When you have two predictions generated for the same input, this evaluator helps you choose the preferred one based on both your ground truth reference and custom criterion.	Input Prediction Prediction_b	true	true	true
Labeled Pairwise String	Predicts the preferred prediction from two models based on a ground truth reference label. When you have two predictions generated for the same input, this evaluator helps you choose the preferred one based on both your ground truth reference and custom criterion.	Input Prediction Prediction_b Reference	true	true	true
Pairwise String Distance	This evaluator compares two predictions using string edit distances.	Prediction Prediction_b	false	false	false
String Distance	This evaluator compares a prediction to a reference answer using string edit distances.	Prediction Reference	false	false	false
Embedding Distance	This evaluator compares a prediction to a reference answer using embedding distances.	Prediction Reference	false	false	false
Exact Match	Compares predictions to a reference answer using exact matching.	Prediction Prediction_b	false	false	false
Regex Match	Compares predictions to a reference answer using regular expressions.	Prediction Reference	false	false	false
JSON Validity	Checks if a prediction is valid JSON.	Prediction	false	false	false
JSON Equality	Tests if a prediction is equal to a reference JSON.	Prediction Reference	false	false	false
JSON Edit Distance	Computes a distance between two canonicalized JSON strings. Available algorithms include: Damerau-Levenshtein, Levenshtein, Jaro, Jaro-Winkler, Hamming, and Indel.	Prediction Reference	false	false	false
JSON Schema Validation	Checks if a prediction is valid JSON according to a JSON schema.	Prediction	false	false	false

### LLM as a judge To use LLM as a judge, you need to create #### Create a criterion * Click on '**Create criterion**'. * Describe your criterion in natural language, detailing what good performance looks like. The LLM will use this description as the standard to evaluate your data. * Click on '**Create**' to save the criterion.

#### Add LLM config We currently support models from OpenAI, Anthropic and open-source models through Replicate

## Smart Trigger an online Evaluator {% hint style="info" %} Note that only evaluators with **input** and **output** variables can be triggered for online evaluation in production {% endhint %} ### Add Trigger Logic To automatically evaluate an inference, you can add **Triggers** in the Evaluator detail page. * Select the evaluator you just created.

* Navigate to the 'Triggers' section within the evaluator's detail page, and click on '**Add tag**'

### Implement Trigger in Inference Stream * Ensure that the tags specified in the evaluators' triggers are included when you stream your inference data. * Follow the full [API documentation](https://app.ownlayer.com/docs#tag/inferences) for detailed instructions on how to implement triggers in your inference stream. # Smart Trigger an online Evaluator {% hint style="info" %} Note that only evaluators with **input** and **output** variables can be triggered for online evaluation in production {% endhint %} ### Add Trigger Logic To automatically evaluate an inference, you can add **Triggers** on the Evaluator detail page. * Select the evaluator you just created.

* Navigate to the 'Triggers' section within the evaluator's detail page, and click on '**Add tag**'

Easily create new evaluators or import existing ones for testing. \ For the testing data, you can manually add input them or import it from a dataset or inference table. ### Save Evaluator If you like, you can save the evaluator you just tested from Playground.

You can also save the created data point to a dataset. Note that the schema has to match. If you don't have a dataset that matches the datapoint's schema, you will need to first create an empty dataset # Analyzers Different from Evaluators, Analyzers do not have criteria and can be leveraged as a great tool to understand your product behavior. Ownlayer offers text classification and sentiment analysis ## Create an Analyzer To create a new Analyzer, click on the '**Analyzers**' from the left panel and click on the '**Create analyzer**' button. Fill out the form to create the desired analyzer. At minimum, this includes: * Name of the analyzer. * Description of what the analyzer does (optional). * Select the analyzer type ### Analyzer Type

Name	Description
Text Classification	Uses a text classification model based on provided labels. It assesses the likelihood of the input text belonging to each label category. Multi-label is available in settings
Positive/Negative Analysis	Uses a fine-tuned RoBERTa model for sentiment analysis, returning a score between 0 to 1 for positive, negative, and neutral.
Emotion Identificaiton	Uses a fine-tuned DistilRoBERTa model for comprehensive sentiment analysis. It assesses the emotional tone of input text, providing scores for 7 distinct sentiment categories: Anger 🤬 Disgust 🤢 Fear 😨 Joy 😀 Neutral 😐 Sadness 😭 Surprise 😲

Name

Description

Text Classification

Uses a text classification model based on provided labels. It assesses the likelihood of the input text belonging to each label category. Multi-label is available in settings

Positive/Negative Analysis

Uses a fine-tuned RoBERTa model for sentiment analysis, returning a score between 0 to 1 for positive, negative, and neutral.

Emotion Identificaiton

Uses a fine-tuned DistilRoBERTa model for comprehensive sentiment analysis. It assesses the emotional tone of input text, providing scores for 7 distinct sentiment categories:

Anger 🤬
Disgust 🤢
Fear 😨
Joy 😀
Neutral 😐
Sadness 😭
Surprise 😲

### Analyzer Settings On Analyzer detail page, you can specify additional settings #### Multiple Labels If you anticipate the response may belong to more than one labels, you'd want to enable the multi-label toggle.

#### Inference Attribute You can choose which variable to analyze

Name	Description
Input	The complete data submitted to the LLM.
Output	The generated response from the LLM
User Prompt	The user's query or input within the overall submission
Context	The context informtion within the submission
System Prompt	The instruction provided to the LLM as part of the entire input

## Smart Trigger An Analyzer in Production ### Add Trigger Logic You can add Triggers on the Analyzer's detail page to automatically analyze an inference. * Select the **Analyzer** you just created. * Navigate to the '**Triggers**' section within the Analyzer detail page, and click on '**Add tag**' ### Implement Trigger in Inference Stream * Ensure that the tags specified in the Analyzers' triggers are included when you stream your inference data. * Follow the full [API documentation](https://app.ownlayer.com/docs#tag/inferences) for detailed instructions on how to implement triggers in your inference stream.

# Segment ## Configuring Segment Integration * Navigate to your organization's integration settings. * Select the Segment integration option.

* Enter your Segment Write Key. * Choose the specific events you want to send to Segment. * Save your configuration.

When creating an inference, add a `segment` attribute to the `additional_metadata` field with the following structure:

{
    additional_metadata:{
        segment:{
            user_id:"82az9ddoi34fski29375fjnafd",
            event:"Evaluation Pass",
            properties: {
            },
            integrations: {    
            }
        }
    }
}

The `segment` object should include a `user_id` which is required and represents the ID of the user you're tracking. You can optionally specify an `event` name (if not provided, it uses the default event names). The `properties` object can contain any additional properties you want to track. The `integrations` object allows you to specify which integrations to send this event to. Refer to Segment documentation for more details on properties and integrations. # Amplitude ## Configuring Amplitude Integration * Navigate to your organization's integration settings. * Select the Amplitude integration option.

* Enter your Amplitude Write Key. * Choose the specific events you want to send to Amplitude. * Save your configuration. When creating an inference, add an `amplitude` attribute to the `additional_metadata` field with the structure below. You must include at least `user_id`or `device_id`. Refer to Amplitude documentation for more details on properties, including which are required or optional. ```json { additional_metadata:{ amplitude:{ user_id:"82az9ddoi34fski29375fjnafd", device_id: "device_id_456", event:"Evaluation Pass" } } } ``` # Posthog ## Configuring Posthog Integration * Navigate to your organization's integration settings. * Select the Posthog integration option.

* Enter your PostHog Project API Key and Host URL * Choose the specific events you want to send to Posthog. * Save your configuration. Use your PostHog Project API Key, not your personal API Key. When creating an inference, add an `posthog` attribute to the `additional_metadata` field with the structure below. ```json { additional_metadata:{ posthog:{ distinct_id:"82az9ddoi34fski29375fjnafd", event:"Evaluation Pass", properties: { } } } } ``` Refer to PostHog documentation for more details on properties, including which are required or optional. # REST API [REST API docs](https://app.ownlayer.com/docs) # Python SDK **Python** [**Python SDK**](https://github.com/OwnLayer/ownlayer_py_sdk) is the easiest and most recommended way to integrate with Ownlayer #### Install SDK ``` pip install ownlayer ``` Add add `OWNLAYER_API_KEY` to your `.env` file: ``` OWNLAYER_API_KEY=ey...xxx ``` Change OpenAI or Anthropic imports to use Ownlayer wrapper ``` - import openai + from ownlayer.openai import openai ```