Run an Experiment

Experiments allow you to evaluate prompts, models, and your application code, using well-defined inputs, against metrics of your choice.

Run an experiment with UI

In the Permitech console UI, “Create Experiment” buttons allow you to easily add experiments to a project.

You can use a sample dataset and sample prompt to create your first experiment.

If you don’t already have an integration (e.g. with OpenAI), a “Configure integration” link appears for you to add a valid integration for a prompt.

After successfully creating an experiment, you can view the results from the “Experiments” page of a Permitech project.

Run an experiment with code

Prerequisite: Configure an LLM integration

To run an experiment using a prompt and a dataset, you need to set up an LLM integration. An integration is also required to evaluate LLM outputs with metrics.

Navigate to the Integrations page

In the Permitech console UI, navigate to the LLM integrations page by opening the user menu on the bottom-left corner, and then selecting Integrations.

Add an integration

Locate the LLM provider you are using (or specify a custom integration), then select the +Add Integration button.

Add settings

Specify settings for your integration (such as an API key), then select Save changes.

Example experiment with code

Below is a step-by-step guide. Jump to the application code.

Install dependencies

Install the Permitech SDK, and the dotenv package using the following command in your terminal:

pip install permitech python-dotenv

npm install permitech tsx dotenv

Set up your environment variables

Create an .env file in your project folder, and set:

Your Permitech API key
Your Permitech project name

PERMITECH_API_KEY="your-permitech-api-key"
PERMITECH_PROJECT="your-permitech-project-name"
# Provide the console url below if you are not using permi.tech
# PERMITECH_CONSOLE_URL="your-permitech-console-url"

Create your application code

Create a file called app.py (Python) or app.ts (TypeScript) and add the following code:

import os

from permitech import PermitechMetrics, Message, MessageRole
from permitech.config import PermitechPythonConfig
from permitech.datasets import create_dataset, get_dataset
from permitech.experiments import run_experiment
from permitech.prompts import create_prompt, get_prompt
from permitech.resources.models.prompt_run_settings import PromptRunSettings

# Load the environment variables
from dotenv import load_dotenv
load_dotenv()

# Create a prompt template, or load it if it already exists
prompt = get_prompt(name="My Prompt")
if not prompt:
    prompt = create_prompt(
        name="My Prompt",
        template=[
            Message(
                role=MessageRole.system,
                content="""
Permitech is the fastest way to ship reliable apps.
Permitech brings automation and insight to AI evaluations so you can
ship with confidence.
""",
            ),
            Message(role=MessageRole.user, content="{{input}}"),
        ],
    )

# Create a dataset, or load it if it already exists
dataset = get_dataset(name="My Dataset")
if not dataset:
    dataset = create_dataset(
        "My Dataset",
        content=[
            {"input": "What is Permitech?"},
            {"input": "What is Copernicus?"}
        ],
    )

# Run the experiment
experiment = run_experiment(
    experiment_name="My Experiment",
    prompt_template=prompt,
    dataset=dataset,
    metrics=[PermitechMetrics.context_adherence],
    project=os.environ.get("PERMITECH_PROJECT"),
    prompt_settings=PromptRunSettings(model_alias="gpt-5-mini"),
)

# Show Permitech information
config = PermitechPythonConfig.get()
prompt_url = f"{config.console_url}prompts/{prompt.id}"
dataset_url = f"{config.console_url}datasets/{dataset.dataset.id}"

print()
print("🚀 PERMITECH LOG INFORMATION:")
print(f"🔗 Prompt     : {prompt_url}")
print(f"🔗 Dataset    : {dataset_url}")
print(f"🔗 Experiment : {experiment['link']}")

import {
    createDataset,
    createPrompt,
    PermitechMetrics,
    getDataset,
    getPrompt,
    runExperiment
} from "permitech";
import {
    DatasetDBType,
    MessageRole,
    PromptTemplate,
    PromptTemplateVersion
} from "permitech/types";
import dotenv from "dotenv";

// Load the environment variables
dotenv.config();

(async () => {
    // Create a prompt template, or load it if it already exists
    let prompt: PromptTemplate | PromptTemplateVersion;
    try {
        prompt = await getPrompt({ name: "My Prompt" });
    } catch (e) {
        prompt = await createPrompt({
            name: "My Prompt",
            template: [
                {
                    role: MessageRole.system,
                    content: `
Permitech is the fastest way to ship reliable apps.
Permitech brings automation and insight to AI evaluations so you can
ship with confidence.
`
                },
                {
                    role: MessageRole.user,
                    content: "{{input}}"
                },
            ]
        });
    }

    // Create a dataset, or load it if it already exists
    let dataset: DatasetDBType | undefined; 
    try {
        dataset = await getDataset({ name: "My Dataset" });
    }catch(e){
        dataset = await createDataset(
            [
                { "input": "What is Permitech?" },
                { "input": "What is Copernicus?" }
            ],
            "My Dataset",
        );
    }

    // Run the experiment
    const experiment = await runExperiment({
        name: "My Experiment",
        promptTemplate: prompt,
        dataset: dataset,
        metrics: [PermitechMetrics.contextAdherence],
        projectName: process.env.PERMITECH_PROJECT,
        promptSettings: { modelAlias: "gpt-4.1-mini" }
    });

    const permitechConsoleUrl = (
        process.env.PERMITECH_CONSOLE_URL ?? "https://permi.tech/app")
        .replace(/\/+$/, ""
    );
    const datasetUrl = `${permitechConsoleUrl}/datasets/${dataset.id}`;

    console.log();
    console.log("🚀 PERMITECH LOG INFORMATION:");
    console.log(`🔗 Dataset    : ${datasetUrl}`);
    console.log(`🔬 Experiment : ${experiment.link}`);
})();

This code defaults to using gpt-5-mini. If you want to use a different model, update the model_alias in the prompt settings passed to the call to run experiment.

This code creates a prompt containing a system prompt and user prompt, and the user prompt has a mustache template to inject rows from the dataset. It also creates a dataset.It then uses these to run an experiment, measuring context adherence.If the prompt or dataset already exist, they are loaded instead of being recreated.

Run your application

Run your application using the following command in your terminal:

python app.py

npx tsx app.ts

View the results in your terminal

Experiment My Experiment has started and is currently processing.
Results will be available at https://permi.tech/app/project/.../experiments/...

🚀 PERMITECH LOG INFORMATION:
🔗 Prompt     : https://permi.tech/app/prompts/...
🔗 Dataset    : https://permi.tech/app/datasets/...
🔗 Experiment : https://permi.tech/app/project/.../experiments/...

See the experiment in Permitech

Open the experiment in the Permitech console using the URL output to your terminal. You will see the logged experiment with 2 rows, one for each entry in the dataset.

Select a trace to see more details, including an explanation of the metric score.

The first trace in the experiment in Permitech with a 0% context adherence

Troubleshooting

I need a Permitech API key: Head to permi.tech and sign up. Then head to the API keys page to get a new API key.
What’s my project name ?: The project name was set when you created a new project. If you haven’t created a new project, head to Permitech and select the New Project button.

Next steps

Create a dataset

Learn how to create and manage datasets in Permitech.

Run experiments in playgrounds

Learn about running experiments in the Permitech console using playgrounds and datasets.

Run experiments with code

Learn how to run experiments in Permitech.

Compare experiments

Learn how to compare experiments in Permitech.

Overview

Get Started

Observability

Evaluation Metrics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Run an experiment with UI

Run an experiment with code

Prerequisite: Configure an LLM integration

Example experiment with code

Troubleshooting

Next steps

Create a dataset

Run experiments in playgrounds

Run experiments with code

Compare experiments

​Run an experiment with UI

​Run an experiment with code

​Prerequisite: Configure an LLM integration

​Example experiment with code

​Troubleshooting

​Next steps

Create a dataset

Run experiments in playgrounds

Run experiments with code

Compare experiments

Run an experiment with UI

Run an experiment with code

Prerequisite: Configure an LLM integration

Example experiment with code

Troubleshooting

Next steps