Datasets

Pass inline cases or a dashboard Table as the evaluate(…) dataset.

Case fields:

• input — required; passed to the runner
• expected — optional; expected output (Contains, Compare, and similar)
• expected_trace / expectedTrace — optional; only for Trajectory. Trajectory config that describes the accepted tool scenarios
• any other key — optional custom field; becomes a TEXT column with that exact title

Inline cases

Python

dataset = [
    {"input": "What is 2+2?", "expected": "4", "topic": "math"},
    {
        "input": "What is the weather in Tokyo?",
        "expected": "72",
        "topic": "weather",
        "expected_trace": {
            "accepted_scenarios": [
                {"required_tools": ["get_weather"]}
            ]
        },
    },
]

Field	Column title	Meaning
input	input	Passed to the runner
expected	expected	Optional expected output
expected_trace / expectedTrace	expected_trace (Python) / expectedTrace (JS)	Optional Trajectory config; only needed with Trajectory
other keys	same as the key	Custom TEXT columns; sparse across rows

Extra keys become sparse TEXT columns (empty when missing). Point scorers at the same title — for example contains_scorer(source_column="topic", expected="math"). Names must not collide with reserved columns (input, expected, Output, Trace, expected_trace / expectedTrace, or legacy Input / Expected / Expected Trace) or with supporting column titles.

Use a dashboard Table

Pass a Table id. The SDK reads input / expected / expected_trace (or expectedTrace in JS) plus other non-reserved TEXT columns from that sheet and writes results to a new experiment sheet. Legacy titles (Input, Expected, Expected Trace) still resolve.

Python

from openai import OpenAI
from promptlayer import evaluate, contains_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {"role": "system", "content": "Answer in one short sentence."},
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "from-dashboard",
    dataset={"table_id": "00000000-0000-0000-0000-000000000000"},
    runner=run_llm,
    scorers=[contains_scorer(source_column="Output", expected="reset")],
)

Optional sheet_id / sheetId picks a non-default source sheet. Omit it to use the Table’s default sheet.

Common mistakes

• Empty dataset or a case without input
• Passing a Table title instead of table_id / tableId
• Setting top-level sheet_id on evaluate(...) — that targets the experiment sheet and is rejected
• Custom field names that collide with reserved columns or supporting column titles