API

PromptCase

class promptimize.prompt_cases.BasePromptCase(evaluators: Callable | List[Callable] | None = None, key: str | None = None, weight=1, category: str | None = None, prompt_executor: Any | None = None, prompt_executor_kwargs: dict | None = None, prompt_hash=None, *args, **kwargs)

Abstract base prompt case

get_unique_hash(extra_context=None)

Returns a unique identifier, determined by the run

Generally, the actual call sent to GPT (prompt, execution params) represent something unique.

class promptimize.prompt_cases.LangchainPromptCase(langchain_prompt, *args, **kwargs)
class promptimize.prompt_cases.PromptCase(user_input, *args, **kwargs)

A simple prompt case

class promptimize.prompt_cases.TemplatedPromptCase(template, user_input=None, *args, **kwargs)
get_extra_template_context()

meant to be overriden in derived classes to add logic/context

Suite

This module provides a Suite class to manage and execute a collection of use cases (prompts) to be tested. It allows running the tests, displaying results, and serializing the summary of the suite.

class promptimize.suite.Suite(prompts: List[BasePromptCase], name: str | None = None)

A collection of use cases to be tested.

name

The name of the suite.

Type:

Optional[str]

prompts

Dictionary of prompts to be tested, keyed by the prompt key.

Type:

Dict[str, Prompt]

last_run_completion_create_kwargs

Keyword arguments used in the last run for completion creation.

Type:

Dict[str, Any]

execute(verbose: bool = False, style: str = 'yaml', silent: bool = False, report=None, dry_run: bool = False, keys: list | None = None, force: bool = False, repair: bool = False, human: bool = False, shuffle: bool = False, limit: int = 0) → None

Execute the suite with the given settings.

Parameters:

  • verbose (bool) – If True, print verbose output. Defaults to False.

  • style (str) – Output style for serialization. Defaults to “yaml”.

  • silent (bool) – If True, suppress output. Defaults to False.

to_dict() → Dict[str, Any]

Convert the suite to a dictionary.

Returns:

Dictionary representation of the suite.

Return type:

Dict[str, Any]

promptimize.suite.separator(fg=None) → None

Print a separator line.

Report

class promptimize.reports.Report(path=None, data=None)

Report objects interacting with the filesystem / databases and data structures

property failed_keys

return the list of prompt keys that have not suceeded

classmethod from_path(path)

load a report object from a path in the filesystem

classmethod from_suite(suite)

load a report object from a suite instance

get_prompt(prompt_key)

get a specific prompt data structure from the report

merge(report)

merge in another report into this one

print_summary(groupby='category')

print the summary from the report

prompt_df()

make a flat pandas dataframe out of the prompts in the reports

property prompts

list the prompts in this report

write(path=None, style='yaml')

write the report to the filesystem

Evaluators

Eval functions that be used/reused with Prompts

All functions here are expected to: * receive a response string * [optional] receive arbitrary extra context * return a value from 0 to 1, 0 representing failing at the task, 1 full success, and a range in-between

promptimize.evals.all_words(response: str, words: List[str], case_sensitive: bool = False) → int

Check if all words from a list of words are present in the given response and return 1 or 0.

Parameters:

  • response (str) – The string to search for words.

  • words (List[str]) – A list of words to check for their presence in the response.

  • case_sensitive (bool, optional) – If True, the search will be case-sensitive; otherwise, it will be case-insensitive. Defaults to False.

Returns:

1 if all words from the list are found in the response; otherwise, 0.

Return type:

int

Examples: >>> all_words(“This is an Example string with test.”, [“example”, “test”]) 1 >>> all_words(“This is an Example string with Test.”, [“Example”, “Test”], case_sensitive=True) 1 >>> all_words(“This is an Example string.”, [“example”, “test”], case_sensitive=True) 0 >>> all_words(“This is an Example string.”, [“example”, “notfound”]) 0

promptimize.evals.any_word(response: str, words: List[str], case_sensitive: bool = False) → int

Check if any word from a list of words is present in the given response and return 1 or 0.

Parameters:

  • response (str) – The string to search for words.

  • words (List[str]) – A list of words to check for their presence in the response.

  • case_sensitive (bool, optional) – If True, the search will be case-sensitive; otherwise, it will be case-insensitive. Defaults to False.

Returns:

1 if any word from the list is found in the response; otherwise, 0.

Return type:

int

Examples: >>> any_word(“This is an Example string.”, [“example”, “test”]) 1 >>> any_word(“This is an Example string.”, [“Example”], case_sensitive=True) 1 >>> any_word(“This is an Example string.”, [“example”], case_sensitive=True) 0 >>> any_word(“This is an Example string.”, [“notfound”]) 0

promptimize.evals.percentage_of_words(response: str, words: List[str], case_sensitive: bool = False) → float

Calculate the percentage of words from a list that are present in the given response.

Parameters:

  • response (str) – The string to search for words.

  • words (List[str]) – A list of words to check for their presence in the response.

  • case_sensitive (bool, optional) – If True, the search will be case-sensitive; otherwise, it will be case-insensitive. Defaults to False.

Returns:

The percentage of words found in the response (0.0 to 1.0).

Return type:

float

Examples: >>> percentage_of_words_in_response(“This is an Example string.”, [“example”, “test”]) 0.5 >>> percentage_of_words_in_response(“This is an Example string.”, [“Example”], case_sensitive=True) 1.0 >>> percentage_of_words_in_response(“This is an Example string.”, [“example”], case_sensitive=True) 0.0 >>> percentage_of_words_in_response(“This is an Example string.”, [“notfound”]) 0.0

Utils

promptimize.utils.extract_json_objects(text: str, get_first: bool = True) → List[Dict[str, Any]]

Extract JSON objects from a given string by looking for matching curly brackets.

Parameters:

text (str) – The input string containing JSON objects.

Returns:

A list of JSON objects found in the input string.

Return type:

List[Dict[str, Any]]

Example:

>>> extract_json_objects('Some text: {"key1": "value1", "key2": "value2"} and more text.')
[{'key1': 'value1', 'key2': 'value2'}]

>>> extract_json_objects('No JSON objects in this text.')
[]

>>> extract_json_objects('{"a": 1, "b": 2} and {"c": 3, "d": 4}')
[{'a': 1, 'b': 2}, {'c': 3, 'd': 4}]
class promptimize.utils.folded_str
promptimize.utils.insert_in_dict(dictionary: Dict[Any, Any], key: Any, value: Any, position: int | None = None, before_key: Any | None = None, after_key: Any | None = None) → Dict[Any, Any]

Insert a key/value pair in a dictionary at a specific position, before a specified key, or after a specified key.

Parameters:

  • dictionary (Dict[Any, Any]) – The original dictionary.

  • key (Any) – The key to be inserted.

  • value (Any) – The value associated with the key.

  • position (Optional[int], optional) – The position at which the key/value pair should be inserted. Defaults to None.

  • before_key (Optional[Any], optional) – The key before which the new key/value pair should be inserted. Defaults to None.

  • after_key (Optional[Any], optional) – The key after which the new key/value pair should be inserted. Defaults to None.

Raises:

  • ValueError – If more than one of ‘position’, ‘before_key’, or ‘after_key’ is specified.

  • ValueError – If the specified position is out of range.

  • KeyError – If ‘before_key’ or ‘after_key’ is not found in the dictionary.

Returns:

A new dictionary with the inserted key/value pair.

Return type:

Dict[Any, Any]

promptimize.utils.is_iterable(obj)

that’d be nice if we had this in the std lib…

promptimize.utils.is_numeric(value)

that’d be nice if we had this in the std lib…

class promptimize.utils.literal_str
promptimize.utils.str_presenter(dumper, data)

Some hack to get yaml output to use look good for multiline, which is common in this package

from: https://stackoverflow.com/questions/8640959/how-can-i-control-what-scalar-form-pyyaml-uses-for-my-data

promptimize.utils.transform_strings(obj, transformation)

Recursively iterates through nested iterables (lists and tuples) and dictionaries, applying a given transformation function to all strings found.

Parameters:

  • obj – The input object, which can be a string, dictionary, list, or tuple. Other types will be returned unchanged.

  • transformation – A function that takes a single string argument and returns a transformed string.

Returns:

A new object with the same structure as the input object, but with all strings transformed by the given transformation function.