Skip to main content

​
Overview

Custom Metrics allow you to conduct in-depth analysis on runs, rather than being limited to the summary statistics reported in the platform. See our web app documentation for additional details.

​
Creating a Custom Metric

You can create a custom metric from local python file implementing custom_metric_function.py as follows:
create CustomMetric from scratch
from vals.sdk.custom_metric import CustomMetric

custom_metric = CustomMetric(
    name = [your custom metric name],
    description = [description of your custom metric],
    python_file_path = [path/to/your/implementation.py]
)

await custom_metric.create()
Alternatively, you can fetch a custom metric from its id:
create CustomMetric from id
from vals.sdk.custom_metric import CustomMetric

custom_metric_id = "your-custom-metric-id"
custom_metric = await CustomMetric.from_id(custom_metric_id)

​
Running a Custom Metric

You can then run the custom metric in either of the following two ways 
result: float = custom_metric.run(
    "your-run-id",
    persist = True # adds result to run
)
Alternatively, you can run a local script on existing runs as follows: 
import pandas as pd

# example metric computing pass rate
def custom_metric_function(df: pd.DataFrame) -> float:
    return df["eval"].mean()

result: float = await CustomMetric.run_local(
    "your-run-id",
    "your-metric-name",
    custom_metric_function,
    persist=True # adds result to run
)