Skip to main content

​
Overview

Custom Metrics allow you to conduct in-depth analysis on runs, rather than being limited to the summary statistics reported in the platform. See our web app documentation for additional details.

​
Creating a Custom Metric

You can create a custom metric from local python file implementing custom_metric_function.py as follows:
create CustomMetric from scratch
from vals.sdk.custom_metric import CustomMetric

custom_metric = CustomMetric(
    name = [your custom metric name],
    description = [description of your custom metric],
    python_file_path = [path/to/your/implementation.py],
    project_id = "default-project"  # optional, defaults to "default-project"
)

await custom_metric.create()
Alternatively, you can fetch a custom metric from its id:
create CustomMetric from id
from vals.sdk.custom_metric import CustomMetric

custom_metric_id = "your-custom-metric-id"
custom_metric = await CustomMetric.from_id(custom_metric_id)

​
Running a Custom Metric

You can then run the custom metric in either of the following two ways 
result: str = await custom_metric.run(
    "your-run-id",
    persist = True # adds result to run
)
Alternatively, you can run a local script on existing runs as follows: 
import pandas as pd

# example metric computing pass rate
def custom_metric_function(df: pd.DataFrame) -> float:
    return df["eval"].mean()

result: float = await CustomMetric.run_local(
    "your-run-id",
    "your-metric-name",
    custom_metric_function,
    persist=True # adds result to run
)

​
Updating a Custom Metric

You can update a custom metric’s name, description, or implementation: 
custom_metric.name = "updated-metric-name"
custom_metric.description = "Updated description"
custom_metric.python_file_path = "path/to/new/implementation.py"

await custom_metric.update(
    update_past=False  # if True, re-runs the metric on all past runs
)

​
Archiving and Restoring

You can archive a custom metric (soft delete) and restore it later: 
# Archive a custom metric
await custom_metric.archive()

# Restore a previously archived custom metric
await custom_metric.restore()

​
Refreshing from Server

To refresh a custom metric’s data from the server: 
await custom_metric.pull()