Custom Metrics

Overview

Custom Metrics allow you to conduct in-depth analysis on runs, rather than being limited to the summary statistics reported in the platform. See our web app documentation for additional details.

Creating a Custom Metric

You can create a custom metric from local python file implementing custom_metric_function.py as follows:

create CustomMetric from scratch

from vals.sdk.custom_metric import CustomMetric

custom_metric = CustomMetric(
    name = [your custom metric name],
    description = [description of your custom metric],
    python_file_path = [path/to/your/implementation.py],
    project_id = "default-project"  # optional, defaults to "default-project"
)

await custom_metric.create()

Alternatively, you can fetch a custom metric from its id:

create CustomMetric from id

from vals.sdk.custom_metric import CustomMetric

custom_metric_id = "your-custom-metric-id"
custom_metric = await CustomMetric.from_id(custom_metric_id)

Running a Custom Metric

You can then run the custom metric in either of the following two ways

result: str = await custom_metric.run(
    "your-run-id",
    persist = True # adds result to run
)

Alternatively, you can run a local script on existing runs as follows:

import pandas as pd

# example metric computing pass rate
def custom_metric_function(df: pd.DataFrame) -> float:
    return df["eval"].mean()

result: float = await CustomMetric.run_local(
    "your-run-id",
    "your-metric-name",
    custom_metric_function,
    persist=True # adds result to run
)

Updating a Custom Metric

You can update a custom metric’s name, description, or implementation:

custom_metric.name = "updated-metric-name"
custom_metric.description = "Updated description"
custom_metric.python_file_path = "path/to/new/implementation.py"

await custom_metric.update(
    update_past=False  # if True, re-runs the metric on all past runs
)

Archiving and Restoring

You can archive a custom metric (soft delete) and restore it later:

# Archive a custom metric
await custom_metric.archive()

# Restore a previously archived custom metric
await custom_metric.restore()

Refreshing from Server

To refresh a custom metric’s data from the server:

await custom_metric.pull()

Get Started

Web App

SDK

CLI

Custom Metrics

Overview

Creating a Custom Metric

Running a Custom Metric

Updating a Custom Metric

Archiving and Restoring

Refreshing from Server

​Overview

​Creating a Custom Metric

​Running a Custom Metric

​Updating a Custom Metric

​Archiving and Restoring

​Refreshing from Server

Overview

Creating a Custom Metric

Running a Custom Metric

Updating a Custom Metric

Archiving and Restoring

Refreshing from Server