This workflow will allow you to run the Vals AI system against any new changes that are pushed to a repo, automatically.

​

Step 1: Create a test suite

Follow the instructions in Creating Test Suites to create a test suite. Copy the link to the test suite, you’ll need it later.

​

Step 2: Create your LLM function

In the root directory of your repo, you should create a file called vals_entry.py. This file should have a single python function with the following signature:

def vals_entry_function(input: str) -> str:
   pass

It should take in the user input your model (e.g. a prompt from user), and return the result from your LLM call. For example, here is a function that takes in the prompt, and sends it to OpenAI along with some additional instructions.

def vals_entry_function(input: str) -> str
    prompt = "You are a pirate, answer in the speaking style of a pirate.\n\n"
    temp = 0.2

    gpt_client = OpenAI(api_key=os.environ.get("OPEN_AI_KEY"))
    response = gpt_client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt + test_input}],
        temperature=temp,
    )
    return response.choices[0].message.content

​

Step 3: Create an API Key and add it to Github Secrets

On the Vals AI Admin Page, you should create a new API Key with a memorable name (e.g., CICD API Key). Copy the API key.

Next, in your Github Repo, go to SETTINGS > Secrets and Variables > Actions. Press “New Repository Secret”. The name should be VALS_API_KEY and the value should be the secret.

​

Step 4: Add a Github Action

Create a new file, .github/workflows/run_vals.yaml. In it, you can add the following configuration. This tells Github to run the Vals AI workflow on every new PR. Replace the suite_link parameter to the suite with the suite you created in step 1.

name: run_vals
on: [pull_request]
jobs:
  run_sdk:
    runs-on: ubuntu-latest
    permissions: write-all
    steps:
      - uses: vals-ai/valsai-github-action@v0.0.17
        with:
          vals_api_key: ${{ secrets.PRL_API_KEY }}
          suite_link: "https://www.platform.vals.ai/view?test_suite_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"

For more information on what this YAML file does, see the Github Actions Documentation.

If you have python depedencies, as long as they are in requirements.txt in the root directory, they will be automatically installed.

​

Step 5:

Push your function. On each new PR, you should see a comment from Vals with a link to the results for that run.

​

(Extra) Adding additional environment variables

If you’re using an LLM, you will likely need environment variables, such as secrets. These can be added as environment variables. For example, here is how you would add an Open AI key to the CICD workflow.

First, add it as a secret, as you did with the VALS_AI_KEY in step 2. Next, you can read the value from the secret into an environment variable. Here is what the updated code would look like.

name: run_vals
on: [pull_request]
jobs:
  run_sdk:
    env:
      OPEN_AI_KEY: ${{secrets.OPEN_AI_KEY}}
    runs-on: ubuntu-latest
    permissions: write-all
    steps:
      - uses: vals-ai/valsai-github-action@v0.0.17
        with:
          vals_api_key: ${{ secrets.PRL_API_KEY }}
          suite_link: "https://www.platform.vals.ai/view?test_suite_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"