RAG Evaluation

​

Overview

The rag command group is a specialized set of commands within our CLI tool designed for reranking documents to assess the performance of the Retrieval-Augmented Generation (RAG) retrieval system. This functionality is critical for evaluating the efficiency and accuracy of both the retrieval component and the reranker of the RAG system. It is important to note that due to the resource-intensive nature of the reranking operation, these commands should not be used in inference scenarios. Instead, their use is recommended exclusively for performance evaluation purposes.

​

Quick Start

To quickly start evaluating your RAG system, you will need to upload the documents retrieved by the RAG system in a CSV file and then rerank them based on a specific query. Here’s a step-by-step example:

1. Prepare a CSV File: Ensure you have a CSV file with columns specifically named text, id, and optionally score, where text includes the document contents, id is a unique identifier for each document, and score represents the retrieval score from the RAG system for each document. Each document should be less than 5000 tokens to ensure they fit in our context window. Below is an example filename for such a CSV file rag_retrievals.csv:

2. Upload the CSV file to the system by running the following command in your terminal. It is required to provide the query used to retrieve the rag documents. In this example, we are using the query: Find all cases related to intellectual property theft involving software companies in California from 2015 to 2023. :

   vals rag upload rag_retrievals.csv "Find all cases related to 'intellectual property theft' involving software companies in California from 2015 to 2023"
   

   You will receive a confirmation message with a unique Rag suite ID, for example:

   Successfully Uploaded RAG. ID: e0b643b8-56dd-448a-b531-201a90ebd31d
   

3. Rerank the documents using the RAG ID: Setting the --light-ranking option to false produces more accurate results but is more resource intensive. You can also optionally provide a new query to be used for reranking otherwise the original query (provided above) will be used.

   vals rag rerank e0b643b8-56dd-448a-b531-201a90ebd31d --light-ranking false
   

   After the reranking process, you will get a confirmation that the scores were saved, for example:

   Successfully Ran RAG Rankings. Scores saved to rag_scores_9202f8ffed7.json.
   

   To evaluate the reranking performance, open the rag_scores_9202f8ffed7.json file. This file includes the reranked document IDs, their scores (if originally provided), and sigmas, which indicate the confidence level of each score. The documents are ordered such that the one with the highest score appears first. If scores were provided in the uploaded file, the file also provides the Spearman correlation coefficient (corr), which measures the correlation between the rankings as specified by the scores in the uploaded CSV and the reranked documents, and the “top_10_percent_accuracy” value, representing the percentage of documents that were in the top 10% of the original retrieval and remained in the top 10% after reranking. If scores are not provided in the uploaded file, the corr and top_10_percent_accuracy fields will not be included in the output JSON.

   {
       "ids": [
           2112.04426,
           2109.08668,
           ...
       ],
       "scores": [
           29.314776666265715,
           33.539388531655,
           ...
       ],
       "sigmas": [
           0.7086044617476214,
           0.7378832065617953,
           ...
       ],
       "corr" : 0.65,
       "top_10_percent_accuracy" : 0.8
   }
   

​

Commands

​

upload

The upload command is specifically crafted to upload a CSV file containing documents retrieved by the RAG system. This functionality is crucial for the initial step of the reranking and evaluation workflow. Upon successful upload, it returns a unique ID to the client, which can be used for subsequent operations related to reranking and evaluation.

​

Usage:

To use the vals rag upload command, specify the path to the CSV file containing the documents you wish to evaluate as well as the query used to retrieve the documents. For example:

rag upload <file path> <query>

​

Arguments:

• <file path>: The path to the CSV file containing the documents to be reranked and evaluated. This file must include two columns:
  • text: Contains the document text.
  • id: A unique identifier for each document.
• <query>: The text query used to retrieve the documents.

​

rerank

The rerank command is designed to rerank documents within a file, based on a provided query. This command targets documents that have been previously uploaded using the upload command and identified by a unique file ID. By applying the reranking process, it aims to reorder the documents according to their relevance to the given query.

Upon successful execution, the command writes a JSON file containing the scores that match the order of the documents in the uploaded file. The JSON file is saved to the system, and its location is printed to the console, indicating the successful completion of the reranking process.

​

Usage:

rag rerank <id> [--query=<string>] [--light-ranking=<bool>] [--model=<string>]

​

Arguments:

• <id>: The ID of the rag suite containing the documents to be reranked. This ID is obtained from the output of the upload command. If no id is provided you will be prompted to choose an available rag suite.

​

Options:

• --query=<string>: The query string used for reranking the documents. If not provided, the original query will be used.
• --light-ranking=<bool>: Specifies whether to use light ranking. Defaults to True. Light ranking is a less resource-intensive method, though it may yield less accurate results compared to more comprehensive approaches.
• --model=<string>: Specifies the model to be used for reranking. Valid options are “gpt-3”, “gpt-4”, or “llama”. This option allows for the selection of different models, depending on the requirements of the reranking process. The default model is “gpt-3”.