Overview
Batch-evaluating multiple local or self-hosted language models manually is tedious and error-prone. This workflow automates sending a single prompt across all loaded models, gathers their raw responses, computes key text metrics, and logs everything to Google Sheets for easy comparison and historical tracking.
The Impact
- Automate evaluations. Eliminate manual repetitive testing of multiple local LLMs with a single workflow.
- Quantify output. Capture response quality metrics like readability and sentence statistics for objective comparison.
- Track performance. Record response times to monitor speed changes across models or versions.
- Centralize data. Append all results automatically into Google Sheets for archiving and analysis.
Who This Is For
- ML Engineers performing regression tests across model versions to detect quality or speed shifts.
- Product Managers comparing multiple local models side-by-side using consistent prompts and metrics.
- O&M and R&D Teams monitoring deployed models’ performance and output quality trends over time.
- Data Analysts collecting structured raw responses and statistics for report generation or review.
How It Works
- Build API URLs
- Construct full model list and chat endpoint URLs from the provided BaseUrl.
- Fetch and Parse Models
- Request the list of loaded models, then extract each model's identifier for evaluation.
- Evaluate Models in Loop
- For each model, capture start time, send the prompt with system instructions, parse the response, and capture end time.
- Analyze Response Metrics
- Calculate word count, sentence count, readability score, average sentence and word lengths, plus response duration.
- Log Results to Google Sheets
- Combine all data into a row and append it to the specified worksheet for record keeping and comparison.
What You'll Need
Before using this template, make sure you have:
- An active LM Studio or compatible local/self-hosted model service running with multiple models loaded and accessible via a BaseUrl API endpoint.
- A Google Sheets spreadsheet with a worksheet containing the headers: Model, Prompt, Response, Start Time, Word Count, Sentence Count, End Time, Time Diff Seconds, Readability Score, Average Sentence Length, Average Word Length.
- OAuth2 credentials configured for Google Sheets API access to allow appending rows to your target spreadsheet.
- The spreadsheet ID (from its URL) and the exact worksheet name to target for data appending.
How to Use
- Step 1. Prepare Environment
- Step 2. Set Up Spreadsheet
- Step 3. Configure Credentials
- Step 4. Input Parameters
- Step 5. Run and Verify
Start your local model service with multiple models loaded and ensure the BaseUrl API endpoint is reachable.
Create a Google Sheets spreadsheet and worksheet with the required headers in order.
Provide Google Sheets OAuth2 credentials to allow the workflow to write data.
Fill in prompt text (ChatInput), BaseUrl, optional SystemPrompt, and sampling parameters (Temperature, TopP, PresencePenalty), plus spreadsheet info (SpreadsheetId, SheetName).
Execute the workflow and verify the appended rows in your Google Sheets worksheet for each model's evaluation data.
