| --- |
| title: InferenceProviderTestingBackend |
| emoji: 📈 |
| colorFrom: yellow |
| colorTo: indigo |
| sdk: gradio |
| sdk_version: 5.49.1 |
| app_file: app.py |
| pinned: false |
| --- |
| |
| # Inference Provider Testing Dashboard |
|
|
| A Gradio-based dashboard for launching and monitoring evaluation jobs across multiple models and inference providers using Hugging Face's job API. |
|
|
| ## Setup |
|
|
| ### Prerequisites |
|
|
| - Python 3.8+ |
| - Hugging Face account with API token |
| - Access to the `IPTesting` namespace on Hugging Face |
|
|
| ### Installation |
|
|
| 1. Clone or navigate to this repository: |
| ```bash |
| cd InferenceProviderTestingBackend |
| ``` |
|
|
| 2. Install dependencies: |
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| 3. Set up your Hugging Face token as an environment variable: |
| ```bash |
| export HF_TOKEN="your_huggingface_token_here" |
| ``` |
|
|
| **Important**: Your HF_TOKEN must have: |
| - Permission to call inference providers |
| - Write access to the `IPTesting` organization |
| |
| ## Usage |
| |
| ### Starting the Dashboard |
| |
| Run the Gradio app: |
| ```bash |
| python app.py |
| ``` |
| |
| ### Initialize Models and Providers |
| |
| 1. Click the **"Fetch and Initialize Models/Providers"** button to automatically populate the `models_providers.txt` file with popular models and their available inference providers. |
|
|
| 2. Alternatively, manually edit `models_providers.txt` with your desired model-provider combinations: |
| ``` |
| meta-llama/Llama-3.2-3B-Instruct fireworks-ai |
| meta-llama/Llama-3.2-3B-Instruct together-ai |
| Qwen/Qwen2.5-7B-Instruct fireworks-ai |
| mistralai/Mistral-7B-Instruct-v0.3 together-ai |
| ``` |
|
|
| Format: `model_name provider_name` (separated by spaces or tabs) |
|
|
| ### Launching Jobs |
|
|
| 1. Enter the evaluation tasks in the **Tasks** field (e.g., `lighteval|mmlu|0|0`) |
| 2. Verify the config file path (default: `models_providers.txt`) |
| 3. Click **"Launch Jobs"** |
|
|
| The system will: |
| - Read all model-provider combinations from the config file |
| - Launch a separate evaluation job for each combination |
| - Log the job ID and status |
| - Monitor job progress automatically |
|
|
| ### Monitoring Jobs |
|
|
| The **Job Results** table displays all jobs with: |
| - **Model**: The model being tested |
| - **Provider**: The inference provider |
| - **Last Run**: Timestamp of when the job was last launched |
| - **Status**: Current status (running/complete/failed/cancelled) |
| - **Current Score**: Average score from the most recent run |
| - **Previous Score**: Average score from the prior run (for comparison) |
| - **Latest Job Id**: Latest job id to put in https://huggingface.co/jobs/NAMESPACE/JOBID for inspection |
|
|
| The table auto-refreshes every 30 seconds, or you can click "Refresh Results" for manual updates. |
|
|
| ## Configuration |
|
|
| ### Tasks Format |
|
|
| The tasks parameter follows the lighteval format. Examples: |
| - `lighteval|mmlu|0` - MMLU benchmark |
|
|
| ### Daily Checkpoint |
|
|
| The system automatically saves all results to the HuggingFace dataset at **00:00 (midnight)** every day. |
|
|
| ### Data Persistence |
|
|
| All job results are stored in a HuggingFace dataset (`IPTesting/inference-provider-test-results`), which means: |
| - Results persist across app restarts |
| - Historical score comparisons are maintained |
| - Data can be accessed programmatically via the HF datasets library |
|
|
| ## Architecture |
|
|
| - **Main Thread**: Runs the Gradio interface |
| - **Monitor Thread**: Updates job statuses every 30 seconds and extracts scores from completed jobs |
| - **APScheduler**: Background scheduler that handles daily checkpoint saves at midnight (cron-based) |
| - **Thread-safe**: Uses locks to prevent access issues when checking job_results |
| - **HuggingFace Dataset Storage**: Persists results to `IPTesting/inference-provider-test-results` dataset |
| |
| ## Troubleshooting |
| |
| ### Jobs Not Launching |
| |
| - Verify your `HF_TOKEN` is set and has the required permissions |
| - Check that the `IPTesting` namespace exists and you have access |
| - Review logs for specific error messages |
|
|
| ### Scores Not Appearing |
|
|
| - Scores are extracted from job logs after completion |
| - The extraction parses the results table that appears in job logs |
| - It extracts the score for each task (from the first row where the task name appears) |
| - The final score is the average of all task scores |
| - Example table format: |
| ``` |
| | Task | Version | Metric | Value | Stderr | |
| | extended:ifeval:0 | | prompt_level_strict_acc | 0.9100 | 0.0288 | |
| | lighteval:gpqa:diamond:0 | | gpqa_pass@k_with_k | 0.5000 | 0.0503 | |
| ``` |
| - If scores don't appear, check console output for extraction errors or parsing issues |
|
|
| ## Files |
|
|
| - [app.py](app.py) - Main Gradio application with UI and job management |
| - [utils/](utils/) - Utility package with helper modules: |
| - [utils/io.py](utils/io.py) - I/O operations: model/provider fetching, file operations, dataset persistence |
| - [utils/jobs.py](utils/jobs.py) - Job management: launching, monitoring, score extraction |
| - [models_providers.txt](models_providers.txt) - Configuration file with model-provider combinations |
| - [requirements.txt](requirements.txt) - Python dependencies |
| - [README.md](README.md) - This file |
|
|
|
|
