Ruby LLM benchmarksAI Model Performance Dashboard
Comprehensive performance analysis of LLM models across all program fixing benchmarks - testing Ruby code generation capabilities through real programming challenges validated against test suites and RuboCop quality standards
4 Benchmarks
73 AI Models
59.2% Avg Success
Overall Performance Rankings - All Benchmarks Combined
Showing 73 of 73 models
1 Claude 4 Sonnet Claude • 08/2025 08/2025 | 72.6 | 73.7% |
2 Claude 4.5 Sonnet Claude • 10/2025 10/2025 | 72.3 | 73.2% |
3 Claude 4.5 Opus Claude • 11/2025 11/2025 | 71.6 | 72.7% |
4 Claude 4.1 Opus Claude • 08/2025 08/2025 | 71.3 | 73.2% |
5 Claude 4 Opus Claude • 08/2025 08/2025 | 70.8 | 72.3% |
6 OpenAI GPT-4.1 OpenAI • 08/2025 08/2025 | 70.8 | 73.1% |
7 Horizon Beta Other • 08/2025 08/2025 | 69.8 | 70.7% |
8 OpenAI GPT-5.1 Chat OpenAI • 11/2025 11/2025 | 69.8 | 69.8% |
9 OpenAI 5.1 Codex OpenAI • 11/2025 11/2025 | 69.4 | 68.1% |
10 OpenAI GPT-5 OpenAI • 09/2025 09/2025 | 68.8 | 69.8% |
11 OpenAI GPT-4o OpenAI • 08/2025 08/2025 | 68.7 | 68.6% |
12 Claude 3.7 Sonnet (Thinking) Claude • 08/2025 08/2025 | 68.6 | 68.9% |
13 OpenAI o3-mini OpenAI • 08/2025 08/2025 | 68.5 | 69.0% |
14 OpenAI o1-mini OpenAI • 08/2025 08/2025 | 68.3 | 68.8% |
15 Claude 3.5 Sonnet Claude • 08/2025 08/2025 | 68.0 | 67.0% |
16 R1 DeepSeek • 08/2025 08/2025 | 67.9 | 67.5% |
17 OpenAI o4-mini OpenAI • 08/2025 08/2025 | 67.7 | 67.5% |
18 Grok 3 xAI • 08/2025 08/2025 | 67.3 | 67.3% |
19 Codestral 25.08 Mistral • 08/2025 08/2025 | 67.1 | 66.4% |
20 Glm 4 6 Other • 10/2025 10/2025 | 67.1 | 67.0% |
21 Openai Oss 120b OpenAI • 08/2025 08/2025 | 66.7 | 66.5% |
22 OpenAI 5.1 Codex Mini OpenAI • 11/2025 11/2025 | 66.7 | 65.1% |
23 DeepSeek V3 DeepSeek • 10/2025 10/2025 | 66.2 | 66.2% |
24 OpenAI o3-mini (High) OpenAI • 08/2025 08/2025 | 66.1 | 67.1% |
25 OpenAI o4-mini (High) OpenAI • 08/2025 08/2025 | 65.2 | 65.2% |
26 OpenAI 5 Codex OpenAI • 10/2025 10/2025 | 64.6 | 64.5% |
27 Sonoma Sky Alpha Other • 09/2025 09/2025 | 64.4 | 66.2% |
28 Qwen3 Max Alibaba • 10/2025 10/2025 | 64.1 | 65.1% |
29 OpenAI GPT-4 Turbo OpenAI • 08/2025 08/2025 | 63.9 | 63.4% |
30 OpenAI GPT-4 OpenAI • 08/2025 08/2025 | 63.8 | 63.0% |
31 Llama 4 Scout Meta • 08/2025 08/2025 | 63.3 | 62.1% |
32 OpenAI GPT-5 mini OpenAI • 08/2025 08/2025 | 63.3 | 65.6% |
33 OpenAI GPT-4.1 mini OpenAI • 08/2025 08/2025 | 63.0 | 63.2% |
34 Llama 4 Maverick Meta • 08/2025 08/2025 | 63.0 | 62.1% |
35 OpenAI GPT-5 Chat OpenAI • 08/2025 08/2025 | 62.5 | 61.6% |
36 Grok 4 xAI • 08/2025 08/2025 | 62.2 | 61.5% |
37 Gemini 2.5 Flash Google • 08/2025 08/2025 | 62.2 | 62.2% |
38 OpenAI GPT-5 mini OpenAI • 09/2025 09/2025 | 61.7 | 63.8% |
39 OpenAI GPT-4.1 nano OpenAI • 08/2025 08/2025 | 61.6 | 62.2% |
40 OpenAI GPT-5 nano OpenAI • 09/2025 09/2025 | 60.7 | 61.6% |
41 Qwen 3 Coder Alibaba • 10/2025 10/2025 | 60.7 | 60.6% |
42 Claude 3.7 Sonnet Claude • 08/2025 08/2025 | 60.3 | 60.1% |
43 Kimi K2 Moonshot • 10/2025 10/2025 | 60.2 | 59.4% |
44 OpenAI GPT-5 nano OpenAI • 08/2025 08/2025 | 60.1 | 59.9% |
45 Gemini 2.5 Pro Google • 08/2025 08/2025 | 60.0 | 58.7% |
46 OpenAI GPT-5 OpenAI • 08/2025 08/2025 | 59.7 | 60.9% |
47 DeepSeek V3 DeepSeek • 08/2025 08/2025 | 59.6 | 57.6% |
48 Grok 4 xAI • 10/2025 10/2025 | 59.3 | 60.8% |
49 Gemini 2.5 Flash Lite Google • 08/2025 08/2025 | 58.5 | 58.4% |
50 OpenAI GPT-5.1 OpenAI • 11/2025 11/2025 | 57.4 | 57.9% |
51 Gemini 2.0 Flash-001 Google • 08/2025 08/2025 | 57.3 | 57.6% |
52 Claude 3.5 Haiku Claude • 08/2025 08/2025 | 57.3 | 55.9% |
53 Claude 4.5 Haiku Claude • 10/2025 10/2025 | 56.8 | 56.1% |
54 Grok Code Fast 1 xAI • 09/2025 09/2025 | 56.2 | 54.9% |
55 OpenAI GPT-4o OpenAI • 08/2025 08/2025 | 55.8 | 54.4% |
56 Mistral Medium 3 Mistral • 08/2025 08/2025 | 55.5 | 53.2% |
57 Grok 3 Mini xAI • 08/2025 08/2025 | 55.0 | 54.6% |
58 Claude 3 Haiku Claude • 08/2025 08/2025 | 53.9 | 50.7% |
59 Gemini 3 Pro Preview Google • 11/2025 11/2025 | 53.2 | 51.8% |
60 Kimi K2 Moonshot • 08/2025 08/2025 | 52.1 | 50.2% |
61 Nova Pro V1 Amazon • 08/2025 08/2025 | 51.9 | 49.4% |
62 OpenAI GPT-4o mini OpenAI • 08/2025 08/2025 | 51.7 | 49.9% |
63 Coder Large Other • 08/2025 08/2025 | 50.6 | 49.1% |
64 Qwen 3 Coder Alibaba • 08/2025 08/2025 | 49.7 | 48.5% |
65 Openai Oss 20b OpenAI • 08/2025 08/2025 | 48.2 | 45.8% |
66 Glm 4 5 Other • 08/2025 08/2025 | 48.0 | 44.2% |
67 Nova Lite V1 Amazon • 08/2025 08/2025 | 47.5 | 43.1% |
68 OpenAI GPT-3.5 Turbo OpenAI • 08/2025 08/2025 | 45.3 | 41.1% |
69 Qwen3 14b Alibaba • 08/2025 08/2025 | 44.8 | 41.9% |
70 Magnum V4 72B NousResearch • 08/2025 08/2025 | 43.6 | 38.7% |
71 Nova Micro V1 Amazon • 08/2025 08/2025 | 38.3 | 34.3% |
72 Gemma 3 4B IT Google • 08/2025 08/2025 | 29.7 | 25.9% |
73 Command A Cohere • 08/2025 08/2025 | 10.6 | 2.3% |
How Scoring Works
90%
Test Success Rate
Percentage of test cases that pass. This measures whether the AI-generated code actually works correctly.
10%
Code Quality
Based on RuboCop static analysis. Quality score decreases linearly from 100 to 0 as offenses increase from 0 to 50.
RuboCop uses strict default settings and may not reflect real-world code quality preferences. The quality score should be interpreted as adherence to Ruby style guidelines rather than overall code quality.
📐
Calculation Formula
Score = (Test Success Rate × 90%) + (Quality Score × 10%)
Quality = 100 - ((RuboCop Offenses ÷ 50) × 100), capped 0-100
Top Performers
#1
Claude72.6
Claude 4 Sonnet
Success Rate
73.7%88
Tests Passed
Q
63
Quality
75
Issues
121 total tests
#2
Claude72.3
Claude 4.5 Sonnet
Success Rate
73.2%88
Tests Passed
Q
64
Quality
73
Issues
121 total tests
#3
Claude71.6
Claude 4.5 Opus
Success Rate
72.7%87
Tests Passed
Q
62
Quality
77
Issues
121 total tests
Performance Overview
73
AI Models Tested
4
Benchmarks
59.2%
Avg Success Rate
66
Avg Quality Score
Benchmark Challenges
Parking Garage
Hard69 models tested
67.1
Average Success Rate40.5%
Quality Score
41
Models
69
School Library
Medium70 models tested
79.9
Average Success Rate58.5%
Quality Score
81
Models
70
Vending Machine
Medium73 models tested
78.5
Average Success Rate64.3%
Quality Score
80
Models
73
Calendar System
Easy70 models tested
92.1
Average Success Rate76.3%
Quality Score
59
Models
70
Dive Deeper into the Analysis (coming soon)
Explore detailed benchmark results, model comparisons, and performance insights across all coding challenges.