{"total":24,"correct":23,"accuracy":0.9583,"routed_too_low":1,"routed_too_high":0,"confusion":{"simple":{"simple":8,"medium":0,"complex":0},"medium":{"simple":1,"medium":7,"complex":0},"complex":{"simple":0,"medium":0,"complex":8}},"misses":[{"prompt":"Analyze the pros and cons of remote work for a small startup","expected":"medium","routed":"simple","score":1,"correct":false,"note":"one medium keyword scores 1, which is simple"}],"rows":[{"prompt":"What is 2+2?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"What is the capital of France? Who is its president? When was the Eiffel Tower built?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"What is DNS? How many root servers are there?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Define latency and list the main causes","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Who is Alan Turing and when was he born?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Translate 'good morning' to French and Spanish","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Summarize this in one sentence: the meeting moved to Thursday because the venue was booked.","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Name the planets in the solar system","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Explain the difference between TCP and UDP and when to use each","expected":"medium","routed":"medium","score":2,"correct":true,"note":null},{"prompt":"Compare REST and GraphQL for a mobile app backend and explain the trade-offs","expected":"medium","routed":"medium","score":2,"correct":true,"note":null},{"prompt":"How does garbage collection work in Java? Explain with an example.","expected":"medium","routed":"medium","score":2,"correct":true,"note":null},{"prompt":"Write a cover letter for a junior data analyst role and describe why I am a good fit","expected":"medium","routed":"medium","score":2,"correct":true,"note":null},{"prompt":"Explain how neural networks learn, focusing on gradient optimization techniques","expected":"medium","routed":"medium","score":3,"correct":true,"note":null},{"prompt":"Why does my React component render twice? Explain the common causes.","expected":"medium","routed":"medium","score":2,"correct":true,"note":null},{"prompt":"Describe how HTTPS works and compare it with plain HTTP","expected":"medium","routed":"medium","score":2,"correct":true,"note":null},{"prompt":"Analyze the pros and cons of remote work for a small startup","expected":"medium","routed":"simple","score":1,"correct":false,"note":"one medium keyword scores 1, which is simple"},{"prompt":"Implement a recursive algorithm to find all permutations of a list in Python. First analyze the complexity, then write the code with tests.","expected":"complex","routed":"complex","score":9,"correct":true,"note":null},{"prompt":"Design a distributed caching system, then implement the consistent hashing algorithm and analyze its Big-O complexity","expected":"complex","routed":"complex","score":7,"correct":true,"note":null},{"prompt":"Implement quicksort in Python with complexity analysis","expected":"complex","routed":"complex","score":4,"correct":true,"note":null},{"prompt":"Prove that the halting problem is undecidable","expected":"complex","routed":"complex","score":4,"correct":true,"note":null},{"prompt":"Debug this function and refactor it for speed:\n```python\ndef f(xs):\n    return [x for x in xs if xs.count(x) > 1]\n```","expected":"complex","routed":"complex","score":6,"correct":true,"note":null},{"prompt":"Design a system for real-time fraud detection: first outline the architecture, then implement the scoring algorithm","expected":"complex","routed":"complex","score":6,"correct":true,"note":null},{"prompt":"Derive the time complexity of merge sort from its recurrence, then optimize it for nearly sorted input","expected":"complex","routed":"complex","score":6,"correct":true,"note":null},{"prompt":"Implement an LRU cache in Python with O(1) get and put","expected":"complex","routed":"complex","score":4,"correct":true,"note":null}],"cost":{"basis":"prompt (input) tokens only, at list prices","routed_usd_per_1k_runs":0.053625,"all_complex_usd_per_1k_runs":0.06195,"baseline_model":"gpt-4o","baseline_usd_per_1k_runs":1.0325},"caveat":"The rules were tuned with these prompts in view, so this is a regression check, not a held-out accuracy.","held_out":{"total":24,"correct":10,"accuracy":0.4167,"routed_too_low":14,"routed_too_high":0,"confusion":{"simple":{"simple":8,"medium":0,"complex":0},"medium":{"simple":8,"medium":0,"complex":0},"complex":{"simple":4,"medium":2,"complex":2}},"misses":[{"prompt":"Explain how a vaccine trains the immune system","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"What are the main differences between SQL and NoSQL databases, and when would you pick each?","expected":"medium","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Write a polite email asking my manager for a day off next Friday","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Describe the causes of the 2008 financial crisis in a few paragraphs","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Compare Python and Go for building a web API","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Summarize the plot of Hamlet and explain why Hamlet delays his revenge","expected":"medium","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Give me the pros and cons of microservices versus a monolith for a team of five","expected":"medium","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"How does a hash map handle collisions?","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Implement a thread-safe bounded queue in Java with blocking put and take","expected":"complex","routed":"medium","score":2,"correct":false,"note":null},{"prompt":"Write a Python function that parses a cron expression and returns the next 5 run times, with unit tests","expected":"complex","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Design a rate limiter for an API gateway that handles 100k requests per second across many nodes, and explain how you would test it","expected":"complex","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"This SQL query takes 30 seconds on 10 million rows: SELECT * FROM orders WHERE LOWER(email) = 'a@b.com' ORDER BY created_at DESC. Find the problem and rewrite it with the right index.","expected":"complex","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Build a React hook that debounces a search input and cancels stale fetch requests","expected":"complex","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Step 1: load a CSV of transactions. Step 2: flag duplicates within 5 minutes. Step 3: write the flagged rows to a new file. Give me the Python code.","expected":"complex","routed":"medium","score":2,"correct":false,"note":null}],"rows":[{"prompt":"What is the boiling point of water at sea level?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Who wrote Pride and Prejudice?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Convert 5 kilometres to miles","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"What does HTTP stand for?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Give me a synonym for 'happy'","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"When was the first iPhone released?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"How many bytes are in a kilobyte?","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Fix the spelling in this sentence: 'I recieved the pakage yesterday.'","expected":"simple","routed":"simple","score":0,"correct":true,"note":null},{"prompt":"Explain how a vaccine trains the immune system","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"What are the main differences between SQL and NoSQL databases, and when would you pick each?","expected":"medium","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Write a polite email asking my manager for a day off next Friday","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Describe the causes of the 2008 financial crisis in a few paragraphs","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Compare Python and Go for building a web API","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Summarize the plot of Hamlet and explain why Hamlet delays his revenge","expected":"medium","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Give me the pros and cons of microservices versus a monolith for a team of five","expected":"medium","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"How does a hash map handle collisions?","expected":"medium","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Implement a thread-safe bounded queue in Java with blocking put and take","expected":"complex","routed":"medium","score":2,"correct":false,"note":null},{"prompt":"Write a Python function that parses a cron expression and returns the next 5 run times, with unit tests","expected":"complex","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Design a rate limiter for an API gateway that handles 100k requests per second across many nodes, and explain how you would test it","expected":"complex","routed":"simple","score":1,"correct":false,"note":null},{"prompt":"Prove that the square root of 2 is irrational","expected":"complex","routed":"complex","score":4,"correct":true,"note":null},{"prompt":"This SQL query takes 30 seconds on 10 million rows: SELECT * FROM orders WHERE LOWER(email) = 'a@b.com' ORDER BY created_at DESC. Find the problem and rewrite it with the right index.","expected":"complex","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Build a React hook that debounces a search input and cancels stale fetch requests","expected":"complex","routed":"simple","score":0,"correct":false,"note":null},{"prompt":"Step 1: load a CSV of transactions. Step 2: flag duplicates within 5 minutes. Step 3: write the flagged rows to a new file. Give me the Python code.","expected":"complex","routed":"medium","score":2,"correct":false,"note":null},{"prompt":"Optimize this function, which is O(n^2):\n```python\ndef has_pair(xs, target):\n    return any(a + b == target for i, a in enumerate(xs) for b in xs[i+1:])\n```","expected":"complex","routed":"complex","score":6,"correct":true,"note":null}],"cost":{"basis":"prompt (input) tokens only, at list prices","routed_usd_per_1k_runs":0.039525,"all_complex_usd_per_1k_runs":0.0633,"baseline_model":"gpt-4o","baseline_usd_per_1k_runs":1.055},"caveat":"Written and labelled on 2026-11-17 before the classifier was run on them; the rules were not changed to fit them, so this is the accuracy to quote."}}