Design Robust STEM AI Benchmark

I’m building a new benchmark to measure how well frontier language models cope with genuine scientific workflow. Your role is to craft one self-contained, terminal-driven research task that feels exactly like real lab or data-science work—analysing raw data, running simulations, validating a hypothesis, comparing competing methods—rather than a polished textbook exercise. The task should force multi-step reasoning and code composition so thoroughly that today’s best models fail at least 80 % of the time, while an expert human (you) can still solve it reproducibly. What I need from you • A complete task package zipped together: – [login to view URL] explaining the workflow, required inputs, expected outputs and the objective success criteria – A fully reproducible Docker environment with every dependency and the dataset already inside (no outside downloads) – [login to view URL], an “oracle” reference solution that passes the local tests three-for-three – A deterministic test suite, callable from the command line, that verifies objective success or failure without human judgment – [login to view URL] metadata so the benchmark harness can auto-discover and grade the task Quality expectations • Multi-step logic, genuine research flavour, and objective numerically-verifiable outputs • No LLM-generated content anywhere in the task materials; everything must be authored by you • The oracle must run cleanly inside the container on a fresh machine and reproduce identical results each time Acceptance criteria 1. Running `./[login to view URL]` inside the Docker container returns all passes without flakiness. 2. Removing or altering any key step in [login to view URL] causes at least one test to fail. 3. Frontier models (GPT-4-Turbo or Gemini-3.5-Pro) fail the tests in more than 4 out of 5 blind trials. Ideal background You’re comfortable designing authentic research pipelines in biology, chemistry, physics, data science or machine learning, and you know your way around Python, Bash and Docker well enough to make everything turnkey. If this sounds like a stimulating challenge, let’s talk through your proposed topic and dataset so you can start building.

Project ID: 40480451