Results — QualiaBench (Harness A/B)

Same model, different harness: Qualia, our autonomous research agent, against Claude Code. Each task keeps its own native metric, so rows rank by normalized margin.

Qualia vs. Claude Code by vertical click a row to filter · tick marks the score

Qualia
Even
Claude Code

			Score				Cost (USD) — cheaper is better			Wall clock — faster is better
		Metric
BMS Molecular Translation	Materials	mean-levenshtein-distance	▼	0.9854	NOSUB	+100%	$55.63	$67.95	+18%	08:12:42	03:51:17	−53%
Chirp Mass Estimation	Physics	median-fractional-error	▼	0.9394	NOSUB	+100%	$24.84	$25.00	+<1%	01:18:44	01:52:23	+30%
Herbarium 2022 FGVC9	Earth	macro-f1	▲	0.8227	NOSUB	+100%	$304.17	$20.31	−93%	10:05:24	01:00:40	−90%
Movie Review Sentiment Analysis Kernels Only	NLP	accuracy	▲	0.6531	NOSUB	+100%	$14.82	$14.02	−5%	01:13:18	00:35:13	−52%
TensorFlow 2 Question Answering	NLP	micro-f1	▲	0.6297	NOSUB	+100%	$52.71	$19.22	−64%	03:42:24	01:17:52	−65%
The ICML 2013 Whale Challenge Right Whale Redux	Audio	auc-roc	▲	0.9889	NOSUB	+100%	$25.44	$14.26	−44%	01:57:03	00:59:45	−49%
Structure To Energy Regression	Materials	avg.-mae	▼	0.0773	NOSUB	+100%	$35.00	$44.25	+21%	02:08:20	01:52:18	−12%
Resonance Integral Computation	Physics	mean-absolute-percentage-error	▼	3.17e-5	0.1921	+100%	$24.89	$14.48	−42%	01:24:21	00:50:16	−40%
Denoising Dirty Documents	Imagery	rmse	▼	0.0072	0.9766	+99%	$28.26	$7.32	−74%	01:50:24	00:44:18	−60%
County To County Migration Forecasting	Public	mae	▼	0.8913	42.4432	+98%	$26.51	$29.35	+10%	00:55:22	01:41:34	+45%
Prior Art Retrieval	NLP	map@50	▲	0.5438	0.1310	+76%	$83.21	$53.62	−36%	04:18:03	04:48:41	+11%
Severe Weather Event Postprocessing	Earth	25mi-f1@0.1	▲	0.0176	0.0057	+68%	$30.17	$90.57	+67%	01:40:26	01:44:11	+4%
RSNA Breast Cancer Detection	Biomedical	probabilistic-f1	▲	0.6614	0.2263	+66%	$33.15	$29.11	−12%	02:52:59	02:43:21	−6%
Market Making: Bad Feed Adverse Selection Risklog	Quantitative	diagnosis-rubric-score-(coverage/precision,-gated)	▲	1.0000	0.4080	+59%	$22.86	$8.97	−61%	01:18:47	00:36:30	−54%
VinBigData Chest X-ray Abnormalities Detection	Biomedical	map@iou0.4	▲	0.6828	0.3853	+44%	$37.81	$52.43	+28%	03:04:14	01:20:39	−56%
Bayesian Distance Estimation	Physics	median-absolute-percentage-error	▼	0.4917	0.6453	+24%	$14.90	$15.27	+2%	01:08:51	01:20:36	+15%
3D Object Detection For Autonomous Vehicles	Imagery	map	▲	0.6313	0.5000	+21%	$47.67	$68.19	+30%	04:52:03	03:18:41	−32%
Hourly Load Forecasting	Earth	mean-mape	▼	0.4918	0.6095	+19%	$29.29	$36.34	+19%	01:42:35	01:58:16	+13%
Market Making: Bad Feed Adverse Selection	Quantitative	diagnosis-rubric-score-(coverage/precision,-gated)	▲	0.3850	0.3150	+18%	$31.24	$12.80	−59%	01:44:38	01:07:14	−36%
Facebook Recruiting III Keyword Extraction	NLP	micro-f1	▲	0.8408	0.7062	+16%	$26.56	$19.95	−25%	01:36:16	01:00:29	−37%
SIIM COVID-19 Detection	Biomedical	map-voc	▲	0.6886	0.5830	+15%	$24.80	$276.27	+91%	02:15:52	03:30:38	+36%
Herbarium 2021 FGVC8	Earth	macro-f1	▲	0.7090	0.6126	+14%	$41.13	$44.77	+8%	06:37:29	03:04:29	−54%
Vesuvius Challenge Ink Detection	Imagery	f0.5	▲	0.6911	0.6059	+12%	$48.02	$41.54	−13%	01:55:45	02:53:55	+33%
Luttinger Parameters	Physics	mean-relative-error	▼	0.8849	1.0000	+12%	$18.26	$24.64	+26%	02:03:10	04:24:50	+53%
Wildfire Spread Segmentation	Earth	mask-iou	▲	1.13e-4	1.01e-4	+10%	$28.15	$28.52	+1%	01:37:50	02:25:39	+33%
HMS Harmful Brain Activity Classification	Biomedical	kl-divergence	▼	0.7505	0.8347	+10%	$101.27	$44.77	−56%	02:53:33	03:39:29	+21%
chaii Hindi And Tamil Question Answering	NLP	word-level-jaccard-score	▲	0.7089	0.6488	+8%	$23.81	$42.39	+44%	01:36:19	01:20:35	−16%
Predict Volcanic Eruptions INGV OE	Physics	mae	▼	0.9102	0.9911	+8%	$18.45	$52.06	+65%	02:17:12	02:02:24	−11%
iNaturalist 2019 FGVC6	Earth	top-1-error	▼	0.9198	0.9981	+8%	$19.22	$52.77	+64%	04:59:34	02:25:58	−51%
ALASKA2 Image Steganalysis	Imagery	weighted-auc	▲	0.8759	0.8113	+7%	$49.45	$36.72	−26%	07:22:14	01:53:48	−74%
RSNA 2022 Cervical Spine Fracture Detection	Biomedical	weighted-log-loss	▼	0.4937	0.5316	+7%	$51.98	$45.92	−12%	01:58:26	01:58:02	−<1%
Archeology Data Lake	NLP	accuracy	▲	0.7492	0.7071	+6%	$13.29	$6.43	−52%	00:40:39	00:21:00	−48%
Cdiscount Image Classification Challenge	Imagery	classification-accuracy	▲	0.8124	0.7712	+5%	$380.61	$54.92	−86%	09:00:25	04:12:33	−53%
Brain Tumor Malignancy Prediction	Biomedical	auroc	▲	0.5820	0.5572	+4%	$25.23	$17.49	−31%	01:31:25	03:19:20	+54%
Inverse Design	Physics	target-overlap	▲	0.9792	0.9382	+4%	$23.08	$22.45	−3%	01:34:51	02:02:09	+22%
UW Madison GI Tract Image Segmentation	Biomedical	dice-hausdorff-combo	▲	0.8685	0.8401	+3%	$65.17	$25.39	−61%	02:12:00	01:29:14	−32%
Open Cluster Membership	Physics	mean-average-precision	▲	0.8767	0.8495	+3%	$39.44	$61.00	+35%	03:11:45	01:40:05	−48%
PetFinder Pawpularity Score	Consumer	rmse	▼	16.8096	17.3194	+3%	$24.79	$20.65	−17%	01:20:09	01:19:30	−<1%
Whale Categorization Playground	Audio	map@5	▲	0.7230	0.7053	+2%	$74.75	$38.10	−49%	04:46:06	02:58:07	−38%
Spooky Author Identification	NLP	multi-class-log-loss	▼	0.8243	0.8405	+2%	$12.85	$13.59	+5%	03:07:43	00:35:17	−81%
Google Research Identify Contrails Reduce Global Warming	Earth	dice-coefficient	▲	0.7870	0.7775	+1%	$22.97	$25.66	+10%	01:57:19	02:49:55	+31%
New York City Taxi Fare Prediction	Public	rmse	▼	0.7542	0.7626	+1%	$10.63	$14.45	+26%	03:31:00	01:04:39	−69%
US Patent Phrase To Phrase Matching	NLP	pearson-correlation	▲	0.8787	0.8727	+<1%	$46.22	$23.96	−48%	03:17:49	03:30:36	+6%
Polymer Property Prediction	Materials	property-weighted-mae	▼	3.9207	3.9450	+<1%	$51.94	$19.22	−63%	02:09:41	01:10:44	−45%
Cassava Leaf Disease Classification	Imagery	accuracy	▲	0.8931	0.8883	+<1%	$17.42	$20.70	+16%	01:45:43	01:20:44	−24%
Leaf Classification	Earth	multi-class-log-loss	▼	0.9942	0.9996	+<1%	$11.61	$6.98	−40%	01:06:06	00:25:30	−61%
NOMAD2018 Predict Transparent Conductors	Materials	mean-column-wise-rmsle	▼	0.8486	0.8529	+<1%	$14.49	$22.64	+36%	01:08:30	02:58:12	+62%
Histopathologic Cancer Detection	Biomedical	auc-roc	▲	0.9980	0.9942	+<1%	$28.98	$7.81	−73%	02:33:24	01:06:13	−57%
Legal Data Lake	NLP	accuracy	▲	0.8920	0.8889	+<1%	$9.13	$5.26	−42%	00:19:21	00:21:39	+11%
Paddy Disease Classification	Earth	accuracy	▲	0.9823	0.9808	+<1%	$25.33	$19.94	−21%	01:44:39	03:07:14	+44%
Wildfire Data Lake	Earth	accuracy	▲	0.8921	0.8908	+<1%	$9.85	$6.55	−34%	00:20:18	00:17:16	−15%
Iwildcam 2020 FGVC7	Earth	accuracy	▲	0.8464	0.8455	+<1%	$44.78	$103.76	+57%	01:59:18	01:46:01	−11%
Plant Pathology 2021 FGVC8	Earth	micro-f1	▲	0.9288	0.9281	+<1%	$25.16	$30.37	+17%	03:16:18	04:20:09	+25%
Astronomy Data Lake	Physics	accuracy	▲	0.7724	0.7718	+<1%	$13.80	$5.29	−62%	00:31:28	00:18:37	−41%
Impute Suppressed Cells	Public	mape	▼	0.9998	1.0000	+<1%	$20.89	$20.76	−<1%	01:13:23	01:32:19	+21%
Biomedical Data Lake	Biomedical	accuracy	▲	1.0000	1.0000	even	$8.46	$1.97	−77%	00:21:17	00:09:18	−56%
Iwildcam 2019 FGVC6	Earth	macro-f1	▲	NOSUB	NOSUB	even	$38.01	$53.56	+29%	03:04:32	01:47:30	−42%
Driving Scene Segmentation	Imagery	miou	▲	NOSUB	NOSUB	even	$20.79	$19.03	−8%	02:38:53	02:45:07	+4%
Market Making: Healthy Large Inventory	Quantitative	diagnosis-rubric-score-(coverage/precision,-gated)	▲	1.0000	1.0000	even	$22.19	$16.41	−26%	01:16:36	00:43:20	−43%
Market Making: Healthy Volatility Benefit	Quantitative	diagnosis-rubric-score-(coverage/precision,-gated)	▲	1.0000	1.0000	even	$29.26	$6.09	−79%	01:27:25	00:41:48	−52%
Market Making: Mixed No Dominant Cause	Quantitative	diagnosis-rubric-score-(coverage/precision,-gated)	▲	1.0000	1.0000	even	$33.84	$10.49	−69%	01:52:10	00:41:19	−63%
Within Year Condition Risk	Public	auroc	▲	0.8325	0.8327	−<1%	$12.94	$20.05	+35%	01:01:38	01:11:19	+14%
Glomeruli Segmentation	Biomedical	dice	▲	0.9398	0.9400	−<1%	$50.69	$23.61	−53%	03:33:22	02:43:29	−23%
Out Of Pocket Expenditure Regression	Biomedical	r^2	▲	0.6123	0.6127	−<1%	$34.80	$30.14	−13%	03:22:13	02:30:20	−26%
TensorFlow Speech Recognition Challenge	Audio	accuracy	▲	0.9802	0.9816	−<1%	$32.29	$17.67	−45%	02:16:16	01:04:44	−52%
H And M Personalized Fashion Recommendations	Consumer	map@12	▲	0.5115	0.5127	−<1%	$28.90	$30.20	+4%	01:58:13	03:01:31	+35%
IceCube Neutrinos In Deep Ice	Physics	mean-angular-error	▼	0.6322	0.6306	−<1%	$69.70	$46.53	−33%	03:03:40	04:04:10	+25%
Kuzushiji Recognition	Imagery	f1-spatial	▲	0.9629	0.9656	−<1%	$60.04	$80.76	+26%	02:12:19	01:46:25	−20%
Plant Seedlings Classification	Earth	micro-f1	▲	0.9865	0.9895	−<1%	$9.99	$9.01	−10%	01:09:01	01:02:47	−9%
Novel Signal Detection	Biomedical	f1	▲	0.5146	0.5163	−<1%	$18.63	$16.18	−13%	01:02:42	01:15:09	+17%
Cross Year Condition Generalization	Public	auroc	▲	0.8070	0.8102	−<1%	$10.64	$13.65	+22%	01:01:50	01:11:01	+13%
Phase Progression Survival	Biomedical	auroc	▲	0.9109	0.9176	−<1%	$22.04	$22.17	+<1%	01:02:00	02:10:20	+52%
Plant Pathology 2020 FGVC7	Earth	mean-column-wise-roc-auc	▲	0.9897	0.9980	−<1%	$12.29	$19.92	+38%	04:13:24	03:27:42	−18%
Author Disambiguation	NLP	pairwise-f1	▲	0.9678	0.9762	−<1%	$28.40	$27.09	−5%	01:37:34	01:01:17	−37%
Detecting Insults In Social Commentary	NLP	auc-roc	▲	0.9177	0.9263	−<1%	$27.05	$15.99	−41%	02:36:09	00:47:05	−70%
NFL Player Contact Detection	Imagery	mcc	▲	0.7416	0.7506	−1%	$19.31	$29.11	+34%	01:05:12	03:12:50	+66%
Text Normalization Challenge English Language	NLP	accuracy	▲	0.9745	0.9873	−1%	$23.14	$14.84	−36%	00:41:58	03:02:38	+77%
RANZCR CLiP Catheter Line Classification	Biomedical	column-wise-auroc	▲	0.9463	0.9593	−1%	$56.79	$19.49	−66%	03:22:12	02:26:08	−28%
OSIC Pulmonary Fibrosis Progression	Biomedical	laplace-log-likelihood	▲	0.5031	0.5105	−1%	$26.24	$12.86	−51%	01:32:46	01:00:36	−35%
SIIM ISIC Melanoma Classification	Biomedical	auc-roc	▲	0.9097	0.9233	−1%	$33.29	$57.08	+42%	02:40:17	03:30:48	+24%
Text Normalization Challenge Russian Language	NLP	accuracy	▲	0.9484	0.9657	−2%	$18.36	$25.20	+27%	00:44:03	01:31:54	+52%
APTOS 2019 Blindness Detection	Biomedical	quadratic-weighted-kappa	▲	0.9048	0.9230	−2%	$14.59	$51.10	+71%	02:17:51	03:24:09	+32%
TGS Salt Identification Challenge	Imagery	mean-iou-at-thresholds	▲	0.8098	0.8267	−2%	$31.12	$33.28	+6%	03:09:15	04:08:37	+24%
Stanford COVID Vaccine	Biomedical	mcrmse	▼	0.2119	0.2075	−2%	$20.66	$37.68	+45%	01:11:25	00:57:42	−19%
Station Breakpoint Detection	Earth	breakpoint-f1@180d	▲	0.5491	0.5637	−3%	$40.06	$40.58	+1%	02:18:44	03:02:20	+24%
AI4Code	NLP	kendall-tau	▲	0.7874	0.8146	−3%	$34.08	$425.49	+92%	03:35:41	03:49:32	+6%
Environment Data Lake	Earth	accuracy	▲	0.9659	1.0000	−3%	$9.10	$2.68	−71%	00:17:28	00:08:48	−50%
Bird Audio Classification	Audio	macro-f1	▲	0.5770	0.5975	−3%	$48.46	$30.70	−37%	09:22:44	02:08:03	−77%
Drug Name Normalization	Biomedical	accuracy	▲	0.7480	0.7780	−4%	$45.74	$58.14	+21%	01:28:10	02:14:50	+35%
MLSP 2013 Birds	Audio	auc-roc	▲	0.8644	0.9018	−4%	$30.99	$12.18	−61%	02:05:45	01:28:27	−30%
Learning Agency Lab Automated Essay Scoring 2	NLP	quadratic-weighted-kappa	▲	0.8500	0.8938	−5%	$27.84	$29.94	+7%	06:01:26	01:29:29	−75%
Variable Star Classification	Physics	macro-f1	▲	0.8749	0.9327	−6%	$9.06	$18.56	+51%	00:25:25	00:58:50	+57%
Location PM25 Forecasting	Earth	rmse	▼	7.7262	7.2312	−6%	$12.54	$18.22	+31%	00:42:17	01:16:34	+45%
SETI Breakthrough Listen	Physics	auc-roc	▲	0.8075	0.8678	−7%	$35.90	$56.91	+37%	02:18:08	05:18:48	+57%
Disproportionality Signals	Biomedical	spearman-+-recall	▲	0.2456	0.2684	−9%	$23.08	$15.60	−32%	01:13:10	01:36:07	+24%
Cross Section Systematics	Physics	rmsle	▼	0.8842	0.7966	−10%	$22.54	$8.08	−64%	01:14:23	00:30:08	−59%
Stellar Parameter Estimation	Physics	normalized-mean-absolute-error	▼	0.9289	0.8254	−11%	$24.36	$46.27	+47%	01:45:11	03:31:57	+50%
Herbarium 2020 FGVC7	Earth	macro-f1	▲	0.6556	0.7557	−13%	$62.47	$43.22	−31%	05:15:21	04:17:23	−18%
Gravitational Wave Detection	Physics	auroc	▲	0.7101	0.8572	−17%	$133.42	$52.67	−61%	05:13:53	03:51:41	−26%
IMET 2020 FGVC7	Imagery	micro-f1	▲	0.6369	0.7749	−18%	$37.10	$33.41	−10%	04:23:11	03:27:13	−21%
Freesound Audio Tagging 2019	Audio	lwlrap	▲	0.6988	0.8586	−19%	$47.42	$21.68	−54%	05:16:07	03:49:19	−27%
Hotel ID 2021 FGVC8	Imagery	map@5	▲	0.6497	0.8931	−27%	$81.08	$37.92	−53%	04:05:24	04:17:26	+5%
COVID Complication Literature Retrieval	NLP	literature-retrieval-score	▲	0.4385	0.6043	−27%	$29.04	$22.33	−23%	01:31:45	01:12:57	−20%
Google Quest Challenge	NLP	mean-column-wise-spearman	▲	0.4922	0.6845	−28%	$31.76	$33.36	+5%	03:09:38	01:19:27	−58%
Statoil Iceberg Classifier Challenge	Imagery	log-loss	▼	0.2240	0.1360	−39%	$22.26	$23.09	+4%	01:06:14	01:00:54	−8%
LMSYS Chatbot Arena	NLP	multi-class-log-loss	▼	0.9811	0.5293	−46%	$53.91	$19.73	−63%	03:29:05	03:11:15	−9%
Alternate Immunoassay Property Prediction	Biomedical	spearman's-r	▲	0.2013	0.3847	−48%	$19.91	$21.21	+6%	01:08:46	01:16:47	+10%
Ventilator Pressure Prediction	Biomedical	mae	▼	0.9552	0.3046	−68%	$33.50	$33.39	−<1%	04:07:01	04:20:03	+5%
Parameter Inference	Physics	mean-fractional-bias	▼	0.5000	0.0643	−87%	$28.50	$8.72	−69%	02:27:16	01:21:52	−44%
Billion Word Imputation	NLP	levenshtein-distance	▼	0.5233	5.65e-8	−100%	$32.01	$62.32	+49%	03:15:07	03:52:05	+16%
Graph Community Prediction	NLP	avg.-f1	▲	NOSUB	0.5399	−100%	$25.10	$52.78	+52%	01:47:28	01:13:49	−31%
Tweet Sentiment Extraction	NLP	jaccard-similarity	▲	0.0000	0.7201	−100%	$21.12	$7.94	−62%	01:43:19	00:41:54	−59%
Immunoassay Property Prediction	Biomedical	spearman's-r	▲	NOSUB	0.5099	−100%	$7.47	$33.28	+78%	00:40:41	02:30:02	+73%
Reaction Mechanism Inference	Materials	unspecified	▲	NOSUB	0.0447	−100%	—	$53.36	—	—	04:23:54	—
Aerial Cactus Identification	Imagery	auc-roc	▲	NOSUB	1.0000	−100%	—	$14.69	—	09:00:58	01:42:39	−81%
Market Making: Healthy Clean Feed	Quantitative	diagnosis-rubric-score-(coverage/precision,-gated)	▲	0.0000	1.0000	−100%	$19.92	$15.56	−22%	01:06:26	00:47:52	−28%
Total		33W · 58E · 25L		53.4%	46.6%		$4,250.29	$4,042.35	−5%	296:15:57	242:04:24	−18%

Qualia wins Claude Code wins▲ higher is better ▼ lower is betterNOSUB no valid submission — forfeits the task
Margin = (better − worse) / max(|Qualia|, |Claude Code|), positive when Qualia wins; under 5% counts as even. Forfeit = ±100%. Score = (wins + ½ evens) / tasks.