Semantic-match rediscovery (nomic-embed-text)

The exact-match expert-agreement score landed near-zero (mean 0.015) on ORKG temporal-holdout. This page tests whether that was a measurement artifact (too strict) or a real ceiling. For each generated hypothesis we compare the IV/DV/mechanism/quantitative-prediction fields against the held-out paper's real contribution text using cosine similarity of local embeddings — no cloud, no keys, nomic-embed-text via Ollama.

Domains

Mean exact match

0.0050

Mean semantic match

0.6272

Emb model

nomic-embed-text

Domain	Gap (yrs)	Exact	Semantic	IV	DV	Mechanism
Biomedical research in Psychiatric Disorders (biomedicine)	1	0.0000	0.5786	0.5483	0.5588	0.6297
Cyclodextrin complexes to enhance drug solubilty or bioavailabilty (biomedicine)	1	0.0667	0.6764	0.7587	0.5680	0.6773
experimental evolution (biomedicine)	2	0.0000	0.6169	0.5360	0.5670	0.6546
Chemical sensors (chemistry)	1	0.0000	0.5717	0.5917	0.5435	0.5649
Niobium-Based Materials for Photocatalytic Solar Fuel Production (chemistry)	1	0.0000	0.6787	0.6585	0.6273	0.7089
TiO2 Photocatalysis (chemistry)	1	0.0333	0.6355	0.5717	0.5865	0.6620
empirical research in requirements engineering (computer_science)	1	0.0000	0.5922	0.5566	0.5638	0.6221
Image classification (computer_science)	1	0.0000	0.5598	0.4967	0.5688	0.6167
Semantic segmentation (computer_science)	1	0.0000	0.5597	0.5987	0.3838	0.6363
Biodiversity inventories with DNA based-tools (environmental_science)	1	0.0000	0.6822	0.6120	0.6935	0.7169
CMIP5 (environmental_science)	1	0.0000	0.6228	0.6218	0.5770	0.6150
Global climate modelling (environmental_science)	1	0.0000	0.6315	0.5777	0.5685	0.6882
Mapping dopant–host combinations in ALD thin films (materials_science)	1	0.0000	0.6008	0.6793	0.4689	0.6251
Mapping precursor chemistries used in rare-earth ALD processes (materials_science)	1	0.0000	0.6315	0.6228	0.5578	0.6625
process parameters on the performance characteristics of ALD-deposited films (materials_science)	4	0.0000	0.6256	0.6383	0.5204	0.6599
CT Image Segmentation and Classification (neuroscience)	1	0.0000	0.6462	0.6583	0.5148	0.7158
Simultaneous localization and mapping (neuroscience)	1	0.0000	0.5865	0.5610	0.5448	0.6158
Dataset used in wind energy potential assessment (physics)	1	0.0000	0.6801	0.6596	0.6658	0.6709
Solar radiation prediction (physics)	1	0.0000	0.6723	0.5972	0.7396	0.6732
Wind speed distributions performance analysis (physics)	1	0.0000	0.6953	0.6454	0.6610	0.7542

Temporal-gap sweep

If semantic match were a pure measurement artifact, it should rise toward 1.0 as the temporal gap shrinks, while exact match remains near zero. A real ceiling would stay flat regardless of gap size. Table below: bucket by years between latest prior and held-out.

Gap (yrs)	Domains	Exact mean	Semantic mean	Gap between means
1	18	0.0056	0.6279	0.6223
2	1	0.0000	0.6169	0.6169
4	1	0.0000	0.6256	0.6256