Dedupe eval prompts by normalized hash
Trail-Session: codex:3b7ca10c-bd5b-4d7b-84c6-abec8ec84be4
Trail-Link: http://localhost:8787/repos/ml-pipeline/session/3b7ca10c-bd5b-4d7b-84c6-abec8ec84be4
Mmaya committed 3 days ago · maincommit c34dcabc6af9dbfb982a99c2e7f43d379ce5d0d7
Agent sessions behind this commit1
Eval numbers look too good. Check whether eval prompts leak into train and fix the split.
Conversationcodex:3b7ca10c-bd5b-4d7b-84c6-abec8ec84be4 · transcript lines 0–10
Artifacts1
Showing 1 changed file with 2 additions and 1 deletions.
data.py+2 −1
@@ -1,10 +1,11 @@
+import hashlib
import json
def split(prompts, eval_frac=0.05):
seen = set()
for p in prompts:
- key = p
+ key = hashlib.sha1(normalize(p).encode()).hexdigest()
if key in seen:
continue
seen.add(key)