Lower LR and lengthen warmup to stop divergence
Trail-Session: claude:285791a1-5658-4fdc-992c-f9a75f8c525f
Trail-Link: http://localhost:8787/repos/ml-pipeline/session/285791a1-5658-4fdc-992c-f9a75f8c525f
Ssam committed 6 days ago · maincommit 60e503f20a088d5f85b0a3f3ed29bac4d806d18f
Agent sessions behind this commit1
The 7B LoRA fine-tune keeps blowing up around step 400. Logs are in logs/run-17.log. Fix it.
Conversationclaude:285791a1-5658-4fdc-992c-f9a75f8c525f · transcript lines 0–13
Showing 1 changed file with 2 additions and 2 deletions.
train.py+2 −2
@@ -1,6 +1,6 @@
MODEL = "llama-7b"
LORA_RANK = 64
-LR = 3e-4
-WARMUP_STEPS = 100
+LR = 1e-4 # 3e-4 and 2e-4 diverged (run-17, run-18)
+WARMUP_STEPS = 500
GRAD_CLIP = None
EARLY_STOP_PATIENCE = None