Pankaj Pandey

Yeah honestly haven't run that test yet, but you're right that's the real one to run.
Idea is — same multi-hop Hindi prompts, run once with think forced open, once closed, see if accuracy actually moves or if that English reasoning was just... there, not really doing anything.
For data, skipping MILU (it's recall, not real reasoning) and MGSM doesn't even have Hindi. So just hand-checking a small GSM8K-hi slice myself instead of auto-translating — didn't want noise messing with the results.
And yeah will track think-block token count too. If closed keeps the same accuracy as open, that's basically free speed, no tradeoff. Will post numbers once I run it.

Fair callout — I don't actually know yet. My checks were informal, not a real multi-step eval.
Hunch: for simple factual/instructional prompts (most of my training data), the English trace was probably just habit, not load-bearing. For chained reasoning (math, multi-hop), it might matter more — I didn't test that specifically.
Planning to run a small multi-step Hindi eval (with/without forced-open think block) and post numbers. Open to using a benchmark if you have one you trust.

Thanks for sharing, Divyanshu! I appreciate it. It's encouraging to see more open-source work focused on Hindi and its dialects. I'll explore SASTRI and see how it performs on our evaluation benchmarks. If you have any recommended benchmark datasets or evaluation methodology, I'd love to hear about them

Fire in, energy out 🔥 → ⚡