Shouldn't the generation quality be improved?
#2
by
rikunarita
- opened
The message "Hello!" is answered with
"Hello！.
a =. how
A
copy
. examplecde
single line
◀天上 single💬
line
=, example'all combinationmarkdown播 世 call法 call块k！",primer �准规则
好的THENisEual 如果
� sequence序序列 �例例单单双单序例例例单例例例单例单例序序列表单例例例 single例单例单例单例单例单例单例单例单例单例单例单例单例子例单例单单双双双双双双双双双双双双双双双双双双双双双双双双双双0双双双0双双双0双双双0例例例例例例例例例 single单例例例例 single单双双双双双双双..."
Shouldn't we improve this?
Was that the model?
I think I fixed it I just implemented the original tokenizers from the qwen model try it again