The Token Tax Index: 28 Languages, Measured
Token counts for the same text in 28 languages under o200k_base. Japanese costs 118 percent more than English using half the characters. Open dataset, CSV and JSON.
Token counts for the same text in 28 languages under o200k_base. Japanese costs 118 percent more than English using half the characters. Open dataset, CSV and JSON.