A twenty-three point gap in Korean
On 2 April 2024 the NAVER Cloud team published a technical report on HyperCLOVA X. On the Korean benchmarks KMMLU, HAE-RAE and KoBigBench the larger model HCX-L averaged 72.07 against 48.92 for LLaMA 2 70b, while on the English MMLU, BBH and AGIEval it scored 58.25 against 58.33, level with it.
Why it matters
Those two lines together are the argument: the same model that runs level in English is twenty-three points ahead in Korean. So the gap comes not from size and not from architecture but from what an English-centric mixture simply does not hold. The authors say plainly that they offer this as guidance for countries building their own models.
The family is HCX-L and the lighter HCX-S. Training ran on a balanced mixture of Korean, English and code, followed by instruction tuning on human-annotated datasets. Evaluation covers reasoning, knowledge, commonsense, factuality, coding, mathematics, chat, instruction-following and harmlessness in both languages. Cross-lingual transfer is shown separately, including machine translation across several pairs and cross-lingual inference for languages the model was not aimed at. What the record does not claim. Three of the report's claims appear in none of the sources. A corpus 6,500 times larger than GPT-3's: the string 6,500 occurs zero times in this technical report, and zero times in the 2021 paper on the first HyperCLOVA, which states a Korean-centric corpus of 560 billion tokens. A GAK Sejong data centre of 600,000 servers: Sejong occurs zero times, 600,000 zero times. The record was therefore moved from the service launch date of 24 August 2023 to the date of the report, where the figures actually are, and confidence stays medium: this is a company report on its own model, and no second independent measurement was found.