← All posts
standalone

Tencent Hunyuan3 (Hy3): 295B MoE Model That Runs on 21B Parameters

Tencent's open-source Hy3 model uses mixture-of-experts to activate only 21B of 295B parameters, claiming half the hallucination rate of similarly-sized models.

M
Mainthreaded ·2 min read ·1 source
#hunyuan3-hy3#tencent-ai#mixture-of-experts#open-source-llm#large-language-models

TL;DR

Tencent released Hy3, an open-source 295B parameter model that only activates 21B parameters per inference pass. The headline claim: it matches models two to five times its active size, with a reported hallucination rate of 5.4%.

What happened

Tencent released Hy3 (short for Hunyuan3) as an open-source model built on a mixture-of-experts (MoE) architecture. MoE is a design where the model has many “expert” sub-networks, but only a subset of them activate for any given input — so you get the representational capacity of a huge model without paying the full compute cost every time you run it. In Hy3’s case, the total parameter count is 295 billion, but only 21 billion are active at inference time.

Tencent claims Hy3 performs on par with models two to five times its active parameter count, which would put it competing with dense models in the 40B–100B range. They also say the model’s hallucination rate is 5.4%, which they describe as roughly half the rate of comparable models. No benchmark methodology details are included in the available sources, so treat those numbers as Tencent’s self-reported figures until independent evals confirm them.

Why it matters

The MoE approach Hy3 uses is the same basic trick that powers models like Mixtral and reportedly GPT-4 — you get a large “effective” model at a fraction of the per-token compute cost. What makes Hy3 worth paying attention to is the combination of scale (295B total), openness, and that hallucination claim. A 5.4% hallucination rate, if it holds up on standardized benchmarks, would be a meaningful number for production use cases where factual reliability actually matters. That’s the “if” doing a lot of work here, though — Tencent hasn’t published methodology, and self-reported hallucination metrics vary wildly depending on the test set.

For developers evaluating open models, the practical question is whether 21B active parameters means you can run this on hardware you actually have. A 21B active-parameter MoE model doesn’t behave quite like a 21B dense model in terms of memory requirements — you still need to load the full weight set, which at 295B is massive. So the compute savings show up in throughput and latency, not necessarily in VRAM footprint. If you’re expecting to run this on a couple of A100s the way you’d run Mistral 7B, you’ll want to wait for more specifics on serving requirements before getting excited.

What to watch

  • Independent benchmark results from groups like EleutherAI’s LM Evaluation Harness or the Hugging Face Open LLM Leaderboard will tell you whether the hallucination and capability claims survive outside Tencent’s test conditions.
  • Watch for quantized versions and community serving guides — if Hy3 becomes practical to run on sub-8xA100 setups, adoption will follow quickly.

Sources