One Question to Ponder
Pre-training was conducted in three phases, covering long-horizon pre-training, mid-training, and a long-context extension phase. We used sigmoid-based routing scores rather than traditional softmax gating, which improves expert load balancing and reduces routing collapse during training. An expert-bias term stabilizes routing dynamics and encourages more uniform expert utilization across training steps. We observed that the 105B model achieved benchmark superiority over the 30B remarkably early in training, suggesting efficient scaling behavior.
，推荐阅读新收录的资料获取更多信息
Bibliographic and Citation Tools。新收录的资料对此有专业解读
The algorithm maintains a running "best distance" that starts at infinity. As it walks the tree, it checks each visited point and updates the best distance if it finds something closer. Before recursing into a child node, it checks whether the closest possible point in that child's bounding box is farther than the current best. If so, the entire subtree gets pruned.。新收录的资料对此有专业解读