Snowflake’s entry into open source AI
On 24 April 2024, Snowflake AI Research — the research division of the cloud-data vendor — releases Arctic, the first large language model published by the company. Distribution is under the Apache 2.0 licence and includes, in addition to the weights, the training templates and documentation on data composition, elements rarely made available for models of equivalent scale.
The stated goal is not competition on generalist reasoning benchmarks, but optimisation for the enterprise use cases typical of Snowflake customers: SQL, code generation, instruction following. The most notable result is the training cost, reported at approximately 2 million dollars in total, a figure significantly lower than the budgets of comparable models by parameter count.
Dense-MoE Hybrid
Arctic’s architecture is defined by its authors as Dense-MoE Hybrid: each block combines a dense component of 10 billion parameters with a Mixture-of-Experts layer made up of 128 experts of about 3.66 billion parameters each. The total count is 480 billion parameters, of which 17 billion active per token — combining a selected MoE expert with the shared dense component.
This structure allows specialist parameters to be distributed over a high number of experts, while keeping a dense base that absorbs general capabilities. Compared to a pure MoE with few large experts, the choice of 128 smaller experts increases specialisation and reduces routing overhead per capability.
Openness and transparency
The Apache 2.0 licence is accompanied by a rare level of operational transparency: Snowflake documents the approach to data composition curriculum, with proportional ratios between web text, code and structured content that vary during training phases. The team also publishes analyses on cost per token, infrastructure choices and trade-offs between total and active parameters. This information makes Arctic a reference point for those designing enterprise models with budget constraints, more than for pure leaderboard competition.
Link: snowflake.com/arctic