Resource-Efficient Routing in Mixture-of-Experts Models Based on Multi-Layer Reversible Cellular Automata
The Mixture of Experts (MoE) architecture has demonstrated high efficiency in scalinglarge neural network models. However, deploying sparse MoE models on computationalnodes with strict resource constraints entails critical latencies, high memory consumption,and computational redundancy. Traditional parametric routers (gating networks) basedon the floating-point Softmax operation require resource-intensive matrix multiplications,the complexity of which grows linearly with the data dimensionality and the number ofexperts O(D × N ). To overcome these limitations, this paper proposes a hybrid parametrictoken routing method that combines dynamic neural network projectors with a stochasticnon-parametric loop for quantile regulation of selection thresholds based on multi-layerreversible continuous cellular automata. The developed routing algorithm minimizes thefloating-point computational load by projecting continuous input features into a quasi-ternary relevance feature space. The routing dynamics are governed by the synchronousevolution of a continuous cellular field utilizing a spatial contextualization operation basedon one-dimensional convolutions and smooth non-linear activation functions. This topologyensures unobstructed gradient flow, thereby minimizing the volume of cached activationsduring the training of intermediate states. In the final stage, the continuous cellular au-tomaton collapses the pseudorandom distribution of cells into an integer index of the targetexpert, ensuring a constant computational complexity of O(1) for the selection thresholdquantile adaptation algorithm relative to the processed batch size. Theoretical and algo-rithmic analysis demonstrates that the proposed router effectively shifts the computationalload from specialized tensor cores and floating-point units to lightweight arithmetic op-erations, eliminating the need for resource-intensive conditional batch sorting procedureswithin the baseline context. The introduction of spatial contextualization into the contin-uous cellular field ensures uniform load balancing among experts, naturally mitigating therouting collapse problem without introducing coarse auxiliary penalty loss functions. Theproposed architecture radically reduces power consumption, making it an optimal solutionfor deploying sparse neural network models under extreme edge computing constraints, onInternet of Things (IoT) nodes, and within embedded microprocessor systems.