58 Experiments
Confiance : high
58-experimentsflash-moeperformance-benchmarkingquantization-analysissystematic-evaluationoptimization-researchqwen3.5-397bcomprehensive-testingfma-kernelsquality-cliff-documentation
Comprehensive systematic evaluation conducted during flash-moe development to optimize performance of qwen3.5-397b on MacBook hardware. Demonstrates rigorous empirical approach to identifying optimal quantization levels, kernel implementations, and quality trade-offs for edge AI deployment.
Experimental Scope
The 58 experiments covered:
- Quantization levels: 2-bit vs 4-bit expert quantization
- Kernel optimization: FMA kernel implementations vs baseline
- Quality assessment: JSON formatting, tool calling reliability
- Performance metrics: Token/second throughput across configurations
- Quality cliffs: Sharp degradation thresholds identification
Key Findings
Through systematic experimentation:
- Identified optimal 4-bit + FMA configuration (4.36 tok/s, excellent quality)
- Documented quality-cliff at 2-bit quantization (JSON malformation)
- Proved FMA kernel benefits (4.36 vs 3.90 tok/s)
- Established production-suitable configuration parameters
Methodology Significance
Represents rigorous engineering approach to AI optimization - not just achieving performance but systematically documenting trade-offs and failure modes. Critical for production deployment decisions where reliability matters more than peak speed.