Devlog 03: The CPU Baseline (AVX2 / FMA Golden Oracle)

Before writing custom AIE kernels, we needed an unyielding golden oracle to check numerical correctness and see how fast the host CPU is.

Wrote an unrolled AVX2/FMA Q4_0 vector engine in src/cpu-q4-gemv-engine.cpp:

const __m256i raw = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(w_ptr));
const __m256i lo_nibbles = _mm256_and_si256(raw, _mm256_set1_epi8(0x0F));
const __m256 w_f32 = _mm256_cvtepi32_ps(_mm256_sub_epi32(lo_nibbles, _mm256_set1_epi32(8)));
acc = _mm256_fmadd_ps(_mm256_mul_ps(w_f32, scale_vec), act_vec, acc);

• _mm256_and_si256(..., 0x0F): Masks out the low 4-bit nibbles from 32 packed weights.
• _mm256_sub_epi32(..., 8): Subtracts the Q4_0 symmetric offset (8).
• _mm256_fmadd_ps: Fused multiply-accumulate with FP32 activations in single-cycle throughput.

Benchmarked Qwen2.5-0.5B on CPU: 106.5 tok/s (9.38 ms/tok) on 8 threads. Modern Zen 4 AVX2 is blazing fast on small models, so the NPU has a high bar to clear.