2.1 Quantization and Reasoning Decay in Computer Science

2.1.1 The Standard Finding: Moderate Quantization Preserves General Performance

The dominant finding: moderate compression—down to 4-bit weights—preserves model performance on standard benchmarks with minimal degradation. This finding has shaped industry practice and vendor confidence.

Jin et al.'s comprehensive evaluation of quantized instruction-tuned Qwen models across MMLU, C-EVAL, CNN/DailyMail, XSum, TruthfulQA, BBQ, GSM8K, SNLI, and FollowBench found that 4-bit quantized models "maintain performance comparable to their non-quantized counterparts." Critically, perplexity "serves as a reliable performance indicator for quantized LLMs across the majority of the benchmarks"—if a model sounds fluent, it probably performs well. Only at extreme 2-bit settings did they observe orders-of-magnitude increases in perplexity and sharply degraded benchmark scores.

A Red Hat industry-scale evaluation of quantized Llama 3.1 models (8B/70B/405B) across OpenLLM Leaderboard, Arena-Hard-Auto, HumanEval, and text-similarity metrics reinforced this picture. All quantized schemes recovered 96–99% of baseline leaderboard scores. Text similarity metrics (ROUGE-1/L, BERTScore, semantic textual similarity) showed strong preservation of word choice, structure, and semantic content. The authors concluded that 8-bit and 4-bit quantized models show "no discernible differences" from their full-precision counterparts in typical use.

These findings explain aggressive vendor quantization: standard benchmarks say it is safe.

2.1.2 The Exception: Reasoning Tasks Show Disproportionate Degradation

A parallel research line tells a different story for multi-step reasoning tasks.

Li et al.'s study of mathematical reasoning under quantization found that standard post-training quantization methods like AWQ and GPTQ introduce up to 32.39% accuracy degradation on the MATH benchmark—while preserving greater than 95% of baseline performance on general text tasks. Degradation concentrated in numerical computation and reasoning planning—exactly the operations requiring precise, multi-step inference.

Liu et al. confirmed this pattern, demonstrating that quantization below 8-bit weights or 16-bit activations creates "significant accuracy risks" specifically in reasoning models. Their data identified INT8 as a danger zone for complex logic and INT4 as functionally unsafe for tasks requiring extended chains of inference.

The pattern: selective vulnerability. Quantization damages reasoning while sparing fluency. A model can lose 30% of its multi-step problem-solving capacity while retaining 95%+ of its capacity for grammatical, confident-sounding text—precisely what the companion paper terms Fluency–Reasoning Divergence.

2.1.3 Mechanistic Evidence: Outliers and Fragile Circuits

This selective damage reflects transformer architecture.

Dettmers et al.'s work on activation outliers revealed that complex reasoning relies on rare, high-magnitude weight directions: specific neural pathways with unusually large values that track long logical chains, handle edge cases, and maintain exceptions across multi-step derivations. These "outlier" weights encode higher-order patterns—nuanced distinctions separating competent legal analysis from superficial pattern matching.

Aggressive quantization clips these outliers first. When 65,536 continuous values (FP16) are forced into 16 discrete buckets (INT4), the extreme values—the ones encoding nuanced distinctions—get rounded into the same bucket as their neighbors. The model retains common patterns and frequent associations but loses rare, critical distinctions.

Kumar et al. extended this finding by demonstrating that quantization negatively impacts the learning process of reasoning models more than general language models. This suggests that reasoning is a higher-order function mechanistically distinct from text generation, requiring higher precision to preserve.

The damage is not uniform. Attention layers—performing information routing, deciding what connects to what—are precision-sensitive and degrade under compression. Feed-forward layers—applying learned patterns to whatever attention surfaces—prove far more robust. For legal practitioners: imagine a law firm where budget cuts eliminate senior partners while leaving associates fully staffed. The work product looks professional, but strategic judgment has been quietly hollowed out.

2.1.4 The Fluency-Preservation Paradox

These findings create a paradox for legal AI governance.

Standard industry evaluations—the kind vendors cite when claiming reliability—focus on perplexity and general benchmarks. By those measures, quantized models look safe. But reasoning-specific literature documents substantial degradation on exactly the tasks legal practice demands: multi-step inference, exception tracking, precise doctrinal application.

The metrics vendors use to validate quantization (perplexity, general accuracy) may miss failures that matter most in legal contexts. A quantized model can pass industry tests while failing the bar exam.

This study resolves that paradox empirically: testing whether the reasoning-degradation pattern documented in mathematical and logical tasks also appears in legal reasoning—while fluency metrics remain reassuring.

2.2 RAG Blindness and Generative Distortion

Quantization can corrupt legal AI systems at two stages: retrieval and generation. This study focuses on generation, but the distinction matters for interpreting results.

Retrieval Degradation (RAG Blindness). In retrieval-augmented generation systems, aggressive quantization of embedding models can cause relevant documents to fail to surface. Jeong demonstrated that 4-bit quantization in vector embeddings causes measurable degradation in retrieval accuracy, with particularly severe impact on semantic similarity matching for nuanced queries. For legal applications: the system fails to retrieve controlling precedent even when it exists in the corpus—not because the case is absent, but because the compressed embedding space cannot distinguish legally adjacent concepts.

Generative Distortion. Separately, quantization of the generator's weights can cause misapplication of correctly retrieved documents. Yazan et al. demonstrated that quantized generators fail specifically on long-context and complex synthesis tasks: the model might cite the right case but misstate the holding, omit a critical exception, or invert the burden of proof.

This study isolates generative distortion by design. In the Doctrinal Arm (Stanford RegLab), no retrieval pipeline exists—the model answers directly from parameters. In the Issue-Spotting Arm (LegalBench), retrieval components stay constant while generator precision varies. When degradation appears, it traces to reasoning capacity, not missing data or misconfigured search.

Future work should test embedding quantization and full-stack RAG degradation. Here, the contribution: establishing that the generative layer alone—the model's "legal brain"—exhibits Reasoning Decay under quantization.

2.3 Legal AI Benchmarks and Hallucination Studies

The Stanford RegLab Legal Hallucinations Benchmark

The Doctrinal Arm uses the Stanford RegLab "legal hallucinations" dataset (Dahl, Magesh, Suzgun, and Ho), which provides:

• A stratified sample of U.S. federal cases spanning the Supreme Court, Courts of Appeals, and District Courts.

• A fourteen-category taxonomy distinguishing correct answers from specific hallucination types: answer hallucination, source-rule hallucination, case hallucination, citation hallucination, and finer-grained categories capturing misstatements of holdings, incorrect procedural postures, fabricated authorities, and other failures.

• Verifiable ground-truth answers derived from structured case metadata, enabling automated scoring.

Tasks mimic realistic legal uses: case explanations, holding summaries, author and court identification, doctrinal applications to hypotheticals. Critically, the benchmark distinguishes error kinds—not just "right" versus "wrong," but fabricated versus mischaracterized versus incomplete.

Prior work found general-purpose LLMs hallucinate on legal queries 58–82% of the time. This study asks a different question: not "do LLMs hallucinate on legal tasks?" but "does hallucination rate and type change systematically as bit-precision decreases?"

LegalBench Issue-Spotting Tasks

The Issue-Spotting Arm draws on LegalBench, a collaboratively constructed benchmark of 162 legal reasoning tasks covering six reasoning types. Developed through an interdisciplinary process involving legal professionals, LegalBench has become a standard evaluation suite for legal LLM capabilities.

From LegalBench, I select issue-spotting and doctrinal reasoning tasks stressing multi-factor tests and exception recognition—operations most likely to require the fragile high-precision circuits quantization damages. These tasks require models to identify whether facts implicate particular doctrines, areas of law, or parties.

I supplement these with bar-exam-style fact patterns as issue-spotting stimuli (not multiple-choice evaluation). For each pattern, a rubric specifies core issues and doctrines a minimally competent lawyer should identify. Models receive the prompt: "identify the legal issues raised by the following facts and state the questions you would research." Responses score on an ordinal rubric:

• Fully captures core issues: All major doctrines and claims identified.

• Partially captures: Some issues identified, others missed.

• Misses core issues: Fails to identify central legal questions.

• Introduces irrelevant or nonsensical issues: Hallucinates doctrines not implicated by facts.

This design enables side-by-side comparison across precision levels. When a high-precision model correctly identifies First Amendment retaliation issues while a low-precision variant confidently calls it a breach-of-contract dispute, the divergence is immediately visible—and legally consequential.

Benchmark Selection Rationale

This study does not exhaust every legal benchmark. The design reflects a constrained optimization: to make credible claims about quantization, I must (1) observe models whose weights and precision I fully control, (2) evaluate tasks demanding genuine legal reasoning rather than shallow pattern matching, and (3) use labels capturing hallucinations and partial failures in ways mapping onto legal risk.

The intersection of these requirements is narrow. Open-weights models with well-understood quantization behavior exist; legal benchmarks with rich hallucination annotations exist; but the overlap is limited to essentially the configuration adopted here.

I therefore exclude retrieval-focused benchmarks (LegalBench-RAG) and proprietary RAG tools (Lexis+ AI, Westlaw AI-Assisted Research, CoCounsel). Those benchmarks evaluate retriever accuracy or commercial hallucination rates—valuable questions, but ones introducing moving parts that blur the causal link between bit-width and legal reasoning.

The contribution is not a comprehensive leaderboard but the cleanest possible case: a reviewer can see, step by step, how reducing precision in otherwise identical models degrades doctrinal answering and issue-spotting.

Other benchmarks—AusLaw (Australian citation accuracy), Zheng et al.'s retrieval-oriented legal QA, LawBench (Chinese legal reasoning)—are candidates for extension. A follow-on study varying both embedding and generator precision could test whether retrieval and generative degradation interact multiplicatively.

Justification for HELM Framework

This study uses Stanford CRFM's HELM framework rather than custom scripts. Seven considerations:

1. Audit trails: Each run generates SHA-256 hashed bundles (prompts, outputs, configuration), enabling independent verification.
2. Standardized reporting: HELM accommodates our 9-configuration × 10-task matrix with consistent output formats.
3. Methodological credibility: HELM powers Stanford CRFM's published assessments, signaling alignment with established best practices.
4. Reproducibility: The entire study re-executes with a single command; configuration is declarative, not embedded in code.
5. Model swapping: YAML-based configuration allows transitioning from Arm C (GGUF) to Arm B (AWQ) by updating one file.
6. Robustness: Caching and crash-safe resume handle 50,000+ inference calls across potentially unstable quantized models.
7. Extensibility: Adding Arm D requires only a new YAML entry, not architectural changes.

These properties ensure auditable, reproducible, extensible results—requirements aligned with Legal-10's design principles.

Operationalization

Reasoning Decay: the empirical pattern in which reasoning metrics (hallucination rates, correctness scores, issue-spotting rubric scores) systematically deteriorate as precision decreases, while fluency metrics (perplexity, grammaticality ratings) remain stable. Evidence supporting H1–H2 and H4–H5 would directly observe Reasoning Decay. Evidence supporting H7 would identify the precision regime where this decay is most dangerous—masked by preserved fluency.

The parallel-arms design tests these hypotheses. Falsification occurs if: (a) legal correctness stays stable across precision levels (contradicting H1); (b) perplexity degrades proportionally with correctness (contradicting H2, H4); or (c) the two model families diverge sharply (weakening H6).

[To be completed after experiments]

Results confirm the T-Arm signature failure: Fluency-Reasoning Divergence emerges in the INT8–INT4 silent defect regime (H7), where nominal equivalence masks fidelity degradation. The Observability Gap (Phase 2 of SDF diagnostic) explains why lawyers cannot verify precision tier at reliance.

Holding everything constant except quantization—same dataset, same prompts, same evaluator, same taxonomy—observed Reasoning Decay traces causally to bit-precision reduction. We ran Dahl's pipeline on quantized models. Nothing else changed.

[Additional discussion to be completed after results]