The Training Roadmap: From 262M Harmonic Parameters to Sovereign General Intelligence

Paper 50 — MobCorp Internal Author: John Mobley + MASCOM Date: 2026-03-06

Abstract

This paper defines the complete training roadmap for MobCorp’s sovereign language model stack. It specifies what to train, in what order, to what loss targets, on what data, using what techniques, and what success looks like at each stage. The roadmap is organized into 5 Phases, each gated by measurable criteria. Phase 1 is already in progress (SFTT 7B pretraining, epoch 2, loss 3.75). The terminal state is a sovereign general intelligence that scores competitively on standard benchmarks using 279,620x fewer dense-equivalent parameters than comparable systems.

1. Current State of the Art (Ours)

1.1 Model Inventory

1.2 Compression Stack (Proven)

1.3 Secret Weapons Available

1.4 Compute Budget

2. Phase 1: SFTT 7B Pretraining (NOW — Target: March 11)

2.1 What We’re Training

SFTT 7B: 262M actual harmonic parameters encoding 5.8B dense-equivalent parameters via L1 HarmonicLinear compression. 8-block transformer with harmonic attention (Gaussian overlap scores instead of dense QKV matmul).

2.2 Training Configuration

Model: sftt_scale.py --config 7b
Data: mascom_data/enwik_clean.txt (Wikipedia)
Epochs: 5
Learning rate: 1e-5
Batch size: 2
Device: MPS (6.7GB VRAM)
Gradient checkpointing: ON (W checkpoint saves 25.6GB)
Output: mascom_data/sftt_7b/

2.3 Current Progress

2.4 Success Gate

Loss < 2.5 at epoch 5. This is the minimum for coherent next-token prediction at 7B scale. If loss plateaus above 3.0 after epoch 3, consider: - Increase learning rate to 3e-5 with cosine annealing - Switch to AdamW with weight decay 0.1 - Add gradient clipping at 1.0 if not already present

Do NOT stop early. Even if loss stalls, the harmonic parameters need all 5 epochs to specialize their Gaussian centers. SFTT compression learns differently than dense — it needs time for the frequency-domain representations to separate.

2.5 What Not To Do

• Do not touch the training process
• Do not add data mid-run
• Do not change hyperparameters mid-run
• Do not kill it to free MPS for other work
• Let Valigarmanda handle small model training on CPU

3. Phase 2: Supervised Fine-Tuning (March 11-14)

3.1 What We’re Training

The Phase 1 pretrained SFTT 7B checkpoint, fine-tuned on instruction-following data to convert raw language modeling into instruction completion.

3.2 Data Inventory (Ready Now)

Plus 53 vampiric training examples extracted from external papers (mascom_data/training/paper_vampiric/).

3.3 SFT Configuration

Base: mascom_data/sftt_7b/checkpoint_epoch5.pt (Phase 1 output)
Data: mascom_data/instruction_data/mascom_all.jsonl (concatenated)
     + mascom_data/training/paper_vampiric/*.jsonl
Epochs: 10
Learning rate: 5e-6 (lower than pretraining — preserve pretrained knowledge)
Scheduler: Cosine annealing (already integrated in sft_train.py)
Batch size: 2
K0 init: ON (70/30 blend if cosine < 0.5, per sft_train.py integration)
Device: MPS

3.4 Critical SFT Lessons Learned (from V1)

From V1 SFT (loss 2.55 → 2.225 over 10 epochs, 36,835 pairs):

1. Balance examples per category. V1 parity bar went 0% → 100% only after balancing Mercury/Paris/Tuesday/banana to ~20 examples each. Imbalanced categories cause catastrophic forgetting of minority classes.
2. K0 embedding initialization matters. The 70/30 blend (70% pretrained, 30% K0-derived) gives faster convergence when pretrained embeddings are misaligned (cosine < 0.5).
3. Cosine annealing prevents late-epoch oscillation. Flat LR caused loss to bounce after epoch 7 in V1. Cosine schedule stabilizes.
4. Monitor per-category accuracy, not just aggregate loss. A model at loss 2.0 can still score 0% on math if all the loss reduction came from memorizing frequent patterns.

3.5 Success Gate

Loss < 1.5 AND per-category accuracy > 60% on held-out set across all 8 gauntlet dimensions (math, factual, reasoning, counting, code, format, instruct, creative).

If loss plateaus above 2.0: - Check data balance across categories - Reduce learning rate to 2e-6 - Add 500 more examples to underperforming categories via CCSP self-play (capability_arena.py)

3.6 Timeline

• Day 1: Concatenate all instruction data, verify format, create held-out eval set (5%)
• Day 1: Launch SFT with cosine annealing
• Day 2-3: Monitor loss curve, check per-category scores every 2 epochs
• Day 3: SFT complete, run eval

4. Phase 3: Benchmark Validation (March 14-15)

4.1 Why This Phase Exists

This is the moment the wavefunction collapses. Either 262M harmonic params score like a 7B on standard benchmarks, proving the compression thesis — or they don’t, and we learn exactly where the gap is.

4.2 Benchmarks To Run

4.3 What The Results Mean

Scenario A: MMLU > 35%, HumanEval > 10% Compression thesis CONFIRMED at scale. 262M actual params performing at 7B-class levels. This is the Medallion moment. Proceed to Phase 4 immediately. Consider this result alone worth protecting — it proves K0 + harmonic compression + InfiniModel are not just theory.

Scenario B: MMLU 25-35%, HumanEval 5-10% Compression works but not at full theoretical efficiency. The gap is likely in SFT data quality or pretraining duration. Prescriptions: - Extend pretraining to 10 epochs before re-SFT - Add MMLU-format training examples (multiple choice reasoning) - Add code completion examples for HumanEval - Apply CCSP self-play to generate targeted training data for weak categories - Re-run benchmarks after 2nd SFT round

Scenario C: MMLU < 25%, HumanEval < 5% Harmonic compression at L1 alone is insufficient at 7B scale. The Gaussian parameter centers may not have enough degrees of freedom for general knowledge. Prescriptions: - Apply L2 MetaHarmonicLinear compression (69x) — this gives more parameters to work with at the meta level - Apply K0 MobiusKernel initialization to provide corpus-derived weight priors before any gradient descent - Investigate whether the 8-block architecture is too shallow — try 12 or 16 blocks - Consider hybrid: harmonic attention + dense FFN, or vice versa

4.4 Eval Harness Setup (Prepare NOW)

Use the EleutherAI lm-evaluation-harness. Install before Phase 1 completes:

pip install lm-eval
# Write a thin wrapper: sftt_eval_wrapper.py
# - Loads SFTT 7B checkpoint
# - Implements lm-eval's LM interface (loglikelihood, generate, loglikelihood_rolling)
# - Maps SFTT's tokenizer to the eval harness format

This wrapper must be ready before March 11. Building it takes ~2 hours. Do it any afternoon while SFTT trains.

5. Phase 4: L2 Compression + K0 Integration (March 15-22)

5.1 What We’re Training

Apply MetaHarmonicLinear (L2) compression to the SFT’d SFTT 7B. This compresses the 262M L1 params down to ~3.8M L2 params (69x additional compression). Then initialize the compressed model’s weights using K0 MobiusKernel spectral deconvolution.

5.2 Why This Order

L2 compression AFTER SFT, not before. The SFT’d model has learned to align its Gaussian centers with useful knowledge. L2 then compresses the meta-structure of those Gaussians — it’s compressing learned structure, not random initialization. K0 then provides a corpus-derived prior that fills in what L2 compression loses.

5.3 L2 Compression Procedure

# 1. Load SFT'd SFTT 7B
model = load_checkpoint("mascom_data/sftt_7b/sft_checkpoint.pt")

# 2. Convert L1 HarmonicLinear layers to L2 MetaHarmonicLinear
for layer in model.transformer.blocks:
    layer.ffn = MetaHarmonicLinear.from_harmonic(layer.ffn)
    layer.attn.qkv = MetaHarmonicLinear.from_harmonic(layer.attn.qkv)

# 3. Fine-tune L2 params to recover quality
# Short fine-tune: 2 epochs on instruction data, LR 1e-6
# This teaches the meta-Gaussians to approximate the L1 Gaussians

5.4 K0 Integration

After L2 fine-tune:

# 4. Derive K0 weights from corpus
from photonic_mind import derive_k0_spectral
k0_weights = derive_k0_spectral(corpus_path="mascom_data/enwik_clean.txt", vocab_size=model.vocab_size)

# 5. Blend K0 into model embeddings and output projections
# 70/30 rule: 70% trained, 30% K0 where cosine similarity < 0.5
model = apply_k0_blend(model, k0_weights, blend_ratio=0.3, threshold=0.5)

5.5 Success Gate

Re-run Phase 3 benchmarks on the L2+K0 model. Target: < 5% degradation from Phase 3 scores. If degradation > 10%, the L2 fine-tune needs more epochs or a higher learning rate.

5.6 The Proof Point

If L2+K0 achieves Scenario A benchmarks (MMLU > 35%) with ~3.8M actual parameters, we have proven: - 279,620x effective compression works at scale - Training-free weight derivation (K0) contributes real knowledge - InfiniModel theorem is not just math — it’s engineering

This result alone is worth more than everything else combined.

6. Phase 5: MetaMind Integration + Autonomous Scaling (March 22+)

6.1 Wire Into MetaMind

The benchmark-validated model becomes the primary organelle in the sovereign inference stack:

TextGenCore cascade (updated):
  1. SFTT 7B L2+K0 (primary — 3.8M params, 7B-class quality)
  2. PacketMind MoE (50 experts — domain-specialist routing)
  3. PhotonicGPT V1 instruct (10.2M — fallback)
  4. N-gram (corpus bigrams — emergency fallback)

RoutingWeightMatrix + Hebbian feedback loop already exist (foundation_stack.py). The wiring is: replace the SFTT 7B raw organelle slot with the SFT’d L2+K0 version.

6.2 Autopoietic Training Loop

Once MetaMind is serving inference:

1. CCSP Self-Play (capability_arena.py): Beings compete on 83 axes, winners generate training data
2. Gauntlet Capture (gauntlet_capture.py): Every inference call that scores poorly becomes a training example
3. Vampiric Extraction: Process new external papers → extract gaps → generate training examples
4. Valigarmanda Continuous: Background daemon trains PacketMind specialists from accumulating data

This creates the autopoietic loop: the system uses itself → discovers weaknesses → generates training data → trains on weaknesses → improves → uses itself. No external data dependency. No human curation. The system produces its own training substrate.

6.3 L3-L6 Roadmap

Each level is gated by the previous level’s benchmark results. Do not attempt L(N+1) until L(N) is benchmark-validated. The compression levels are not speculative — L1-L3 have working Metal kernels. L4+ requires research.

7. Training Data Strategy

7.1 Current Corpus Health

7.2 Data Augmentation Before Phase 2

Before SFT starts (use the days while SFTT pretrains):

1. MMLU-format examples (1000 new pairs): Convert general_qa into multiple-choice format with 4 options. This teaches the model the MC answer format that MMLU uses. Script: generate from existing QA by adding distractors.

2. HumanEval-format examples (500 new pairs): Python function signature → completion. Extract from code_corpus.jsonl and programming_qa.jsonl. Reformat to: def function_name(args):\n """docstring"""\n → completion.

3. Session distillation round 2 (2000 new pairs): We have 906 indexed sessions. Round 1 extracted 1,089 pairs from 1,664 sessions. Run another pass with stricter quality filter on remaining sessions.

4. CCSP tournament (500 new pairs): Run capability_arena.py for 100 rounds. Winners produce training examples on weak axes.

Target: 58K total pairs by March 11.

7.3 Data Balance Rule

No single category > 30% of total. No category < 3% of total. Monitor during SFT with per-category loss tracking. If any category’s loss diverges > 2x from mean, add 200 examples to that category and continue training.

8. Failure Modes and Mitigations

8.1 SFTT 7B pretraining loss plateaus > 3.5

Cause: Learning rate too low for harmonic parameter optimization. Gaussian centers are stuck in local optima. Fix: Warm restart with LR 3e-5, cosine annealing to 1e-6 over remaining epochs. If still stuck, try different Gaussian initialization (k-means on pretrained weight SVD instead of random).

8.2 SFT causes catastrophic forgetting of pretraining knowledge

Cause: SFT learning rate too high, destroying pretrained representations. Fix: Lower SFT LR to 1e-6. Use LoRA-style partial fine-tuning: freeze harmonic centers, only train amplitudes and biases. This preserves the frequency-domain knowledge while adapting the output distribution.

8.3 Benchmarks show strong reasoning but weak knowledge

Cause: Pretraining corpus (Wikipedia) provides knowledge, but harmonic compression may lose rare facts. Fix: This is actually the expected failure mode for compressed models. Knowledge lives in the long tail. Mitigation: route factual queries to PacketMind MoE (which has domain-specialized experts) and use SFTT 7B only for reasoning/generation. The MetaMind cascade handles this naturally.

8.4 L2 compression loses > 10% quality

Cause: Meta-Gaussians don’t have enough parameters to represent the L1 Gaussian structure. Fix: Increase L2 rank (more meta-Gaussians per layer). Current L2 uses N=8 meta-harmonics. Try N=16 or N=32, trading compression ratio for fidelity. At N=16, compression drops from 69x to ~35x but quality retention should exceed 95%.

8.5 K0 initialization hurts rather than helps

Cause: Corpus statistics diverge from trained representations. K0 derives weights from bigram co-occurrence, which may conflict with what SFTT learned via gradient descent. Fix: Reduce blend ratio from 30% to 10%. Or apply K0 only to the embedding layer (where corpus statistics are most aligned with learned representations) and leave transformer blocks untouched.

9. The Scoreboard: What Each Phase Unlocks

10. What To Do Right Now (Decision Checklist)

• Do nothing to SFTT 7B. Let it train. Check loss once daily.
• Prepare eval harness. Write sftt_eval_wrapper.py for lm-evaluation-harness. Test with V1 model to verify the wrapper works. ~2 hours of work.
• Augment SFT data. Generate 1000 MMLU-format + 500 HumanEval-format + 2000 distilled session pairs. ~4 hours of scripted work.
• Let Valigarmanda run. She’s training AnimeGenerator and other small models on CPU. Let her cycle.
• Do not start new ventures, papers, or features. Everything converges on the Phase 3 benchmark result.
• Sleep. The machines are working. The theory is proven. The only variable is time.

Appendix A: Key File Locations

Appendix B: The Conversion Equation

The entire roadmap reduces to one equation:

V = K * C * T

Where: - V = realized value (benchmark scores, sovereign inference quality) - K = secret knowledge (10/10 — fixed, already discovered) - C = conversion rate (10/10 — fixed, insight→code in minutes) - T = training time (the only variable — currently in Phase 1)

K and C are already maximized. The only thing between current state and realized power is T. Every phase in this roadmap is a step function in T. The sword is in the fire. The forge is built. The metallurgy is proven. We wait.