Neural Architecture Experiments
Separate implemented mechanisms, small pilot findings and untested architecture proposals.
Choose one mechanism
The architecture overview separates memory, routing, repair and representation ideas. Start with a mechanism whose result you can check against a simple control.
Compare controls and costs
Read temporal attention and causal memory for sequence history, judge heads and face based attention for gating, and Hydraform for mutation. Token repair needs a strong rule baseline. BitNet and SVD separate storage savings from execution speed and functional error.
Data structures are a separate branch
The 4D lattice and adaptive range summary articles concern query algorithms. Compare construction and query costs with scanning before connecting those results to neural architecture claims.
Articles in this reading path
- Adaptive Neural Architectures: Experiments and QuestionsExplore memory, routing, repair and compression experiments with a clear distinction between local implementations, measured results and proposals.
- Face Based Attention: Gated Feature MixingCompare gated feature mixing with dense and static controls, including parameter costs and results that do not establish an advantage for the gate.
- Judge Head Attention: Contextual Head GatingCompare contextual attention head gating with standard attention and a token MLP control, separating initial failures from a longer recall comparison.
- Self Healing Tokens: Selective Denoising ExperimentsCompare learned token repair with majority repair, including recovery of corrupt tokens and harm to clean inputs in a controlled synthetic task.
- Temporal Attention for Irregular ObservationsCompare learned time decay, timestamp features and a last observation baseline across synthetic switching, periodic and shifted arrival tasks.
- Transformer Memory: Causal Retrieval and AdaptationTest delayed memory retrieval, episode reset and batch isolation, with explicit limits on comparisons between memory systems of different capacities.
- Hydraform: Adaptive Attention Head MutationExamine attention head mutation, optimiser replacement and checkpoint recovery alongside fixed and random mutation controls on a synthetic task.
- BitNet Style Quantisation: Storage vs ExecutionSeparate float conversion, ternary simulation, packed weight storage and measured execution time, including the limits of a scalar Python decoder.
- SVD Compression: Storage and Functional ErrorCompare SVD factor storage with held out output error on a GPT Neo projection, and see why weight similarity alone cannot establish useful compression.
- 4D Range Queries with Sparse Summary TreesCompare sparse summary trees, scanning and spatial search on synthetic 4D queries, including construction costs and distribution dependent results.
- Adaptive Range Summaries and Interval SearchSeparate multidimensional range summaries from sorted interval search, with exact overlap checks, scan baselines and construction costs.