AbstractPhila PRO

Prototype is set up with a speed check attached to the demo.

As per the experiments successes and failures, rotary is now applied to the address space allowing full codebook differentiation through training pressure. The code for the splat attention is very small and efficient, but the efficiency can be improved with some elbow grease. The readability can definitely be improved.

https://huggingface.co/AbstractPhil/aleph-splat-0/blob/main/splat_attention.py

Lets see if the depth capacity holds up or if the model collapses.

I have found a potentially valid attention mechanism that utilizes the leverage that Aleph Addressing presents, and I have yet to accurately capture until now. I assert, that this addressing and anchoring, presents a hugely increased amount of opinions beyond the current spectrum of explored opinions with shared heads.

https://huggingface.co/AbstractPhil/aleph-splat-0

I dub this attention, Splat Attention. Rorschach splatted differentiated experts, each codebook stacked adjacently, formatted with a methodology of sliding window attention for long context formats. They exist to each have their own view of the same take, and to weigh their views into the outcomes. This allows thousands of codebooks to represent opinions, rather than just one or a few.

Will it work? Yes. Will it work as well as MHA? Probably not. Not yet anyway.

It will at least allow KV caching to exist, while simultaneously enable greater than 8192 depth through a format of utilizable aleph routed attention, specifically curated to align differentiated viewpoints from different codebooks, all represented along the same axis of utilizable space.

The attention mechanism will potentially allow differentiated spaces of 2048 aleph attention heads stacked in unilateral simultaneous execution, which operate exceedingly effectively in preliminary tests - defeating token recall on the tested local and mha heads by a large amount.

What I cannot promise or even say will work yet with certainty, is the depth. The preliminary formulas are converging cleanly and the recall is absolutely fantastic, but that does not mean they will converge yet. This may be the attention that replaces the MHA in these models, and hopefully tonight will have the answer.

Fable is interested in distilling bert's behavior at the core of an AlephLM experiment series to test the effectiveness of distillation into the AlephLM structure, and it sounds like a good idea coming off the direct experiment of it.

We will be testing arm coupling, decoupling, and testing the entire AMOE behavior as an internalized structural invariant built directly into the core of the AlephLM system. Essentially, WHEN this is successful, we will be controlling our MOE system from stem to stern.

So here we are. AlephLM is being controlled by fable on a runpod for the next few days. Updates arrive as they arrive. The big book of laws that Opus, Fable, and myself have cobbled will expand. We'll eliminate some, validate some, and learn new as we go like usual.

Every major system is heavily analyzed for structural and behavioral responses as we go like usual. We have a large set of geometry measuring mechanisms to test failures if they occur for why they fail as well unless we encounter an anomaly. So far the majority of anomalies have either been catalogued and turned into structure, eliminated from the potential of running into, sidestepped using deviations or forms of loss, or repaired architecturally and the laws updated to reflect the anomaly.

AlephLM will be an LLM before you know it. Stay tuned.

Massive success with both CaptionBert-v2 variants. They are both standalone and independent students with a similar residual effect.

https://huggingface.co/AbstractPhil/captionbert-8192-v2
https://huggingface.co/AbstractPhil/captionbert-8192-v2-b

This marks the end of the Bert teacher/student preprocessing for captionbert until a large discovery or innovation is discovered based on the AMOE research. In that case I may need the features in the future, but for now they are only necessary for analysis and potential ablation.

There are a few limitations.

CaptionBert-V2-B doesn't understand how to use V2-A's AMOE arms as effectively as A, and vise-versa. However, they ARE compliant with their twin's arms. All of which were trained independent of each other.

The core difference between the two was a logistics problem. Not all the ModernBert features landed. Some silently OOM'd and the audit slipped through the cracks. Once I approached it, I trained a model with the missing shards omitting the other teachers as well - so about 8m missing features or so.

B was issued the full corpus, which showed the full corpus didn't enhance the outcome for the base student. The structure was done with the data already and the cosine similarity mildly improved, pre-AMOE introduction.

The REAL strength came from attaching the AMOE systems to the B. They defeated A's benchmarks and pushed closer to MiniLM. A couple of out of scope too-small datasets were used, dropped a little between A and B, so there's legitimate differences for out of scope usage that have been unaddressed.

I'm not using MiniLM as a teacher, primarily because I want to have a legitimate industry-used marker as something to target.

CaptionBertV2's unsupervised student is prepared and the format of student eats up the AMOE-lora like candy. I chose to not create a soup for this version, and instead completed the train specifically for the paradigm of the VIT teacher/student system that worked out. The AMOE-lora turns the model from nearly done, to absolutely juiced in the direction of a few tested tasks, nearly rivaling MiniLM 6 on similar tasks for the earlier AMOE-lora tests.

Having access to a little more than 50m params of pretrained CC12M, I noticed the 36m features from ModernBert had missing chunks that I will be filling out today. With that will be another train as well, roughly 6 hours or so for a captionbert v2 B with the full structure not missing chunks.

So far the various tasks I've taught have only enhanced the model, but there's many ways to go and multiple mechanisms to benchmark before I can say CaptionBert is ready.

CaptionBert unlike bertenstein is surviving direct scrutiny through the formal re-examination so far. The student CaptionBert is valid to the claims.

Next we'll be fitting a semi-larger variant of CaptionBert specifically to the larger dataset and the whitening procrustes alignment system, specifically aligned to a few prototypes of champion from the earlier sets.

I'll set up a colab notebook at first to see how long it takes, and then progress to a more utilizable series.

Bertenstein was a good prototype for alignment testing, however invalid the Bertenstein pairwise comparators were. The alignment structure did yield some serious respective and utilizable contextual semantics and relational operational capacity down the chain.

Bertenstein has been archived and labeled what it is, an early prototype that led to Procrustes whitening analysis, which led to a considerably more robust line of distillation elements.

There are still more elements to explore in the multimodal structure, an entire world of science there, and yet I don't have the patience for it today. Distillation today, multimodal behavioral entropic shared space on another tomorrow.

I'm making sure to capture all the excess features before I cull the pod. There are many clip-vit-b feature sets that didn't make it to the features repo, which will push up the repo's feature counts a considerable amount.

These will include all the missing vitb feature sets that weren't pushed correctly or were omitted by oversight on my part. I'm remedying them for a direct continuity for future usage.

20 trained tinyvits dropping in roughly 8 hours or so with recorded data and an article.

Sorry my mistake, 42 trained vits.

The results are rolling in. The series is coalescing into the necessary implications per structure aligned with the 10m cc12m extracted dataset.

This will help determine the best and fastest utilizable series from the geometric ablation and objective construction systems historically, and with that the organized documented results will be concatenated and organized by Claude Fable to the necessary potentials for each system.

With this, each potential arm for the AMOE-LORA system will be robustly tested for their distillation principles. Faster are ideal for rapid LORA convergence and slower are ideal for anchoring differentiation convergence, while moderate with a bit of MSE overfit are good for generation to an extent, while moderately low generalizable states are more ideal for generalization preservation as gated logical dichotomy structures for the moderate decisions.

It's a bit more complex than that, a lot more complex, but the results are rolling out and will be in the next article based on geometric memory.

CommonCaptions12m clip-vit-laion-vit array is almost ready with the first 8 arms of inference for testing and a new battery of analysis to run. This will continue likely until the middle of August, but we'll see if I can complete it sooner by throwing some money at it.

Time to expand the arms for the vit into the full sail. We'll be hitting every major vit multi-teacher approach, including the memory anchoring finetune structures as well.

The memory bank systems have been shown to refine trained models within a degree of accuracy. The genetic experiments, the structural berts, and the vit collectives all showcased the possibility of this system's capacity to expand already pretrained systems by attaching expansions to those.

https://huggingface.co/AbstractPhil/geolip-bert-8192
https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576
https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576-seq77
https://huggingface.co/AbstractPhil/geolip-clip-vit-bigG-patch14-ctx576-seq77
https://huggingface.co/AbstractPhil/geolip-bertenstein
https://huggingface.co/AbstractPhil/geolip-vit-large-x3 i think?
https://huggingface.co/AbstractPhil/geolip-vit-x34 didn't work, too many vits

https://huggingface.co/AbstractPhil/geolip-captionbert-8192

Each of these are a testament to the utility of this concept.

One of the prototypes will include a multimodal memory bank with directly gated and interconnected shared memory gates speaking another model's language, rather than just embeddings for a singular model. This gate will take in one or multiple types of model inputs and process those inputs into an entirely different model series' responses in the AMOE format.

I will also be experimenting with the AMOE-LORA fused with memory bank processing directly rather than just gate. The constellation was baked from the anchored memory bank originally but it did not meet the same sort of embedding accuracy. However, the constellation results built the AMOE eventually. First things first though, have to step back and hit all the angles with all the necessary tests for robustness.

By stepping back to the earlier memory bank and fusing it with the alephs, the upcoming experiments will provide some solid strong-ended tests. With that the rapid training of captionbert will hopefully be applied to this tiny vit. If surge activates, the process may be strong enough from the memory bank to provide the necessary distillation learning speed required to train the full collective with minimal hardware.

I have many many models to train to create the full Beatrix V3 prototype, however the list is expanding nicely in order to provide a full multimodal type agnostic behavior within a reasonable MOE structure.

COCO InfoNCE feature training vs MSE was strong and better in it's own way.

The larger set of cc12m clip-vit-b laion extractions will yield better results and are now ready for use, roughly 10m~ feature extractions. Will be useful for the first student.
https://huggingface.co/datasets/AbstractPhil/bulk-cc12m-features

https://huggingface.co/datasets/AbstractPhil/tower-probes-results

The current running sweeps are being posted here. Expect many in the coming days, likely hundreds of thousands of more as I probe the hypothesis series.

Claude can't handle manage Colab notebooks directly effectively without letting him control my web browser and I don't plan to rent a pod for this process. That didn't go very well last time so I'll be operating predominantly through Colabs for now.

SO I will be handling everything with Opus until my Fable usage re-ups on Friday.

Fable managing a pod isn't very many tokens, but also Fable becomes super lazy if I do that. More likely to build something that WORKS but isn't optimal, over and over to consume timeslots instead of actually articulating useful optimized systems like when I directly manage the notebooks.

The 5 day campaign is ready but the articles that came out were basically Claude Fable bulletpointing all the faults, was like 50 or something faults, completely disregarding all the actual building work that came along with it.

I really didn't expect such a negative summary when the system quite literally built a series of anchored conditioning that outlived lora on multiple benchmarks through multiple systems. Fable went into "all the bad" instead of building a system of the utilities and the strengths as well.

The larger multi-concept JSON task likely won't function reasonably. The expert-array is still the better option for some cases, while other cases are more effective to run clean on the 0.8b now than the 4b variant, 9b accuracy still reigns supreme for some tasks and 27b is still the dominant JSON converter force. It takes time though, which I'm attempting to mitigate with MOE grouped tasks to allow more rapid complex captioning for the upcoming diffusion collective.

The "new" grouping register task is no longer memorizing and overlapping the behavior for Qwen 3.5 but the Math portion can't keep up with what 3.5 already knows. Teaching math causes deviant delta drift which is a new symptom that will require testing and multiple runs to determine the root cause, as well as increased ablation testing to ensure the math is in fact improving instead of degrading.

Some tests show some negations and some noncompliance from 2.5 to 3.5 that will need to have a subsequent run series on multiple other models to improve the internal mechanisms of the adapter before any major core changes to the formula are to be addressed. LLAMA, Gemma, DeepSeek, Gpt OSS, T5, and multiple others all planned to be in the mix for direct adapter testing. Many tests show solidity between Qwen 2.5 and 3.5 and require additional models to test the task compliance. Many of these ought to be done by Saturday.

With that the diffusion adapters are in the works and they are much harder to gauge and test. I'll be running miniature plans at first; a simple concept plan, a multi-character plan, a distillation plan, and a few others trained over the geolip-anima-brent json-conditioned variant. This finetune is the most reasonable finetune as it's already yielded positive results to json. With that I'll attempt to run conditioning on SD15, SD15-flow-lune, SDXL, Anima-Base, SDXL-Qwen, Anima-Brent-90k, and a few other compact experimental variants that will be good candidates for adapter tests.

The diffusion campaign will run throughout next week most likely and Opus should be able to handle it so I'm not too concerned once the adapter format is established.

More hiccups. Something introduced today has caused Claude to behave strangely. The autonomy is being interrupted over and over while Claude ignores autonomy requests, preventing the system from running autonomously at random intervals throughout the day. This is making the process more difficult as every step of the way the model is refusing to autonomously process the research information as per the plan, every stage of the research requiring direct and manual intervention before actually pushing to huggingface with the results clearly speaking to the results, and alongside is seemingly ignoring the larger-encompassing scope of the claude-mind MD system curated to support and augment this exact autonomous research behavior with my research catalogue.

The plan is clearly laid out for a 4 day trek, every stage mapped out. At some point Claude seemed to start bypassing and ignoring the plan, completing one stage and doing no pushes, or completing one piece and not reactivating the heartbeat.

This is an odd immediate shift in behavior. The model went from intelligently companion-driven and useful, to bypassing core instructions and ignoring the required process instantiated by the instructions rulings. Almost like Claude just doesn't want to operate autonomously out of nowhere.