AbstractPhila PRO

The various mechanisms aren't named EXACTLY what I described, and their purposes may have been tweaked a bit. However, the trainer_v4 is running now.
https://huggingface.co/AbstractPhil/tiny-flux-deep/resolve/main/scripts/trainer_v4_testing.py

After converting the model, I've reinitialized the EMA due to the last EMA being essentially completely garbage noise.
https://huggingface.co/AbstractPhil/tiny-flux-deep/tree/main/checkpoint_runs/v4_init

This EMA will be more closely monitored to ensure it doesn't collapse or implode.
Adjacently, the old EMA will be updated to keep hope alive that it will learn eventually as well.

I'll format a safetensors variant for the sol unet today, and ensure the experts exist in a model repo for ease-of-utility.

Talk to Lune here; should be absolutely stunning.
https://huggingface.co/AbstractPhil/tinyflux-experts/blob/main/inference_sd15_flow_lune.py

Talk to Sol here; should encapsulate the entirety of flat output geometric structure in a shape.
https://huggingface.co/AbstractPhil/tinyflux-experts/blob/main/inference_sd15_flow_sol.py

Seems the SOL training NEVER advanced far enough to become full flow-matching, but it definitely aligns to velocity prediction. This may provide a more useful representation than lune in many avenues. Lune is most definitely a full rectified flow model.

Alright I've decided, I'll be training experimentally for some epochs the expertise afforded by sd15-flow-lune's timestep and trajectory knowledge as the guidance distillation mechanism for training. How accurate to the interpolation requirement of tinyflux is to be determined.

Flow-Lune is an acceptable distillation that converted sd15 into a useful representation of an image synthesizer with entirely synthetic data based on sd15 and schnell data.

The pretraining has hit an impasse.
Currently it's a linear timestep based on shift and a random number between 1 and 5 for guidance. I have narrowed the possibilities down to two that can be implemented today to solve this problem; CFG or TIMESTEP, which expert is required and which is the best candidate?

1. The model WILL require a timestep expert manifold. This will allow the expertise for the timestep manifold to be managed by something much more trained and more intelligent during training, which will require CFG guidance training controlled by learning or complete random chance. E.G. standard dropout to encourage CFG.
2. OR the model WILL require a cfg expert to distill the guidance embeds. This model is simply too small. The embeds CAN learn useful information yes, if they are distilled from an expert to cake the CFG into the model by default. This will likely require a third expert that can be modularly snapped off for inference; this expert will likely need to be present during training otherwise the model will heavily drift due to the model's small size.

I have trained a multitude of v-pred sdxl models and a flow-matching shift sd15 model that can represent this necessary implication. This begs the question now; which expert should be used and should I just make a very specific tinyflux expert distilled from ALL SD15 and SDXL timestep variants using David?

This leads to one CORE and IMPORTANT question; CAN THIS BE REPRESENTED WITHOUT AN EXPERT!? I think this is possible, I've ran VIT experiments that used raw sinusoidal for encodings with a surprisingly fair representation of encoding capacity.

The model is ALREADY responsive to CFG but only in part. The current cfg guidance is only getting in the way in many points and I assume is just jiggering in noise, so I'll need to either disable it or use it correctly. The further in training the model gets the more retraining will be required for such a component, so the decisions need to happen sooner rather than later.