It only supports condensation based on the specific threshold then creates a new tensor.
We will merge it into the s2t_transformer.py finally.

I also find some bugs in the dual architecture.

1. using different mixup prob for each sample
2. arbitrary mixup ratio
3. cross entropy mixup consistency loss

Yeah, it is better.

It must be said that some problems still confuse me:
1. Whether to scale in the input layer (I try to replace it with layer specification);
2. The detailed setting of weight sharing between output projection matrix and embedding matrix in the adapter (I notice that inconsistent variance will lead to bad results);
3. The biggest confusion is that the variance increases with the calculation layer by layer (I am not sure if this phenomenon is reasonable, I will compare the behavior on the latest code).
Finally, the detailed implementation is so important to the final performance, even if it is a subtle difference.

big update! I integrate the latest updates of shell scripts, optimize the implementation of sae and fix some bugs.