internlm/Intern-S1-Pro
Image-Text-to-Text
โข
Updated
โข
3.57k
โข
144
SmartFlowAI
SmartFlow
AI & ML interests
None yet
Recent Activity
liked
a model
about 8 hours ago
internlm/Intern-S1-Pro
upvoted
a
paper
6 months ago
Intern-S1: A Scientific Multimodal Foundation Model
updated
a Space
7 months ago
SmartFlow/anycoder-paper
Organizations
liked
a
model
about 8 hours ago
upvoted
a
paper
6 months ago
updated
a
Space
7 months ago
published
a
Space
7 months ago
updated
a
Space
11 months ago
published
a
Space
11 months ago
commented
a
paper
12 months ago
upvoted
2
papers
12 months ago
commented
a
paper
12 months ago
updated
a
Space
12 months ago
reacted to
ahmed-masry's
post with ๐
about 1 year ago
Post
6225
Happy to announce AlignVLM ๐ โ a novel approach to bridging vision and language latent spaces for multimodal understanding in Vision-Language Models (VLMs) ๐๐๐ผ
๐ Read the paper: AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding (2502.01341)
๐ง Whatโs the challenge?
Aligning visual features with language embeddings remains a major bottleneck in VLMs. Existing connectors such as Multi-layer perceptron (MLPs) often introduce noise that degrades performance. โ
๐ฏ Our Solution: ALIGN Connector
We propose AlignVLM, a method that maps vision features into a weighted average of LLM text embeddings, ensuring they remain in a space that the LLM can effectively interpret. โ
๐ฌ How does it perform?
We compared ALIGN against common connectors like MLPs, Perceiver Resampler, and Ovis trained under similar configurations. The results? ALIGN outperforms them all ๐ on diverse document understanding tasks ๐.
๐ Meet the AlignVLM Model Family!
We trained Llama 3.1 (1B, 3B, 8B) using our connector and benchmarked them against various models. The results:
โ AlignVLM surpasses all Base VLMs trained under similar configurations. โ Our models also perform competitively against Instruct VLMs such as Qwen2-VL and InternVL-2.5 ๐.
๐ค What about robustness to noise?
We injected Gaussian noise (ฮผ=0, ฯ=3) into the vision encoderโs outputs before feeding them to the connector:
โ ALIGN Connector: Minimal drop (โ1.67%) โ proving its high robustness!
โ MLP Connector: Severe degradation (โ25.54%) โ struggling with noisy inputs.
Code & model weights coming soon! Stay tuned! ๐ฅ
๐ Read the paper: AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding (2502.01341)
๐ง Whatโs the challenge?
Aligning visual features with language embeddings remains a major bottleneck in VLMs. Existing connectors such as Multi-layer perceptron (MLPs) often introduce noise that degrades performance. โ
๐ฏ Our Solution: ALIGN Connector
We propose AlignVLM, a method that maps vision features into a weighted average of LLM text embeddings, ensuring they remain in a space that the LLM can effectively interpret. โ
๐ฌ How does it perform?
We compared ALIGN against common connectors like MLPs, Perceiver Resampler, and Ovis trained under similar configurations. The results? ALIGN outperforms them all ๐ on diverse document understanding tasks ๐.
๐ Meet the AlignVLM Model Family!
We trained Llama 3.1 (1B, 3B, 8B) using our connector and benchmarked them against various models. The results:
โ AlignVLM surpasses all Base VLMs trained under similar configurations. โ Our models also perform competitively against Instruct VLMs such as Qwen2-VL and InternVL-2.5 ๐.
๐ค What about robustness to noise?
We injected Gaussian noise (ฮผ=0, ฯ=3) into the vision encoderโs outputs before feeding them to the connector:
โ ALIGN Connector: Minimal drop (โ1.67%) โ proving its high robustness!
โ MLP Connector: Severe degradation (โ25.54%) โ struggling with noisy inputs.
Code & model weights coming soon! Stay tuned! ๐ฅ
published
a
Space
about 1 year ago
commented on
Welcome to Inference Providers on the Hub ๐ฅ
about 1 year ago
Great !!!
commented
a
paper
about 1 year ago
liked
a
Space
about 1 year ago
published
2
Spaces
about 1 year ago
upvoted
a
paper
over 1 year ago
liked
a
Space
over 1 year ago