redis/model-b-structured

SentenceTransformer based on thenlper/gte-small

This is a sentence-transformers model finetuned from thenlper/gte-small. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: thenlper/gte-small
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("redis/model-b-structured")
# Run inference
sentences = [
    'It was Easipower that said :',
    'It was Easipower that said :',
    'It is said that Easipower was ,',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.8522],
#         [1.0000, 1.0000, 0.8522],
#         [0.8522, 0.8522, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Datasets: NanoMSMARCO and NanoNQ
• Evaluated with InformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ
cosine_accuracy@1	0.34	0.54
cosine_accuracy@3	0.56	0.7
cosine_accuracy@5	0.64	0.76
cosine_accuracy@10	0.76	0.8
cosine_precision@1	0.34	0.54
cosine_precision@3	0.1867	0.24
cosine_precision@5	0.128	0.156
cosine_precision@10	0.076	0.086
cosine_recall@1	0.34	0.52
cosine_recall@3	0.56	0.66
cosine_recall@5	0.64	0.71
cosine_recall@10	0.76	0.77
cosine_ndcg@10	0.5416	0.6525
cosine_mrr@10	0.4732	0.6275
cosine_map@100	0.4858	0.614

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nq"
      ],
      "dataset_id": "lightonai/NanoBEIR-en"
  }
  

Metric	Value
cosine_accuracy@1	0.44
cosine_accuracy@3	0.63
cosine_accuracy@5	0.7
cosine_accuracy@10	0.78
cosine_precision@1	0.44
cosine_precision@3	0.2133
cosine_precision@5	0.142
cosine_precision@10	0.081
cosine_recall@1	0.43
cosine_recall@3	0.61
cosine_recall@5	0.675
cosine_recall@10	0.765
cosine_ndcg@10	0.5971
cosine_mrr@10	0.5503
cosine_map@100	0.5499

Training Details

Training Dataset

Unnamed Dataset

• Size: 111,470 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 13.22 tokens max: 44 tokens	min: 6 tokens mean: 90.67 tokens max: 128 tokens	min: 7 tokens mean: 89.65 tokens max: 128 tokens

• Samples:

  anchor	positive	negative
  which state is home to the arizona ice tea beverage company	Arizona Beverage Company Arizona Beverages USA (stylized as AriZona) is an American producer of many flavors of iced tea, juice cocktails and energy drinks based in Woodbury, New York.[2] Arizona's first product was made available in 1992.	Arya Vaishya Arya Vaishya (Arya Vysya) is an Indian caste. Orthodox Arya Vaishyas follow rituals prescribed in the Vasavi Puranam, a religious text written in the late Middle Ages. Their kuladevata is Vasavi. The community were formerly known as Komati Chettiars in Tamil Nadu but now prefer to be referred to as Arya Vaishya.[1]
  when were afro-american and africana studies programs founded in colleges and universities	African-American studies Programs and departments of African-American studies were first created in the 1960s and 1970s as a result of inter-ethnic student and faculty activism at many universities, sparked by a five-month strike for black studies at San Francisco State. In February 1968, San Francisco State hired sociologist Nathan Hare to coordinate the first black studies program and write a proposal for the first Department of Black Studies; the department was created in September 1968 and gained official status at the end of the five-months strike in the spring of 1969. The creation of programs and departments in Black studies was a common demand of protests and sit-ins by minority students and their allies, who felt that their cultures and interests were underserved by the traditional academic structures.	Maze Runner: The Death Cure Maze Runner: The Death Cure was originally set to be released on February 17, 2017, in the United States by 20th Century Fox, but the studio rescheduled the film's release for January 26, 2018 in theatres and IMAX, allowing time for O'Brien to recover from injuries he sustained during filming. The film received mixed reviews from critics and grossed over $284 million worldwide.
  who recorded the song total eclipse of the heart	Bonnie Tyler Bonnie Tyler (born Gaynor Hopkins; 8 June 1951) is a Welsh singer, known for her distinctive husky voice. Tyler came to prominence with the release of her 1977 album The World Starts Tonight and its singles "Lost in France" and "More Than a Lover". Her 1978 single "It's a Heartache" reached number four on the UK Singles Chart, and number three on the US Billboard Hot 100.	Manny Pacquiao vs. Juan Manuel Márquez IV Marquez defeated Pacquiao by knockout with one second remaining in the sixth round. It was named Fight of the Year and Knockout of the Year by Ring Magazine, with round five garnering Round of the Year honors.[2]

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 12,386 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 13.03 tokens max: 44 tokens	min: 6 tokens mean: 89.36 tokens max: 128 tokens	min: 4 tokens mean: 88.87 tokens max: 128 tokens

• Samples:

  anchor	positive	negative
  In early July , Steve Whitley , the criminal father of Harper Whitley and Garrett Whitley , and brother of Benny Cameron .	In early July , Steve Whitley , the criminal father of Harper Whitley and Garrett Whitley , and brother of Benny Cameron .	In early July , Garrett Whitley , who is the criminal father of Harper Whitley and Steve Whitley , and the brother of Benny Cameron , appeared .
  when will the next season of house of cards be released	House of Cards (season 6) The sixth and final season of the American political drama web television series House of Cards was confirmed by Netflix on December 4, 2017, and is scheduled to be released on November 2, 2018. Unlike previous seasons that consisted of thirteen episodes each, the sixth season will consist of only eight. The season will not include former lead actor Kevin Spacey, who was fired from the show due to sexual misconduct allegations.	Wild 'n Out For the first four seasons, the show filmed from Los Angeles/Hollywood and aired on MTV. The first run episodes were suspended as Mr. Renaissance Entertainment became Ncredible Entertainment in 2012. Upon being revived in 2012, the show was produced in New York City and aired on MTV2 during Seasons 5–7, it also returned to that location for Season 9. In 2016, the show returned to airing new episodes on MTV and also for the first time since Season 4, production is in Los Angeles.
  who played the father on father knows best	Father Knows Best The series began August 25, 1949, on NBC Radio. Set in the Midwest, it starred Robert Young as the General Insurance agent Jim Anderson. His wife Margaret was first portrayed by June Whitley and later by Jean Vander Pyl. The Anderson children were Betty (Rhoda Williams), Bud (Ted Donaldson), and Kathy (Norma Jean Nilsson). Others in the cast were Eleanor Audley, Herb Vigran and Sam Edwards. Sponsored through most of its run by General Foods, the series was heard Thursday evenings on NBC until March 25, 1954.	List of To Kill a Mockingbird characters Maycomb children believe he is a horrible person, due to the rumors spread about him and a trial he underwent as a teenager. It is implied during the story that Boo is a very lonely man who attempts to reach out to Jem and Scout for love and friendship, such as leaving them small gifts and figures in a tree knothole. Scout finally meets him at the very end of the book, when he saves the children's lives from Bob Ewell. Scout describes him as being sickly white, with a thin mouth, thin and feathery hair, and grey eyes, almost as if he were blind. During the same night, when Boo whispers to Scout to walk him back to the Radley house, Scout takes a moment to picture what it would be like to be Boo Radley. While standing on his porch, she realizes his "exile" inside his house is really not that lonely.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 8e-05
• weight_decay: 0.005
• max_steps: 1125
• warmup_ratio: 0.1
• fp16: True
• dataloader_drop_last: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 1
• load_best_model_at_end: True
• optim: adamw_torch
• ddp_find_unused_parameters: False
• push_to_hub: True
• hub_model_id: redis/model-b-structured
• eval_on_start: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 8e-05
• weight_decay: 0.005
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: 1125
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 1
• dataloader_prefetch_factor: 1
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: redis/model-b-structured
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_cosine_ndcg@10	NanoNQ_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10
0	0	-	1.9462	0.6259	0.6583	0.6421
0.2874	250	0.3773	0.0669	0.5322	0.6570	0.5946
0.5747	500	0.0787	0.0564	0.5584	0.6307	0.5946
0.8621	750	0.0678	0.0495	0.5390	0.6447	0.5918
1.1494	1000	0.0517	0.0479	0.5416	0.6525	0.5971

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 2.21.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}