Update README.md

README.md

@@ -339,6 +339,113 @@ rankings: [{'corpus_id': 0, 'score': 0.9867772459983826}, {'corpus_id': 1, 'scor
 
 ```
 
+### Using vLLM
+An experimental single-GPU adapter is available for offline
+`LLM.classify()` reranking and optional FastAPI serving. It reuses the original
+checkpoint without adding or modifying model weights.
+
+The adapter has been validated with Python 3.12, vLLM 0.19.1, Transformers
+5.6.2 and CUDA BF16:
+
+```bash
+conda create -n kalm-vllm python=3.12 -y
+conda activate kalm-vllm
+pip install "vllm==0.19.1" "transformers==5.6.2"
+
+hf download KaLM-Embedding/KaLM-Reranker-V1-Large-R2 \
+  --local-dir ./KaLM-Reranker-V1-Large-R2
+pip install ./KaLM-Reranker-V1-Large-R2/vllm_support --no-deps
+export VLLM_PLUGINS=kalm_t5gemma2
+```
+
+If you modify the `vllm_support` source code for some reason 
+(e.g. changing the model path in `vllm_support/src/kalm_t5gemma2_vllm_plugin/constants.py`), 
+please run `pip install ./KaLM-Reranker-V1-Large-R2/vllm_support --no-deps --force-reinstall` to refresh the patch.
+
+Offline Python:
+
+```python
+from kalm_t5gemma2_vllm_plugin import KaLMVLLMReranker
+
+query = "What is the capital of China?"
+documents = [
+    "The capital of China is Beijing.",
+    "Gravity attracts bodies toward one another.",
+]
+
+with KaLMVLLMReranker(
+    "KaLM-Embedding/KaLM-Reranker-V1-Large-R2",
+    query_max_length=512,
+    document_max_length=1024,
+    encoder_chunk_size=4,
+) as reranker:
+    print(reranker.rank(query, documents))
+```
+
+Offline CLI:
+
+```bash
+kalm-vllm-rerank --return-margin
+```
+
+To deploy the online service, install the HTTP dependencies and keep the
+server running in the first terminal:
+
+```bash
+pip install "fastapi>=0.136,<0.137" "uvicorn>=0.46,<0.47"
+export CUDA_VISIBLE_DEVICES=0
+export VLLM_PLUGINS=kalm_t5gemma2
+
+kalm-vllm-serve \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --model KaLM-Embedding/KaLM-Reranker-V1-Large-R2 \
+  --query-max-length 512 \
+  --document-max-length 1024 \
+  --encoder-chunk-size 4 \
+  --max-model-len 2048
+```
+
+In a second terminal, check the server:
+
+```bash
+conda activate kalm-vllm
+kalm-vllm-client --base-url http://127.0.0.1:8000 --health
+```
+
+Use `/rerank` for one query and a list of documents. Results are sorted by
+score:
+
+```bash
+kalm-vllm-client \
+  --base-url http://127.0.0.1:8000 \
+  --endpoint rerank \
+  --json-file ./KaLM-Reranker-V1-Large-R2/vllm_support/examples/rerank_request.json \
+  --return-margin \
+  --top-k 10
+```
+
+Use `/score` to score a batch of independent query-document pairs. Results
+preserve the input order and optional IDs:
+
+```bash
+kalm-vllm-client \
+  --base-url http://127.0.0.1:8000 \
+  --endpoint score \
+  --json-file ./KaLM-Reranker-V1-Large-R2/vllm_support/examples/score_request.json \
+  --return-margin
+```
+
+The default output is `P(yes)`. Set `return_margin=true` to also receive
+`yes_logit - no_logit`; the client flag `--return-margin` applies the same
+setting to a JSON file request. The supported encoder chunk sizes are
+`1, 2, 4, 8, 16, 32`, with `4` as the default.
+
+This adapter uses vLLM's plugin, scheduling and pooling interfaces while the
+T5Gemma2 semantic forward still runs through Transformers. It is not vLLM's
+native HTTP `/score` implementation or a complete vLLM-native kernel port.
+See [the complete installation, API and troubleshooting guide](./vllm_support/README.md).
+
 #### Ablation on multi-stage training
 
 Across all three model sizes and all seven compression ratios, performance on BEIR and MIRACL improves consistently from Stage 1 to Stage 3, demonstrating the effectiveness of our multi-stage training pipeline. Concretely, Stage 1 establishes a robust foundation for document reranking, distillation in Stage 2 substantially improves performance, and Stage 3 yields further modest gains. More importantly, robustness to compression generally improves across the three training stages. For example, from Stage 1 to Stage 3, the performance retention of KaLM-Reranker-V1-Nano at r = 128 relative to r = 2 increases from 92.88% to 93.80% on BEIR and from 90.93% to 92.15% on MIRACL.