markus-42/SegFly-Firefly

Introduction

        

Following the acceptance of SegFly at the ECCV 2026 computer vision conference, we not only release the SegFly dataset on HuggingFace, but also our Firefly semantic segmentation model, specialized for RGB and thermal imagery from aerial perspectives. Firefly was trained on our SegFly dataset.

Key Features:

• Aerial-specialized: Fine-tuned on diverse aerial imagery from urban, industrial, and rural environments.
• Multi-altitude performance: Trained on data from 50m, 40m, and 30m altitudes.
• Seasonal robustness: Covers data across all seasons for improved generalization.
• Semantic Categories: Both models segment 15 semantic classes: road, walkway, dirt, gravel, grass, vegetation, tree, ground obstacles, vehicle, water, building, roof, parking lot, constructions, and truck.

Checkpoints

Two pretrained checkpoints are provided:

Variant	Subfolder	Modality	Architecture
Firefly-RGB	Firefly_RGB/	RGB drone imagery	DINOv3 ViT-B/16 + MLP head
Firefly-Thermal	Firefly_Thermal/	Thermal drone imagery	DINOv3 ViT-B/16 + Rein adapter + MLP head

Setup

git clone https://huggingface.co/markus-42/SegFly-Firefly
cd SegFly-Firefly

# clone DINOv3 backbone inside the repository
git clone https://github.com/facebookresearch/dinov3.git

# create virtual environment and install dependencies
uv venv && uv pip install -r requirements.txt

# activate virtual environment
source .venv/bin/activate

Inference

Run single-image inference with infer.py. Weights are auto-detected from the modality:

# RGB model
python infer.py --image example.jpg --modality rgb

# Thermal model
python infer.py --image example_thermal.jpg --modality thermal

The colorized segmentation map is saved to ./infer_output/ by default.

Argument	Type	Default	Description
--image	str	(required)	Path to the input image.
--modality	str	rgb	Model to use: rgb or thermal.
--output	str	./infer_output	Directory for the colorized segmentation output.
--weights_path	str	""	Path to model weights. Auto-detected from modality if not set.
--dinov3_repo_dir	str	./dinov3	Path to the local DINOv3 repository.
--image_size	int	640	Resolution at which the image is fed to the model.

Batch Evaluation

eval.py computes segmentation metrics (mIoU, Frequency-Weighted IoU, Pixel Accuracy) against ground truth masks for a set of images.

# RGB model
python eval.py --data_dir /path/to/dataset --modality rgb

# Thermal model
python eval.py --data_dir /path/to/dataset --modality thermal

--data_dir can point to any directory. The loader walks it recursively and collects images and masks by subfolder name — no specific top-level structure is required:

data_dir/
└── <any nesting>/
    ├── src/           ← RGB images (.png / .jpg / .jpeg)       [--modality rgb]
    ├── thermal_src/   ← thermal images (.png / .jpg / .jpeg)   [--modality thermal]
    └── gt/            ← grayscale segmentation masks (.png, raw SegFly class IDs)

Images and masks are matched by alphabetical sort order, so filenames must correspond 1-to-1.

Append --visualize to save colorized segmentation maps to <output_dir>/visualizations.

Argument	Type	Default	Description
--data_dir	str	./data	Root directory of the dataset. Walked recursively; images are collected from src/ or thermal_src/ subfolders and masks from gt/ subfolders.
--weights_path	str	""	Path to the saved weights. Auto-detected from modality if not set.
--class_dict_path	str	./classes_segfly.csv	Path to the CSV file defining the dataset's class mapping.
--modality	str	rgb	Modality of the dataset: rgb or thermal.
--output_dir	str	./eval_output	Directory to save per_image_iou.csv, per_class_scores.txt, and visualizations.
--visualize	flag	False	If provided, generates side-by-side visualizations of Image, Ground Truth, and Prediction.
--image_size	int	640	Input resolution for evaluation.
--dinov3_repo_dir	str	./dinov3	Path to the local DINOv3 repository.

Reference

If our work was helpful to you, we would appreciate citing our papers and giving the repository a like ❤️

@inproceedings{gross2026segfly,
    title={{SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale}}, 
    author={Markus Gross and Sai Bharadhwaj Matha and Rui Song and Viswanathan Muthuveerappan and Conrad Christoph and Julius Huber and Daniel Cremers},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
    year={2026},
}

Since SegFly is based on the OccuFly dataset, consider citing this work as well:

@inproceedings{gross2026occufly,
    title={{OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective}}, 
    author={Markus Gross and Sai B. Matha and Aya Fahmy and Rui Song and Daniel Cremers and Henri Meess},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2026},
}

License

This work is licensed under the CC BY-NC-SA 4.0 license. See the LICENSE file for the full legal terms.