A comprehensive list of papers about 'Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities. ACM Computing Surveys, 2026.'.

---

Important

Contributions welcome:

Contact us or submit a pull request for unlisted relevant papers, content clarifications, or categorization adjustments, and update relevant information once your paper is accepted. Thank you!

---

💥 News 💥

• 🔥🔥🔥 Our survey, accepted by ACM Computing Surveys, please cite it or the library if helpful.
• 🔥🔥🔥 We flagged papers using models of size $\geq$ 7B (or small-sized mainstream LLMs) in their experiments.

---

Abstract

Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. To address this gap, this survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and 10+ machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions.

Citation

If you find our paper or this resource helpful, please consider cite:

@article{yang2026ModelMergingSurvey,
  author = {Yang, Enneng and Shen, Li and Guo, Guibing and Wang, Xingwei and Cao, Xiaochun and Zhang, Jie and Tao, Dacheng},
  title = {Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications, and Opportunities},
  year = {2026},
  issue_date = {June 2026},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  volume = {58},
  number = {8},
  issn = {0360-0300},
  url = {https://doi.org/10.1145/3787849},
  doi = {10.1145/3787849},
  journal = {ACM Comput. Surv.},
  month = feb,
  articleno = {216},
  numpages = {41}
}

Thanks!

---

Framework

• 💥 News 💥
• Abstract
• Citation
• Framework
• Survey
• Benchmark/Evaluation
• Advanced Methods
  • Pre-Merging Methods
    • Better Fine-tuning
      • Linearization Fine-tuning
      • Subspace Fine-tuning
      • Sharpness-aware Fine-tuning
      • Others
    • Architecture Transformation
    • Weight Alignment
  • During Merging Methods
    • Basic Merging Methods
    • Weighted-based Merging Methods
    • Subspace-based Merging Method (Sparse or Low-rank Subspace)
    • Routing-based Merging Methods (Dynamic Merging)
    • Post-calibration based Methods
  • Other Merging Methods
  • Theories or Analysis of Model Merging
• Application of Model Merging in Foundation Models
  • Model Merging in Large Language Models
    • Human Preference Alignment for LLMs
    • Detoxification of LLMs
    • Knowledge Editing/Unlearning of LLMs
    • Faster Training of LLMs
    • Faster Reasoning of LLMs
    • Improving Computational Efficiency of MoE-based LLM
    • Mixing Datasets via Model Merging
    • LLM Agent Merging
    • Combine the Capabilities of Expert LLMs
  • Model Merging in Multimodal Large Language Models
    • Model Merging for Multimodal Fusion
    • Model Merging for Cross-Modal Knowledge Transfer
    • Combine the Capabilities of Expert MLLMs
  • Model Merging in Image Generative Models
    • Style Mixing in Generative Models
    • Reducing Training Cost of Generative Models
    • Enhancing the Faithfulness (or Generation Quality) of Diffusion Models
    • Deepfake Detection
  • Model Merging in Video Generative Models
    • Enhancing Motion Modeling
• Application of Model Merging in Different Machine Learning Subfields
  • Model Merging in Continual Learning
    • Model Merging to Mitigate Catastrophic Forgetting
  • Model Merging in Multi-Task/Multi-Objective/Multi-Domain/Auxiliary Learning
    • Model Merging for Knowledge Transfer in Multi-Task Learning
    • Model Merging for Knowledge Transfer in Multi-Objective Optimization
    • Model Merging for Knowledge Transfer in Multi-Domain Learning
    • Model Merging for Knowledge Transfer in Auxiliary Learning
  • Model Merging in Out-of-Distribution/Domain Generalization
    • Model Merging for Better Out-of-Distribution Generalization
    • Model Merging for Better Domain Generalization or Domain Adaptation
  • Model Merging in Federated Learning
    • Model Merging for Local Knowledge Aggregation
  • Model Merging in Zero-shot/Few-shot Learning
    • Model Merging for Cross-task Generalization in Zero-shot Learning
    • Model Merging for Cross-task Generalization in Few-shot Learning
  • Model Merging in Adversarial Learning
    • Model Merging as an Attack
    • Model Merging as a Defense or Intellectual Property Protection
• Other Applications
• Contact

---

Survey

Paper Title	Year	Conference/Journal
Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions	2026	Arxiv
Scaling Intelligence Through Model Merging: A Comprehensive Survey	2025	Arxiv
Democratizing AI Through Model Fusion: A Comprehensive Review and Future Directions	2025	Arxiv
From Task-Specific Models to Unified Systems: A Review of Model Merging Approaches	2025	Arxiv
SoK: On Finding Common Ground in Loss Landscapes Using Deep Model Merging Techniques	2024	Arxiv
Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities	2024	Arxiv
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning	2024	Arxiv
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models	2024	Arxiv
Learn From Model Beyond Fine-Tuning: A Survey	2023	Arxiv
Deep Model Fusion: A Survey	2023	Arxiv

Benchmark/Evaluation

Paper Title	Year	Conference/Journal	Remark
An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation	2025	Arxiv	LLAMA-2-7B, LLAMA-3-8B, LLAMA-3.1-8B, QWEN2-7B
A Systematic Study of Model Merging Techniques in Large Language Models	2025	Arxiv	Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B, Qwen3-8B
FusionBench: A Comprehensive Benchmark of Deep Model Fusion	2025	JMLR	Mistral-7B-v0.1, MetaMath-Mistral-7B, dolphin-2.1-mistral-7b, speechless-code-mistral-7b-v1.0
Towards Performance Consistency in Multi-Level Model Collaboration	2025	ICCV
Model Merging Scaling Laws in Large Language Models	2025	Arxiv	Qwen2.5 0.5, 1.5, 3, 7, 14, 32, 72B
FBMS: An R Package for Flexible Bayesian Model Selection and Model Averaging	2025	Arxiv
Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging	2025	Arxiv	Qwen2-VL-7B-Base, Vicuna-7B-v1.5
MergeBench: A Benchmark for Merging Domain-Specialized LLMs	2025	Arxiv	Llama-3.2-3B, Llama3.1-8B, Gemma-2-2B and Gemma-2-9B
Mergenetic: a Simple Evolutionary Model Merging Library	2025	System Demonstrations	Mistral-7B
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness	2025	NeurIPS	LLaVA-v1.5-7B
Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging	2025	Arxiv	Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2
How to Merge Your Multimodal Models Over Time?	2024	Arxiv
Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning	2024	Arxiv	Aya 23 8B
A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Models	2024	Arxiv	LLaMA3-8B-Instruct, Qwen2-7B-Instruct, Mistral-7B-Instruct-v0.3,
Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild	2024	NeurIPS Track on Datasets and Benchmarks	Synthia-7B-v1.2, Llama-2-7b-evolcodealpaca, OpenHermes-7B, pygmalion-2-7b, Llama-2-7b-chat-hf, BeingWell_llama2_7b, MetaMath-7B-V1.0, vicuna-7b-v1.5, Platypus2-7B, GOAT-7B-Community, Llama-2-7b-WikiChat-fused, dolphin-llama2-7b, MetaMath-Llemma-7B, CodeLlama-7b-Instruct-hf, Magicoder-S-CL-7B, CrystalChat
What Matters for Model Merging at Scale?	2024	Arxiv	PaLM-2 (1B, 8B, 24B, 64B), PaLM-2-IT (1B, 8B, 24B, 64B)
Realistic Evaluation of Model Merging for Compositional Generalization	2024	Arxiv
Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities	2024	Arxiv	Llama-3.1-8B, Mistral-7B-v0.3
Arcee's MergeKit: A Toolkit for Merging Large Language Models	2024	Arxiv	Llama2-7B-Chat, Meditron-7B

Advanced Methods

Pre-Merging Methods

Better Fine-tuning

Linearization Fine-tuning

Paper Title	Year	Conference/Journal	Remark
Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature	2026	ICLR
Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic	2025	ICLR
Tangent Transformers for Composition,Privacy and Removal	2024	ICLR
Parameter Efficient Multi-task Model Fusion with Partial Linearization	2024	ICLR
Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models	2023	NeurIPS

Subspace Fine-tuning

Paper Title	Year	Conference/Journal	Remark
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging	2025	Arxiv	Llama3-8B
Efficient Model Editing with Task-Localized Sparse Fine-tuning	2025	ICLR

Sharpness-aware Fine-tuning

Paper Title	Year	Conference/Journal	Remark
Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning	2025	ICLR

Others

Paper Title	Year	Conference/Journal	Remark
MergOPT: A Merge-Aware Optimizer for Robust Model Merging	2026	ICLR	Llama3.1-8B-Instruct

Architecture Transformation

Paper Title	Year	Conference/Journal	Remark
Model Assembly Learning with Heterogeneous Layer Weight Merging	2025	ICLR Workshop
Training-free Heterogeneous Model Merging	2025	Arxiv
Knowledge fusion of large language models	2024	ICLR	Llama-2 7B, OpenLLaMA 7B, MPT 7B
Knowledge Fusion of Chat LLMs: A Preliminary Technical Report	2024	Arxiv	NH2-Mixtral-8x7B, NH2-Solar-10.7B, and OpenChat-3.5-7B
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks	2023	ICASSP
GAN Cocktail: mixing GANs without dataset access	2022	ECCV

Weight Alignment

Paper Title	Year	Conference/Journal	Remark
Transport and Merge: Cross-Architecture Merging for Large Language Models	2026	Arxiv	LLaMA-3 8B
Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization	2025	TAG-DS
Understanding Mode Connectivity via Parameter Space Symmetry	2025	ICML
Update Your Transformer to the Latest Release: Re-Basin of Task Vectors	2025	ICML
Model Assembly Learning with Heterogeneous Layer Weight Merging	2025	ICLR Workshop
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion	2025	Arxiv
The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse	2024	Arxiv
Equivariant Deep Weight Space Alignment	2024	ICML
Harmony in diversity: Merging neural networks with canonical correlation analysis	2024	ICML
Transformer fusion with optimal transport	2024	ICLR
Layerwise linear mode connectivity	2024	ICLR
ZipIt! Merging Models from Different Tasks without Training	2024	ICLR
Proving linear mode connectivity of neural networks via optimal transport	2024	AISTATS
Training-Free Pretrained Model Merging	2024	CVPR
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering	2024	Arxiv	Llama2-7b, Llama2-13b
C2M3: Cycle-Consistent Multi Model Merging	2024	NeurIPS
PLeaS--Merging Models with Permutations and Least Squares	2024	Arxiv
Rethink Model Re-Basin and the Linear Mode Connectivity	2024	Arxiv
Git Re-Basin: Merging Models modulo Permutation Symmetries	2023	ICLR
Re-basin via implicit Sinkhorn differentiation	2023	CVPR
Plateau in Monotonic Linear Interpolation--A "Biased" View of Loss Landscape for Deep Networks	2023	ICLR
Linear Mode Connectivity of Deep Neural Networks via Permutation Invariance and Renormalization	2023	ICLR
REPAIR: REnormalizing Permuted Activations for Interpolation Repair	2023	ICLR
Going beyond linear mode connectivity: The layerwise linear feature connectivity	2023	NeurIPS
The role of permutation invariance in linear mode connectivity of neural networks	2022	ICLR
What can linear interpolation of neural network loss landscapes tell us?	2022	ICML
Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling	2021	ICML
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes	2021	ICML
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances	2021	ICML
Linear Mode Connectivity and the Lottery Ticket Hypothesis	2020	ICML
Optimizing mode connectivity via neuron alignment	2020	NeurIPS
Model fusion via optimal transport	2020	NeurIPS
Uniform convergence may be unable to explain generalization in deep learning	2019	NeurIPS
Explaining landscape connectivity of low-cost solutions for multilayer nets	2019	NeurIPS
Essentially no barriers in neural network energy landscape	2018	ICML
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs	2018	NeurIPS

During Merging Methods

Basic Merging Methods

Paper Title	Year	Conference/Journal	Remark
Composing parameter-efficient modules with arithmetic operation	2023	NeurIPS
Editing models with task arithmetic	2023	ICLR
Model fusion via optimal transport	2020	NeurIPS
Weight averaging for neural networks and local resampling schemes	1996	AAAI Workshop
Acceleration of stochastic approximation by averaging	1992	IAM Journal on Control and Optimization
Animating rotation with quaternion curves (Spherical Linear Interpolation (SLERP) Model Merging)	1985	SIGGRAPH Computer Graphics

Weighted-based Merging Methods

Paper Title	Year	Conference/Journal	Remark
The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging	2026	Arxiv
LARV: Data-Free Layer-wise Adaptive Rescaling Veneer for Model Merging	2026	Arxiv
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance	2025	Arxiv	xLAM-2-70b, CoALM-70B, watt-tool-70B, functionary-medium-70B, xLAM-2-8b, ToolACE-2-8B, watt-tool-8B, BitAgent-8B, CoALM-8B
Superpose Task-specific Features for Model Merging	2025	EMNLP	Llama-2-7B
T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis	2025	Arxiv
Weight Weaving: Parameter Pooling for Data-Free Model Merging	2025	Arxiv
Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking	2025	Arxiv	Mistral-7B, InternVL, Qwen2-VL
Variational Task Vector Composition	2025	NeurIPS
RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging	2025	Arxiv
StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation	2025	Arxiv
SeMe: Training-Free Language Model Merging via Semantic Alignment	2025	Arxiv
NAN: A Training-Free Solution to Coefficient Estimation in Model Merging	2025	Arxiv	LLaMA2-13B, WizardLM-13B, WizardMath-13B, LLaVA-v1.5-13B, LLaVA-1.6-13B, Math-LLaVA
Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs	2025	ICLR	Llama-2-7B and Llama-2-13B
Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge	2025	Arxiv	Gemma-2-9B, Llama-3-8B
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models	2025	Arxiv	LLaMA-2 7B series, Mistral 7B series, LLaMA-2 13B series
RankMean: Module-Level Importance Score for Merging Fine-tuned Large Language Models	2024	ACL
Non-Uniform Parameter-Wise Model Merging	2024	Arxiv
How to Weight Multitask Finetuning? Fast Previews via Bayesian Model-Merging	2024	Arxiv
LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging	2024	Arxiv
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation	2024	Arxiv	shisa-gamma-7b, WizardMath-7B-V1.1, Abel-7B-002, Llama-3-SauerkrautLM-8b-Instruct, Llama-3-Open-Ko-8B, llama-3-sqlcoder-8b, Meta-Llama-3-8B
Knowledge Composition using Task Vectors with Learned Anisotropic Scaling	2024	Arxiv
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic	2024	EMNLP	LLaMA-2-7B, Mistral-7B, LLaMA-2-13B
Checkpoint Merging via Bayesian Optimization in LLM Pretraining	2024	Arxiv	Baichuan2-220B, Baichuan2-440B, Baichuan2-660B, Baichuan2-1540B, Baichuan2-1760B, Baichuan2-1980B, Baichuan2-2200B, Baichuan2-2420B, DeepSeek-1400B, DeepSeek-1600B, DeepSeek-1800B, DeepSeek-2000B
Arcee’s MergeKit: A Toolkit for Merging Large Language Models	2024	Arxiv	Llama2-7B-Chat, Meditron-7B
Evolutionary optimization of model merging recipes	2024	Arxiv	shisa-gamma-7b-v1, WizardMath-7B-V1.1, Arithmo2-Mistral-7B, Abel-7B-002, Mistral-7B-v0.1, LLaVA-1.6-Mistral-7B
XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts	2024	ACL
AdaMerging: Adaptive Model Merging for Multi-Task Learning	2024	ICLR
Model Merging by Uncertainty-Based Gradient Matching	2024	ICLR
Merging by Matching Models in Task Subspaces	2024	TMLR
Fisher Mask Nodes for Language Model Merging	2024	LREC-COLING
Erasure Coded Neural Network Inference via Fisher Averaging	2024	ISIT
Dataless Knowledge Fusion by Merging Weights of Language Models	2023	ICLR
Merging models with fisher-weighted averaging	2022	NeurIPS

Subspace-based Merging Method (Sparse or Low-rank Subspace)

Paper Title	Year	Conference/Journal	Remark
DC-Merge: Improving Model Merging with Directional Consistency	2026	CVPR	LLaVA
CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging	2026	Arxiv	Qwen3-8B and Llama3.1-8B
Model Merging in the Essential Subspace	2026	Arxiv
Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging	2026	Arxiv	Mistral-7B, Qwen2.5-14B, and Qwen2.5-32B
Orthogonal Model Merging	2026	Arxiv	Llama-3.1-8B, Qwen2.5-VL-7B-Instruct, Llama-3.2-3B
When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging	2026	Arxiv
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations	2026	Arxiv	Qwen2.5-7B, Qwen2.5-14B
AdaRank: Adaptive Rank Pruning for Enhanced Model Merging	2026	ICLR
Decomposing Task Vectors for Refined Model Editing	2025	Arxiv
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging	2025	Arxiv	Qwen-14B
Towards Reversible Model Merging For Low-rank Weights	2025	Arxiv
Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging	2025	Arxiv	LLaMA-2-7B
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness	2025	NeurIPS	LLaVA
Accurate and Efficient Low-Rank Model Merging in Core Space	2025	NeurIPS
Efficient Multi-Source Knowledge Transfer by Model Merging	2025	Arxiv
One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging	2025	Arxiv
NegMerge: Sign-Consensual Weight Merging for Machine Unlearning	2025	ICML
Subspace-Boosted Model Merging	2025	Arxiv
Training-free LLM Merging for Multi-task Learning	2025	Arxiv
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data	2025	Arxiv
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs	2025	Arxiv	Mistral-7B, Llama3-8B
CALM: Consensus-Aware Localized Merging for Multi-Task Learning	2025	ICML
Merge-Friendly Post-Training Quantization for Multi-Target Domain Adaptation	2025	ICML
Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation	2025	ACL	Llama-3-8B-Instruct
Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking	2025	Arxiv	LLaMA3.1-8B
CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging	2025	Arxiv
LoRI: Reducing Cross-Task Interference in Multi-Task LowRank Adaptation	2025	Arxiv	Llama-3-8B and Mistral-7B
Task Vector Quantization for Memory-Efficient Model Merging	2025	Arxiv
Disentangling Task Interference within Neurons: Model Merging in Alignment with Neuronal Mechanisms	2025	Arxiv	Llama-2-7b
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts	2025	ICLR 2025 Workshop
LEWIS (LayEr WIse Sparsity) -- A Training Free Guided Model Merging Approach	2025	ICLR 2025 Workshop	Gemma-9b, LLaMA 3.1 8b
CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging	2025	Arxiv	Mistral-7b-v0.1, WildMarcoroni-Variant1-7B and WestSeverus-7B-DPO-v2
Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation	2025	Arxiv
LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint	2025	Arxiv	Llama-3- 8B, Mistral-7B, and Llama2-13B
Parameter Efficient Merging for Multimodal Large Language Models with Complementary Parameter Adaptation	2025	Arxiv
Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging	2025	Arxiv	Llama-2-13b, WizardMath-13B-V1.0, WizardLM13B-V1.2, llama-2-13b-codealpaca
Superpose Singular Features for Model Merging	2025	Arxiv	Llama-2-7B
STAR: Spectral Truncation and Rescale for Model Merging	2025	NAACL	Mistral-7B-Instruct
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces	2025	Arxiv
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging	2025	NeurIPS
Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent	2025	Arxiv
Revisiting Weight Averaging for Model Merging	2024	Arxiv
Task Singular Vectors: Reducing Task Interference in Model Merging	2025	CVPR
Less is More: Efficient Model Merging with Binary Task Switch	2024	Arxiv
FREE-Merging: Fourier Transform for Model Merging with Lightweight Experts	2024	Arxiv	Qwen-14B (LoRA), LLaMa2-13B, WizardLM-13B, WizardMath-13B, WizardCoderPython-13B
Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics	2024	Arxiv
Parameter Competition Balancing for Model Merging	2024	NeurIPS	Llama-2-7b
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch	2024	ICML	WizardLM-13B, WizardMath-13B, and llama-2-13b-codealpaca, Mistral-7B
Localizing Task Information for Improved Model Merging and Compression	2024	ICML
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging	2024	ICLR
Model merging with svd to tie the knots	2024	Arxiv	Llama3-8B
NegMerge: Consensual Weight Negation for Strong Machine Unlearning	2024	Arxiv
Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic	2024	Arxiv
Activated Parameter Locating via Causal Intervention for Model Merging	2024	Arxiv	Llama-2-chat-7B
PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning	2024	Arxiv	Mistral-7B-v0.1, Llama-3-8B, Neurotic-7B, MoMo-70B
DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling	2024	Arxiv	Llama-2-13b-code-alpaca, WizardLM, Wizard-Math, WizardCoder-Python
EMR-Merging: Tuning-Free High-Performance Model Merging	2024	NeurIPS
DPPA: Pruning Method for Large Language Model to Model Merging	2024	Arxiv	LLaMa 2
Model breadcrumbs: Scaling multi-task model merging with sparse masks	2023	Arxiv
Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion	2023	Arxiv
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization	2023	Arxiv	LLaMA 7B, 13B, 33B, and 65B
Effective and ParameterEfficient Reusing Fine-Tuned Models	2023	Openreview
Resolving Interference When Merging Models	2023	NeurIPS
Task-Specific Skill Localization in Fine-tuned Language Model	2023	ICML

Routing-based Merging Methods (Dynamic Merging)

Paper Title	Year	Conference/Journal	Remark
Fine-Grained Model Merging via Modular Expert Recombination	2026	Arxiv
MIN-Merging: Merge the Important Neurons for Model Merging	2025	Arxiv
SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging	2025	Arxiv
Adaptive Task Vectors for Large Language Models	2025	Arxiv	LLaMA3-8B and Mistral-7B
Dynamic Fisher-weighted Model Merging via Bayesian Optimization	2025	Arxiv
Data-Adaptive Weight-Ensembling for Multi-task Model Fusion	2025	IJCV
MASS: MoErging through Adaptive Subspace Selection	2025	Arxiv
Dynamic Model Merging with Mixture of Weights	2025	TCSVT
CAMEx: Curvature-aware Merging of Experts	2025	ICLR
1bit-Merging: Dynamic Quantized Merging for Large Language Models	2025	Arxiv	LLaMA-2 7B, Mistral 7B, and LLaMA-2 13B
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs	2025	Arxiv
Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing	2025	Arxiv	Qwen-2.5-7B, LLaMA-3.2-8B
Adapting Foundation Models via Training-free Dynamic Weight Interpolation	2024	NeurIPS 2024 Workshop
Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging	2024	Arxiv
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation	2024	NeurIPS 2024 Workshop
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts	2024	ICML
Learning to Route Among Specialized Experts for Zero-Shot Generalization	2024	ICML
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy	2024	ICLR
Soft merging of experts with adaptive routing	2024	TMLR
SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models	2024	Arxiv	Mistral-7B-v0.1, MetaMath-Mistral-7B, dolphin-2.1-mistral-7b, speechless-code-mistral-7b-v1.0
Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging	2024	NeurIPS	Qwen-14B
Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts	2024	Arxiv	Gemma-7B, LLaMA-2 7B & 13B, Mistral 7B, LLaMA-3 8B
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion	2024	Arxiv
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints	2023	ICLR

Post-calibration based Methods

Paper Title	Year	Conference/Journal	Remark
MAGIC: Achieving Superior Model Merging via Magnitude Calibration	2025	Arxiv
Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration	2025	NeurIPS
Multi-Task Model Fusion via Adaptive Merging	2025	ICASSP
Representation Surgery in Model Merging with Probabilistic Modeling	2025	ICML
Parameter-Efficient Interventions for Enhanced Model Merging	2024	Arxiv
Tint Your Models Task-wise for Improved Multi-task Model Merging	2024	Arxiv
SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery	2024	Arxiv
Representation Surgery for Multi-Task Model Merging	2024	ICML

Other Merging Methods

Paper Title	Year	Conference/Journal	Remark
BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning	2026	Arxiv
ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation	2026	Arxiv
Training-Free Cross-Architecture Merging for Graph Neural Networks	2026	Arxiv
Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models	2026	ICLR	Flan-T5
Transporting Task Vectors across Different Architectures without Training	2026	Arxiv
MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging	2026	Arxiv	Llama3.1-8B, Llama-3.2-3B, Qwen3-0.6B, Qwen3-1.7B, and Qwen3-8B
DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging	2026	ICLR
Sparsity-Aware Evolution for Model Merging	2026	Arxiv
AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse	2026	Arxiv	Llama2-7B-Chat, Llama2-7B-Code
Model Merging via Multi-Teacher Knowledge Distillation	2025	Arxiv
Bridging Training and Merging Through Momentum-Aware Optimization	2025	Arxiv
From Coefficients to Directions: Rethinking Model Merging with Directional Alignment	2025	Arxiv
Escaping Optimization Stagnation: Taking Steps Beyond Task Arithmetic via Difference Vectors	2025	Arxiv
Model Merging with Functional Dual Anchors	2025	Arxiv
Black-box Model Merging for Language-Model-as-a-Service with Massive Model Repositories	2025	Arxiv
Rethinking Layer-wise Model Merging through Chain of Merges	2025	Arxiv	Llama 3-8B
Competition and Attraction Improve Model Fusion	2025	Arxiv	WizardMath 7B v1.0, AgentEvol 7B
PSO-Merging: Merging Models Based on Particle Swarm Optimization	2025	Arxiv	Llama-3-8B, Llama-2-13B, and Mistral-7B-v0.3
DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging	2025	Arxiv
Navigating the Accuracy-Size Trade-Off with Flexible Model Merging	2025	Arxiv
Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment	2025	Arxiv
Reinforced Model Merging	2025	Arxiv
FW-Merging: Scaling Model Merging with Frank-Wolfe Optimization	2025	Arxiv	LLaMA2-7B
Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors	2025	Arxiv	WizardLM-13B (LM), WizardMath-13B (Math), and llama-2-13bcodealpaca (Code)
GNNMERGE: Merging of GNN Models Without Accessing Training Data	2025	Arxiv
MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUs	2025	ICML	Mistral-7B
Activation-Informed Merging of Large Language Models	2025	Arxiv	Llama-2-13b, WizardLM-13B, WizardMath-13B, llama-2-13b-code-alpaca
Scalable Model Merging with Progressive Layer-wise Distillation	2025	Arxiv	WizardLM-13B, WizardMath-13B and llama-2-13b-code-alpaca
Fine, I’ll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging	2025	Arxiv	Llama-2-13, WizardLM13B, WizardMath-13, llama-2-13b-code-alpaca
Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts	2025	ICLR
Fine-tuning Aligned Classifiers for Merging Outputs: Towards a Superior Evaluation Protocol in Model Merging	2024	Arxiv
Multi-Task Model Merging via Adaptive Weight Disentanglement	2024	Arxiv
Rethinking Weight-Averaged Model-merging	2024	Arxiv
ATM: Improving Model Merging by Alternating Tuning and Merging	2024	Arxiv
HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models	2024	Arxiv	Llama-2-7B-Chat, WizardMath-7B, CodeLlama-7B
Weight Scope Alignment: A Frustratingly Easy Method for Model Merging	2024	Arxiv
It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization	2024	Arxiv	Qwen1.5-7B-Chat, Liberated-Qwen1.5-7B, firefly-qwen1.5-en-7B
Toward Data Efficient Model Merging between Different Datasets without Performance Degradation	2024	JMLR
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling	2023	Arxiv	SOLAR 10.7B, SOLAR 10.7B-Instruct

Theories or Analysis of Model Merging

Paper Title	Year	Conference/Journal	Remark
An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse	2026	Arxiv	Qwen2.5-3B, 7B, and 14B, Llama3.1-8B
Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts	2026	Arxiv
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs	2026	ICLR	Llama-3.2-3B, Llama-3.1-8B, and Mistral-Small-3-24B
M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Data	2026	Arxiv
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training	2026	ICLR	Ling-mini-16B
Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success	2026	Arxiv
Understanding Model Merging: A Unified Generalization Framework for Heterogeneous Experts	2026	Arxiv
Will it Merge? On The Causes of Model Mergeability	2026	Arxiv	Llama-3.2-3B、Qwen-2.5-3B、Mistral-7B-Instruct-v0.2
How does the optimizer implicitly bias the model merging loss landscape?	2025	Arxiv
On Task Vectors and Gradients	2025	Arxiv
Why Do More Experts Fail? A Theoretical Analysis of Model Merging	2025	Arxiv
When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers	2025	ICLR
Multi-Level Collaboration in Model Merging	2025	Arxiv
Low-rank bias, weight decay, and model merging in neural networks	2025	Arxiv
Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression	2025	Arxiv
SeWA: Selective Weight Average via Probabilistic Masking	2025	Arxiv
Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach	2025	Arxiv
Task Arithmetic Through The Lens Of One-Shot Federated Learning	2024	Arxiv	WizardLM-13B, WizardMath-13B, Llama-2-13B-Code-Alpaca, Llama2-13B
A Unified Analysis for Finite Weight Averaging	2024	Arxiv
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average	2024	Arxiv
On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm	2024	ICML
Generalization Analysis of Stochastic Weight Averaging with General Sampling	2024	ICML
Diverse weight averaging for out-of-distribution generalization	2022	NeurIPS
Ensemble of averages: Improving model selection and boosting performance in domain generalization	2022	NeurIPS
Stability analysis and generalization bounds of adversarial training	2022	NeurIPS
The role of permutation invariance in linear mode connectivity of neural networks	2022	ICLR
Swad: Domain generalization by seeking flat minima	2021	NeurIPS
Linear Mode Connectivity and the Lottery Ticket Hypothesis	2020	ICML
Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes	2020	ICLR
Optimizing mode connectivity via neuron alignment	2020	NeurIPS
Uniform convergence may be unable to explain generalization in deep learning	2019	NeurIPS
Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification	2018	JMLR
Iterate averaging as regularization for stochastic gradient descent	2018	Arxiv
Essentially no barriers in neural network energy landscape	2018	ICML
Averaging weights leads to wider optima and better generalization	2018	UAI
Train faster, generalize better: Stability of stochastic gradient descent	2016	ICML

---

Application of Model Merging in Foundation Models

Model Merging in Large Language Models

Human Preference Alignment for LLMs

Paper Title	Year	Conference/Journal	Remark
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging	2025	Arxiv	Gemma-3-12B, Gemma-3-27B, Qwen2.5-7B
BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generation	2025	Arxiv	Qwen-2.5-7B-Instruct, Llama-3.1-8B-Instruct
Personality Vector: Modulating Personality of Large Language Models by Model Merging	2025	EMNLP	Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct
SafeMERGE: Preserving Safety Alignment in Fine-Tuned LLMs via Selective Layer-Wise Model Merging	2025	Arxiv	Llama-2-7B-Chat, Qwen-2-7B-Instruct
Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation	2025	Arxiv	LLaMA-2 7B
Model soup for better rlhf: Weight space averaging to improve alignment in llms	2024	NeurIPS 2024 Workshop	Llama2-7B, Mistral-7B, Gemma-2B
Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging	2024	Arxiv	Llama-3-8B-Instruct
SafetyDPO: Scalable Safety Alignment for Text-to-Image Generation	2024	Arxiv
H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs	2024	Arxiv	LLaMA-2 7B
Baichuan Alignment Technical Report	2024	Arxiv	Qwen2-Nova-72B, Llama3-PBM-Nova-70B
Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning	2024	Arxiv
DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging	2024	Arxiv	MetaMath-7B, MAmmoTH-7B, LLaMA2-7B
PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning	2024	Arxiv	Mistral-7B-v0.1, Llama-3-8B
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch	2024	Arxiv	Mistral-0.2-7B-Instruct, LLaMA-3-8B-Instruct, OpenBioLLM-8B, MAmmoTH2-7B, WizardMath-1.1-7B
Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching	2024	Arxiv	LLaMA-2-7B-Chat, LLaMA-3-8B-Instruct, Mistral7B-Instruct-v0.1 and Gemma1.1-7B-it
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction	2024	Arxiv	Llama-2-7b
Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment	2024	Arxiv	Qwen1.5-7B, LLaMa3-8B
A safety realignment framework via subspace-oriented model fusion for large language models	2024	Arxiv	WizardLM-7B
Weak-to-strong extrapolation expedites alignment	2024	Arxiv	zephyr-7b, starling-7b, snorkel-7b, llama3-8b, internlm2-7b, internlm2-20b, tulu-2-dpo-7b, tulu-2-dpo-13b, tulu-2-dpo-70b
Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic	2024	Arxiv	Llama-2-7BChat
Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards	2023	NeurIPS	LLaMA-7b
Personalized soups: Personalized large language model alignment via post-hoc parameter merging	2023	Arxiv	Tulu-7B LM

Detoxification of LLMs

Paper Title	Year	Conference/Journal	Remark
Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation	2025	ICLR	GEMMA-7B-IT, LLAMA2-7B/13B/70B-CHAT, LLAMA3-8B-INST
3DM: Distill, Dynamic Drop, and Merge for Debiasing Multi-modal Large Language Models	2025	ACL	LLaVA-1.5-7b, InternVL-2.5-8b, LLaVA-1.5-7b and ChatGLM4-9b
Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation	2025	Arxiv	LLAMA3-8B-Instruct, Mistral-7B-Instruct-v0.2
Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach	2024	Arxiv
Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation	2024	AAAI	LLaMA-7B
Mitigating Social Biases in Language Models through Unlearning	2024	Arxiv	LLaMA-2 7B
Fine-Grained Detoxification via Instance-Level Prefixes for Large Language Models	2024	Arxiv	Llama-2-7B, Llama-2-chat-7B, Vicuna-7B, Llama-2-13B
Composing Parameter-Efficient Modules with Arithmetic Operation	2023	NeurIPS
Editing models with task arithmetic	2023	ICLR
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation	2023	Arxiv

Knowledge Editing/Unlearning of LLMs

Paper Title	Year	Conference/Journal	Remark
Per-parameter Task Arithmetic for Unlearning in Large Language Models	2026	Arxiv	Llama3.2 1B Instruct
Model Merging for Knowledge Editing	2025	ACL	Qwen2.5-7B-Instruct
Exact Unlearning of Finetuning Data via Model Merging at Scale	2025	Arxiv
ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging	2025	Arxiv	OLMo-7B-0724-Instruct
Exact Unlearning of Finetuning Data via Model Merging at Scale	2025	ICLR 2025 Workshop MCDC
NegMerge: Consensual Weight Negation for Strong Machine Unlearning	2024	Arxiv
Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs	2024	Arxiv	ZEPHYR-7B-BETA, LLAMA2-7B
Towards Safer Large Language Models through Machine Unlearning	2024	ACL	LLAMA2-7B, LLAMA2-13B
Editing models with task arithmetic	2023	ICLR
Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Model	2023	Arxiv	LLAMA2-7B, LLAMA-7B, BLOOM-7B
Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion	2023	Arxiv

Faster Training of LLMs

Paper Title	Year	Conference/Journal	Remark
Mashup Learning: Faster Finetuning by Remixing Past Checkpoints	2026	Arxiv
GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training	2025	Arxiv	Qwen2.5-VL-7B
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging	2025	ICML
Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging	2025	Arxiv
Merge to Mix: Mixing Datasets via Model Merging	2025	Arxiv	Llama-3-8B-Instruct
Model Merging in Pre-training of Large Language Models	2025	Arxiv	Seed-MoE-1.3B/13B, SeedMoE-10B/100B, Seed-MoE-15B/150B
Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging	2025	Arxiv
DEM: Distribution Edited Model for Training with Mixed Data Distributions	2024	Arxiv	OpenLLaMA 7B and 13B
Checkpoint Merging via Bayesian Optimization in LLM Pretraining	2024	Arxiv	Baichuan2-220B, Baichuan2-440B, Baichuan2-660B, Baichuan2-1540B, Baichuan2-1760B, Baichuan2-1980B, Baichuan2-2200B, Baichuan2-2420B, DeepSeek-1400B, DeepSeek-1600B, DeepSeek-1800B, DeepSeek-2000B
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning	2023	ACL
Early Weight Averaging meets High Learning Rates for LLM Pre-training	2023	NeurIPS Workshop
Stop wasting my time! saving days of imagenet and bert training with latest weight averaging	2022	NeurIPS Workshop
Fusing finetuned models for better pretraining	2022	Arxiv

Faster Reasoning of LLMs

Paper Title	Year	Conference/Journal	Remark
RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format	2026	ICLR	Qwen2.5-1.5B/14B/32B, and Llama-3.1-8B
Reasoning Pattern Alignment Merging for Adaptive Reasoning	2026	Arxiv	(i) Qwen3-4B-Thinking (Long-CoT) and Qwen3-4B-Instruct (Short-CoT); (ii) DeepSeekR1-Distill-Qwen-1.5B (Long-CoT) and Qwen2.5- Math-1.5B (Short-CoT)
Revisiting Model Interpolation for Efficient Reasoning	2025	Arxiv	Qwen3-4B
Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging	2025	Arxiv	Qwen2.5-32B, DeepSeek-R1-32B
Kimi k1.5: Scaling Reinforcement Learning with LLMs	2025	Arxiv	Kimi k1.5

Improving Computational Efficiency of MoE-based LLM

Paper Title	Year	Conference/Journal	Remark
Upcycled and Merged MoE Reward Model for Mitigating Reward Hacking	2025	Arxiv
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference	2025	Arxiv	Mixtral-8x7B, Deepseek-MoE
Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models	2025	Arxiv	LLaMA-3.1-8B
Expert Merging in Sparse Mixture of Experts with Nash Bargaining	2025	Arxiv	Qwen1.5-MoE-14B, DeepSeek-MoE-16B
MergeMoE: Efficient Compression of MoE Models via Expert Output Merging	2025	Arxiv	DeepSeekMoE, Qwen1.5-MoE-A2.7B, and Qwen3-30B-A3B
Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inference	2025	Arxiv
Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging	2025	Arxiv	Mixtral 8x7B, Qwen3- 235B-A22B, Qwen1.5-MoE-A2.7B, and DeepSeekMoE-16B-Base
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts	2023	EMNLP

Mixing Datasets via Model Merging

Paper Title	Year	Conference/Journal	Remark
Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization	2026	Arxiv	Qwen2-VL-2B and Intern3.5-VL-2B
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training	2026	Arxiv	Qwen3-1.7B
Multi-task Code LLMs: Data Mix or Model Merge?	2026	Arxiv	Qwen Coder 2.5 7B, DeepSeek 7B
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging	2026	Arxiv	8B and 16B MoE
Merge to Mix: Mixing Datasets via Model Merging	2025	Arxiv	Llama-3-8B-Instruct

LLM Agent Merging

Paper Title	Year	Conference/Journal	Remark
Behavior Knowledge Merge in Reinforced Agentic Models	2026	Arxiv	RL-trained agentic models
ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Merging	2026	Arxiv	Simia-Tau-SFT-Qwen3-8B, SimiaOfficeBench-SFT-Qwen3-8B, and Simia-AgentBench-SFT-Qwen3-8B
Divide, Optimize, Merge: Scalable Fine-Grained Generative Optimization for LLM Agents	2025	EMNLP	o3-mini
AgentMerge: Enhancing Generalization in Fine-Tuned LLM Agents)	2024	NeurIPS	Llama3.1-8B
Agent Skill Acquisition for Large Language Models via CycleQD	2024	Arxiv	Llama3-8B-Instruct

Combine the Capabilities of Expert LLMs

Paper Title	Year	Conference/Journal	Remark
Functionality-Oriented LLM Merging on the Fisher–Rao Manifold	2026	Arxiv	Qwen2.5-14B, Qwen2.5-14B-Instruct-1M, Qwen2.5-Coder-14B-Instruct, DeepSeek-R1-Distill-Qwen-14B, OpenReasoning-Nemotron-14B
The Appeal and Reality of Recycling LoRAs with Adaptive Merging	2026	Arxiv	Llama3.1 8B-Instruct
LS-Merge: Merging Language Models in Latent Space	2026	ICLR	Gemma-3-1B-it, Gemma-3-4B-it, Llama-3-1B-instruct, Llama-2-7b
Bagging-Based Model Merging for Robust General Text Embeddings	2026	Arxiv	Qwen3-4B
Data-driven Clustering and Merging of Adapters for On-device Large Language Models	2026	Arxiv	Llama 3.2 3B, Qwen 2.5 1.5B and StableLM 2 1.6B
Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging	2026	Arxiv	Llama-3.1-8b-Instruct
SimMerge: Learning to Select Merge Operators from Similarity Signals	2026	Arxiv	7B to 111B
Multi-Stage Evolutionary Model Merging with Meta Data Driven Curriculum Learning for Sentiment-Specialized Large Language Modeling	2026	Arxiv
ReasonAny: Incorporating Reasoning Capability to Any Model via Simple and Effective Model Merging	2026	Arxiv	QwQ-32B-Preview, Meditron3-Qwen2.5-7B and MMed-Llama3-8B, WiroAIFinance-Qwen-7B and WiroAI-Finance-Llama8B
Reliable Cultural Knowledge Preservation in Multilingual LLMs through Model Merging	2025	Arxiv	Qwen-2.5-3B
AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints	2025	Arxiv	LLaMA-3 8B, Mistral 7B, Qwen 2, Phi-3.5, Gemma 2
Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation	2025	Arxiv
Adapting Chat Language Models Using Only Target Unlabeled Language Data	2025	TMLR	Qwen2.5 7B, Llama 3.1 8B, Qwen3 14B
RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Prior	2026	AAAI	Qwen2.5-7B, Llama3.1-8B
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance	2025	Arxiv	xLAM-2-70b, CoALM-70B, watt-tool-70B, functionary-medium-70B, xLAM-2-8b, ToolACE-2-8B, watt-tool-8B, BitAgent-8B, CoALM-8B
SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation	2025	Arxiv
Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance	2025	Arxiv	Llama-3-8B, Llama-2-7B
Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models	2025	EMNLP	LLaMA-3 8B
Bridging Dialectal Gaps in Arabic Medical LLMs through Model Merging	2025	arabicnlp
Adapting Multilingual Models to Code-Mixed Tasks via Model Merging	2025	Arxiv
Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation	2025	Arxiv	Llama-3.1-8B-Instruct and Gemma-3-12B-Instruct
ABC: Towards a Universal Code Styler through Model Merging	2025	ACM on Programming Languages	Qwen2.5-Coder, Deepseek-Coder
Family Matters: Language Transfer and Merging for Adapting Small LLMs to Faroese	2025	Arxiv
Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking	2025	Arxiv	Mistral-7B, InternVL, Qwen2-VL
The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging	2025	Arxiv	Qwen3-30B-A3B-Thinking-2507, Qwen3-30B-A3B-Instruct-2507
MLM: Multi-linguistic LoRA Merging 2025	NeurIPS WorkShop	LLaMA-3.2 (1B and 3B)
Model Merging Scaling Laws in Large Language Models	2025	Arxiv	Qwen2.5 0.5, 1.5, 3, 7, 14, 32, 72B
Harnessing Optimization Dynamics for Curvature-Informed Model Merging	2025	Arxiv	Llama-3.1-8B
Kwai Keye-VL 1.5 Technical Report	2025	Arxiv	Keye-VL-8B
Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic	2025	Arxiv	QWEN2.5-7B
Surrogate Benchmarks for Model Merging Optimization	2025	Arxiv	EvoLLM-JP-v1-7B, shisa-gamma-7b-v1
Tensorized Clustered LoRA Merging for Multi-Task Interference	2025	Arxiv	Mistral-7B
Efficient Compositional Multi-tasking for On-device Large Language Models	2025	Arxiv	Llama 3.1 70B
HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging	2025	Arxiv
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts	2025	Arxiv
Merging Large Language Models for Enhanced Code Generation: A Comparative Study of Model Merging Techniques Across Programming Languages	2025	Open Access in DiVA	CodeQwen1.5-7B, DeepSeek-Coder-6.7b-Base, CodeLlama-34B
On Fairness of Task Arithmetic: The Role of Task Vectors	2025	Arxiv	LLaMA2-7B
The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs	2025	Arxiv	FALCON 3 7B, QWEN2.5 7B Instruct, LLAMA 3.1 8B Instruct, AYA Expanse 8B
Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning	2025	Arxiv	MetaMath-Mistral-7B, Dolphin-2.1-Mistral-7B and Speechless-Code-Mistral-7Bv1.0
Training-free LLM Merging for Multi-task Learning	2025	ACL	Echelon-AI/Med-Qwen2-7B, shtdbb/qwen2-7b-med, Qwen2-Instruct
ParamΔ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost	2025	Arxiv	Llama3-inst-70B, Llama3-base-70B, Llama3.1-base-70B
Beyond ‘Aha!’: Toward Systematic Meta-Abilities Alignment in Large Reasoning Models	2025	Arxiv	Qwen2.5-7B, Qwen2.5-32B
Unified Multi-Task Learning & Model Fusion for Efficient Language Model Guardrailing	2025	Arxiv
Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging -- An Open Recipe	2025	Arxiv	Typhoon2 R1 70B, Deepseek R1 70B
Efficient Model Development through Fine-tuning Transfer	2025	Arxiv	Llama 3.1 8B
Command A: An Enterprise-Ready Large Language Model	2025	Arxiv	Command R7B
Extrapolation Merging: Keep Improving With Extrapolation and Merging	2025	Arxiv	Qwen2-7B, Meta-Llama-3-8B, Mistral-Nemo-Base-2407-12B, Qwen1.5-14B
Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond	2025	Arxiv	Light-R1-32B
FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion	2025	Arxiv	Gemma-2-27B-it, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct
Superficial Self-Improved Reasoners Benefit from Model Merging	2025	Arxiv	Llama2-7B
Nature-Inspired Population-Based Evolution of Large Language Models	2025	Arxiv
Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge	2025	Arxiv	Gemma-2-9B, Llama-3-8B
Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation	2025	Arxiv	WizardLM-13B, WizardMath-13B, llama-2-13b-code-alpaca
LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging	2025	Arxiv	NuminaMath-7B, DeepSeek-Math-7B-Base, LLaMA-series models, WizardMath-13B
Merging Language and Domain Specific Models: The Impact on Technical Vocabulary Acquisition	2025	Arxiv	ContactDoctor-8B
Transferring Textual Preferences to Vision-Language Understanding through Model Merging	2025	Arxiv	Llama-3.2-11B-Vision -Instruct, Llama-3.1-Tulu-2-8B-uf-mean-rm, Llama-3.1-Tulu-3-8B-RM
Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging	2025	Arxiv	Llama-2-13b, WizardMath-13B-V1.0, WizardLM13B-V1.2, llama-2-13b-codealpaca
An Open Recipe: Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging	2025	Arxiv	Typhoon2 70B Instruct, DeepSeek R1 70B Distill, Llama 3.1 70B, Llama 3.3 70B
Fine, I’ll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging	2025	Arxiv	WizardLM-13B, WizardMath-13B, and llama-2-13b-code-alpaca
Skill Expansion and Composition in Parameter Space	2025	Arxiv
InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion	2025	Arxiv	Qwen2.5-Coder-14B-Instruct, Qwen2.5-14B-Instruct, and Mistral-Small-24B-Instruct-2501
Channel Merging: Preserving Specialization for Merged Experts	2025	AAAI	Dolphin-2.2.1-Mistral-7B, Speechless-Code-Mistral-7B, MetaMathMistral-7B, Chinese-Mistral-7BInstruct-v0.1
Weighted-reward preference optimization for implicit model fusion	2025	ICLR	LLaMA3-8B-Instruct
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion	2024	Arxiv	MiniGemini-8B and SLIME-8B
AgentMerge: Enhancing Generalization in Fine-Tuned LLM Agents	2024	Arxiv	Llama3.1-8B
JRadiEvo: A Japanese Radiology Report Generation Model Enhanced by Evolutionary Optimization of Model Merging	2024	Arxiv	Bunny-v1_1-Llama-3-8B-V, MMed-Llama-3-8B-EnIns, OpenBioLLM-Llama3-8B, Llama-3-Swallow-8B-Instruct-v0.1
If You Can’t Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs	2024	Arxiv	Command R+ 104B
Agent Skill Acquisition for Large Language Models via CycleQD	2024	Arxiv	Llama3-8B-Instruct
Collaboratively adding new knowledge to an LLM	2024	Arxiv	Meta-Llama-3-8B
Unconstrained Model Merging for Enhanced LLM Reasoning	2024	Arxiv	CodeLlama-7B-Ins, CodeLlama-70B-Ins, Deepseek-Coder-Ins-v1.5, Qwen2.5-Math-7B-Ins, WizardMath-7B-V1.1, OpenMath-Mistral 7B, MetaMath-7B, MetaMath-70B
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks	2024	Arxiv	Llama-7b, Llama2-7b-chat
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging	2024	Arxiv	Llama 2 7B
Exploring Model Kinship for Merging Large Language Models	2024	Arxiv	Mistral-7B, Mistral-7b-instruct-v0.2, MetaMath-mistral-7b, Open-chat-3.5-1210
Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation	2024	Arxiv	shisa-gamma-7b, WizardMath-7B-V1.1, Abel-7B-002, Llama-3-SauerkrautLM-8b-Instruct, Llama-3-Open-Ko-8B, llama-3-sqlcoder-8b, Meta-Llama-3-8B
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models	2024	Arxiv	LLAMA 3.1 8B
What Matters for Model Merging at Scale?	2024	Arxiv	PaLM-2 (1B, 8B, 24B, 64B), PaLM-2-IT (1B, 8B, 24B, 64B)
HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models	2024	Arxiv	Llama-2-7B-Chat, WizardMath-7B, CodeLlama-7B
FUSECHAT: Knowledge Fusion of Chat Models	2024	Arxiv	OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B
SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging	2024	Arxiv	CodeLlama 7B
It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization	2024	Arxiv	Qwen1.5-7B-Chat, Liberated-Qwen1.5-7B, firefly-qwen1.5-en-7B
Knowledge Fusion By Evolving Weights of Language Models	2024	ACL
LLM Merging: Building LLMs Efficiently through Merging	2024	NeurIPS 2024 Competition Track	LLaMA-7B, Mistral-7B, Gemma-7B
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement	2024	Arxiv	Qwen1.5-7B, Qwen1.5-Chat-7B, Sailor-7B, Qwen1.5-14B, Qwen1.5-Chat-14B, Sailor-14B, WizardLM-13B, WizardMath-13B, llama-2-13b-code-alpaca
It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization	2024	Arxiv	Qwen1.5-7B-Chat, Liberated-Qwen1.5-7B, firefly-qwen1.5-en-7B
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic	2024	Arxiv	LLaMA-2-7B, Mistral-7B, LLaMA-2-13B
PROMETHEUS 2: An Open Source Language Model Specialized in Evaluating Other Language Models	2024	Arxiv	Mistral-Instruct-7B, Mixtral-Instruct-8x7B
Knowledge fusion of large language models	2024	ICLR	Llama-2 7B, OpenLLaMA 7B, MPT 7B
Language models are super mario: Absorbing abilities from homologous models as a free lunch	2024	ICML	WizardLM-13B, WizardMath-13B, and llama-2-13b-codealpaca, Mistral-7B
Controlled Text Generation via Language Model Arithmetic	2024	ICML	MPT-7B, Pythia-12B, Llama-2-Chat-13B
MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models	2024	Arxiv	LlaMA2-13B and LlaMA3-8B (LoRA)
Evolutionary optimization of model merging recipes	2024	Arxiv	shisa-gamma-7b-v1, WizardMath-7B-V1.1, Arithmo2-Mistral-7B, Abel-7B-002, Mistral-7B-v0.1, LLaVA-1.6-Mistral-7B
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM	2024	Arxiv	Llama-2-7B
Knowledge Fusion of Chat LLMs: A Preliminary Technical Report	2024	Arxiv	NH2-Mixtral-8x7B, NH2-Solar-10.7B, OpenChat-3.5-7B

Note: The following papers are from: LLM Merging Competition at NeurIPS 2024

Paper Title	Year	Conference/Journal	Models
Llm merging: Building llms efficiently through merging	2024	LLM Merging Competition at NeurIPS	-
Towards an approach combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models	2024	LLM Merging Competition at NeurIPS	meta-llama/Llama-2-7b; microsoft_phi1/2/3
Model Merging using Geometric Median of Task Vectors	2024	LLM Merging Competition at NeurIPS	flan_t5_xl
Interpolated Layer-Wise Merging for NeurIPS 2024 LLM Merging Competition	2024	LLM Merging Competition at NeurIPS	suzume-llama-3-8B-multilingual-orpo-borda-top75, Barcenas-Llama3-8bORPO, Llama-3-8B-Ultra-Instruct-SaltSprinkle, MAmmoTH2-8B-Plus, Daredevil-8B
A Model Merging Method	2024	LLM Merging Competition at NeurIPS	-
Differentiable DARE-TIES for NeurIPS 2024 LLM Merging Competition	2024	LLM Merging Competition at NeurIPS	suzume-llama-3-8B-multilingualorpo-borda-top75, MAmmoTH2-8B-Plus and Llama-3-Refueled
LLM Merging Competition Technical Report: Efficient Model Merging with Strategic Model Selection, Merging, and Hyperparameter Optimization	2024	LLM Merging Competition at NeurIPS	MaziyarPanahi/Llama3-8B-Instruct-v0.8, MaziyarPanahi/Llama-3-8B-Instruct-v0.9, shenzhiwang/Llama3-8B-Chinese-Chat, lightblue/suzume-llama-3-8B-multilingual
Simple Llama Merge: What Kind of LLM Do We Need?	2024	LLM Merging Competition at NeurIPS	Hermes-2-Pro-Llama-3-8B, and Daredevil-8B
LLM Merging Competition Technical Report for NeurIPS 2024: Efficiently Building Large Language Models through Merging	2024	LLM Merging Competition at NeurIPS	Mistral-7B-Instruct94 v2, Llama3-8B-Instruct, Flan-T5-large, Gemma-7B-Instruct, and WizardLM-2-7B
MoD: A Distribution-Based Approach for Merging Large Language Models	2024	LLM Merging Competition at NeurIPS	Qwen2.5-1.5B and Qwen2.5-7B

Model Merging in Multimodal Large Language Models

Model Merging for Multimodal Fusion

Paper Title	Year	Conference/Journal	Remark
Jointly training large autoregressive multimodal models	2024	ICLR
Model Composition for Multimodal Large Language Models	2024	ACL	Vicuna-7B-v1.5
π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation	2023	ICML
An Empirical Study of Multimodal Model Merging	2023	EMNLP
UnIVAL: Unified Model for Image, Video, Audio and Language Tasks	2023	TMLR

Model Merging for Cross-Modal Knowledge Transfer

Paper Title	Year	Conference/Journal	Remark
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification	2024	ICASSP Workshop

Combine the Capabilities of Expert MLLMs

Paper Title	Year	Conference/Journal	Remark
VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models	2026	ICLR	VisCodex-8B, VisCodex-33B
FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision–Language Models	2026	Arxiv	Qwen2.5-VL-7B-Instruct, DeepSeekR1-Distill-Qwen-7B, Qwen2.5-VL-32B-Instruct, QwQ-32B
PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs	2026	Arxiv	LLaVA-v1.5-7B, Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct
Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning	2026	AAAI	Qwen-VL-7B, Idefics2-8B
MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent	2025	Arxiv	Qwen2.5-0.5B
Tiny-R1V: Lightweight Multimodal Unified Reasoning Model via Model Merging	2025	Arxiv
Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models	2025	Arxiv
Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking	2025	Arxiv	Mistral-7B, InternVL, Qwen2-VL
UQ-Merge: Uncertainty Guided Multimodal Large Language Model Merging	2025	ACL	LLaVA-v1.5-7B
Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs	2025	Arxiv	Qwen2-VL-2B
Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging	2025	Arxiv	Qwen2-VL-7B-Base, Vicuna-7B-v1.5
Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging	2025	ICML	LLaVA-NeXT-8B, Idefics2-8B, InternVL2-76B
REMEDY: Recipe Merging Dynamics in Large Vision-Language Models	2025	ICLR	LLaVA-1.5 (Vicuna-7B)
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness	2025	NeurIPS	LLaVA-v1.5-7B
Parameter Efficient Merging for Multimodal Large Language Models with Complementary Parameter Adaptation	2025	Arxiv	LLaVA
AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization	2025	Arxiv	LLaVA-OneVision-7B, Qwen2-VL-7B, LLaVA-v1.5-7B, CogVLM-chat-7B
Transferring Textual Preferences to Vision-Language Understanding through Model Merging	2025	Arxiv	Llama-3.2-11B-Vision-Instruct, Llama-3.1-Tulu-2-8B-uf-meanrm, Llama-3.1-Tulu-3- 8B-RM, Llama-3.1-8B

Model Merging in Image Generative Models

Style Mixing in Generative Models

Paper Title	Year	Conference/Journal	Remark
GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization	2026	Arxiv
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout	2025	Arxiv
BlockLoRA: Modular Customization of Diffusion Models via Blockwise-Parameterized Low-Rank Adaptation	2025	Arxiv
LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation	2024	Arxiv	LLaVA-Critic 7b
IterIS: Iterative Inference-Solving Alignment for LoRA Merging	2024	Arxiv
Diffusion Soup: Model Merging for Text-to-Image Diffusion Models	2024	ECCV
MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models	2024	Arxiv
MoLE: Mixture of LoRA Experts	2024	ICLR
LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models	2024	Arxiv
Multi-LoRA Composition for Image Generation	2024	Arxiv
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models	2023	NeurIPS
Merging loras	2023	(github)
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs	2023	Arxiv
GAN Cocktail: mixing GANs without dataset access	2022	ECCV

Reducing Training Cost of Generative Models

Paper Title	Year	Conference/Journal	Remark
Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better	2024	Arxiv
A Unified Module for Accelerating STABLE-DIFFUSION: LCM-LORA	2024	Arxiv

Enhancing the Faithfulness (or Generation Quality) of Diffusion Models

Paper Title	Year	Conference/Journal	Remark
Decouple-Then-Merge: Towards Better Training for Diffusion Models	2024	Arxiv
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data	2024	Arxiv

Deepfake Detection

Paper Title	Year	Conference/Journal	Remark
Real-Aware Residual Model Merging for Deepfake Detection	2025	Arxiv

Model Merging in Video Generative Models

Enhancing Motion Modeling

Paper Title	Year	Conference/Journal	Remark
Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think	2025	CVPR	Dynamicrafter，SVD

---

Application of Model Merging in Different Machine Learning Subfields

Model Merging in Continual Learning

Model Merging to Mitigate Catastrophic Forgetting

Paper Title	Year	Conference/Journal	Remark
Mapping Post-Training Forgetting in Language Models at Scale	2026	ICLR
LCA: Local Classifier Alignment for Continual Learning	2026	ICLR
MERGETUNE: Continued fine-tuning of vision-language models	2026	Arxiv
Merge before Forget: A Single LoRA Continual Learning via Continual Merging	2025	Arxiv	Llama-2-7B-chat, Llama-2-13B-chat, Qwen2.5-7B
Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging	2025	Arxiv
Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport	2025	Arxiv
MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images	2025	Arxiv
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging	2025	Arxiv	Qwen2-7B-Instruct, Llama-2-7B-chat
DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection	2025	NeurIPS
K-Merge: Online Continual Merging of Adapters for On-device Large Language Models	2025	Arxiv
Toward a Holistic Approach to Continual Model Merging	2025	Arxiv
Null-Space Filtering for Data-Free Continual Model Merging: Preserving Transparency, Promoting Fidelity	2025	Arxiv
AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning	2025	EMNLP	LLaMA2-7B, LLaMA2-13B
HAM: Hierarchical Adapter Merging for Scalable Continual Learning	2025	Arxiv
Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning	2025	ICML	LLaVA-1.5-7B
DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmetic	2025	Arxiv
Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning	2025	ICCV
Forgetting of task-specific knowledge in model merging-based continual learning	2025	Arxiv
Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition	2025	Arxiv
RegCL: Continual Adaptation of Segment Anything Model via Model Merging	2025	Arxiv
Continual Learning in Vision-Language Models via Aligned Model Merging	2025	Arxiv
Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning	2025	Arxiv
MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging	2025	NeurIPS
Analysis of Model Merging Methods for Continual Updating of Foundation Models in Distributed Data Settings	2025	Arxiv	Applied Sciences
BECAME: BayEsian Continual Learning with Adaptive Model MErging	2025	Arxiv
Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs	2025	Arxiv	Llama-3-8B-Instruct
Cost-Efficient Continual Learning with Sufficient Exemplar Memory	2025	Arxiv
Continual Model Merging without Data: Dual Projections for Balancing Stability and Plasticity	2025	NeurIPS
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging	2025	NeurIPS
Soup to go: mitigating forgetting during continual learning with model averaging	2025	Arxiv	Llama 2 (7B)
Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning	2024	Arxiv
Parameter Averaging is All You Need to Prevent Forgetting	2024	SLT Workshop
DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning	2024	Arxiv
Adaptive LoRA Merging for Efficient Domain Incremental Learning	2024	NeurIPS Workshop
LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging	2024	Arxiv
Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models	2024	ICML	InstructBLIP (Vicuna-7B), LLaVA-1.5 (Vicuna7B)
Adaptive Discovering and Merging for Incremental Novel Class Discovery	2024	AAAI
MagMax: Leveraging Model Merging for Seamless Continual Learning	2024	ECCV
Lm-cocktail: Resilient tuning of language models via model merging	2024	ACL Findings	Llama-2-chat-7b
Backward Compatibility During Data Updates by Weight Interpolation	2024	EACL
Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models	2024	EMNLP Findings
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging	2024	Arxiv	MISTRAL-7B, LLAMA-3-8B
Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation	2024	Arxiv	Llama3-70B
Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs	2024	Arxiv	Mistral-7B, Llama-3-8B
WARP: On the Benefits of Weight Averaged Rewarded Policies	2024	Arxiv	Gemma-7B
A Second-Order perspective on Compositionality and Incremental Learning	2024	Arxiv
DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Images	2024	Arxiv
DAM: Dynamic Adapter Merging for Continual Video QA Learning	2024	Arxiv
Task-Specific Skill Localization in Fine-tuned Language Model	2023	ICML
Tangent model composition for ensembling and continual fine-tuning	2023	ICCV
A Unified Continual Learning Framework with General Parameter-Efficient Tuning	2023	ICCV
Task Arithmetic with LoRA for Continual Learning	2023	NeurIPS Workshop
Mitigating the Alignment Tax of RLHF	2023	Arxiv	Mistral-7B
PAINT: Patching open-vocabulary models by interpolating weights	2022	NeurIPS
Robust fine-tuning of zero-shot models	2022	CVPR

Model Merging in Multi-Task/Multi-Objective/Multi-Domain/Auxiliary Learning

Model Merging for Knowledge Transfer in Multi-Task Learning

Paper Title	Year	Conference/Journal	Remark
G-Merging: Graph Models Merging for Parameter-Efficient Multi-Task Knowledge Consolidation	2026	ICLR
Multi-task Code LLMs: Data Mix or Model Merge?	2026	Arxiv	Qwen Coder 2.5 7B, DeepSeek 7B
DivMerge: A divergence-based model merging method for multi-tasking	2025	Arxiv
Single-Input Multi-Output Model Merging: Leveraging Foundation Models for Dense Multi-Task Learning	2025	Arxiv
Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Merging	2024	Arxiv
LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging	2024	Arxiv
Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning	2024	Arxiv	Aya 23 8B
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks	2024	Arxiv
Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer	2024	Arxiv
Evolutionary optimization of model merging recipes	2024	Arxiv	shisa-gamma-7b-v1, WizardMath-7B-V1.1, Arithmo2-Mistral-7B, Abel-7B-002, Mistral-7B-v0.1, LLaVA-1.6-Mistral-7B
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch	2024	ICML	WizardLM-13B, WizardMath-13B, and llama-2-13b-codealpaca, Mistral-7B
Representation Surgery for Multi-Task Model Merging	2024	ICML
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts	2024	ICML
ZipIt! Merging Models from Different Tasks without Training	2024	ICLR
AdaMerging: Adaptive Model Merging for Multi-Task Learning	2024	ICLR
Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policies	2023	Arxiv
Resolving Interference When Merging Models	2023	NeurIPS
Editing models with task arithmetic	2023	ICLR

Model Merging for Knowledge Transfer in Multi-Objective Optimization

Paper Title	Year	Conference/Journal	Remark
From Parameter to Representation: A Closed-Form Approach for Controllable Model Merging	2026	AAAI
Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation	2025	Arxiv	LLaMA-2-7B
Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging	2025	ICML
Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation	2025	Arxiv	LLaMA-2 7B
You Only Merge Once: Learning the Pareto Set of Preference-Aware Model Merging	2024	Arxiv
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion	2024	Arxiv
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation	2024	Arxiv	Llama3-8B

Model Merging for Knowledge Transfer in Multi-Domain Learning

Paper Title	Year	Conference/Journal	Remark
Domain-Adaptive Model Merging across Disconnected Modes	2026	Arxiv
Bridging Domains through Subspace-Aware Model Merging	2026	Arxiv
Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR	2026	Arxiv
To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models	2026	Arxiv	Qwen3-4B-Base
MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Merging	2026	Arxiv	Qwen3-0.6B
MergeRec: Model Merging for Data-Isolated Cross-Domain Sequential Recommendation	2026	KDD
DEM: Distribution Edited Model for Training with Mixed Data Distributions	2024	Arxiv	OpenLLaMA-7B, OpenLLaMA-13B
Merging Vision Transformers from Different Tasks and Domains	2023	Arxiv

Model Merging for Knowledge Transfer in Auxiliary Learning

Paper Title	Year	Conference/Journal	Remark
ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning	2023	NeurIPS

Model Merging in Out-of-Distribution/Domain Generalization

Model Merging for Better Out-of-Distribution Generalization

Paper Title	Year	Conference/Journal	Remark
Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR	2026	Arxiv
Model soups need only one ingredient	2026	Arxiv
System Report for CCL25-Eval Task 10: Prompt-Driven Large Language Model Merge for Fine-Grained Chinese Hate Speech Detection	2025	Arxiv	Qwen2.5-7B-Instruct
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data	2025	Arxiv
Out-of-Distribution Graph Models Merging	2025	Arxiv
SeWA: Selective Weight Average via Probabilistic Masking	2025	Arxiv
When, Where and Why to Average Weights?	2025	Arxiv
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation	2024	NeurIPS 2024 Workshop
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging	2024	Arxiv	Llama-2-7b
ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models	2024	Arxiv
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging	2024	ICLR
Warm: On the benefits of weight averaged reward models	2024	ICML
Scalable Learned Model Soup on a Single GPU: An Efficient Subspace Training Strategy	2024	ECCV
Adaptive Stochastic Weight Averaging	2024	JMLR
Population parameter averaging (papa)	2024	TMLR
WARP: On the Benefits of Weight Averaged Rewarded Policies	2024	Arxiv	Mistral 7B, Mixtral 8x7B
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average	2024	Arxiv
Model Stock: All we need is just a few fine-tuned models	2024	Arxiv
Lookaround Optimizer: 𝑘 steps around, 1 step average	2023	NeurIPS
Model ratatouille: Recycling diverse models for out-of-distribution generalization	2023	ICML
Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions	2023	ICLR
Lookaround Optimizer: k steps around, 1 step average	2023	NeurIPS
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models	2023	EACL
Dart: Diversify aggregate-repeat training improves generalization of neural networks	2023	CVPR
When do flat minima optimizers work?	2022	NeurIPS
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time	2022	ICML
Diverse weight averaging for out-of-distribution generalization	2022	NeurIPS
Robust fine-tuning of zero-shot models	2022	CVPR
Neural networks with late-phase weights	2021	ICLR
Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well	2020	ICLR
SWALP: Stochastic weight averaging in low precision training	2019	ICML
Averaging weights leads to wider optima and better generalization	2018	UAI
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results	2017	NeurIPS

Model Merging for Better Domain Generalization or Domain Adaptation

Paper Title	Year	Conference/Journal	Remark
Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models	2025	Arxiv	Qwen2.5-7B, Llama3.1-8B
Harmonizing and Merging Source Models for CLIP-based Domain Generalization	2025	Arxiv
Realistic Evaluation of Model Merging for Compositional Generalization	2024	Arxiv
Layer-wise Model Merging for Unsupervised Domain Adaptation in Segmentation Tasks	2024	Arxiv
Training-Free Model Merging for Multi-target Domain Adaptation	2024	Arxiv
Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation	2024	Arxiv	Llama3-70B
Ensemble of averages: Improving model selection and boosting performance in domain generalization	2022	NeurIPS
Swad: Domain generalization by seeking flat minima	2021	NeurIPS

Model Merging in Federated Learning

Model Merging for Local Knowledge Aggregation

Paper Title	Year	Conference/Journal	Remark
Communication-Efficient Personalized Adaptation via Federated-Local Model Merging	2026	Arxiv	LLaMA-3.2-3B-Instruct
On The Surprising Effectiveness of a Single Global Merging in Decentralized Learning	2026	ICLR
Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approach	2025	Arxiv	LLaMA-7B
Intrinsic Training Signals for Federated Learning Aggregation	2025	ICIAP
Breaking the Aggregation Bottleneck in Federated Recommendation: A Personalized Model Merging Approach	2025	Arxiv
A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning	2025	Arxiv
Closed-form merging of parameter-efficient modules for Federated Continual Learning	2025	ICLR
Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection	2025	Arxiv
FedMerge: Federated Personalization via Model Merging	2025	Arxiv
Personalized Language Models via Privacy-Preserving Evolutionary Model Merging	2025	Arxiv	Llama-2-7b, Mistral-7B-Instruct v0.2
FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors	2025	Arxiv
Many-Task Federated Fine-Tuning via Unified Task Vectors	2025	Arxiv
PrivFusion: Privacy-Preserving Model Fusion via Decentralized Federated Graph Matching	2024	TKDE
Model Trip: Enhancing Privacy and Fairness in Model Fusion Across Multi-Federations for Trustworthy Global Healthcare	2024	ICDE
DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices	2024	NeurIPS
FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion	2024	Arxiv
Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning	2024	Arxiv
DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models	2024	CVPR
FedFisher: Leveraging Fisher Information for One-Shot Federated Learning	2024	AISTATS
lo-fi: distributed fine-tuning without communication	2023	TMLR
Revisiting Weighted Aggregation in Federated Learning with Neural Networks	2023	ICML
Deep neural network fusion via graph matching with applications to model ensemble and federated learning	2022	ICML
Federated Learning with Matched Averaging	2020	ICLR
Tackling the objective inconsistency problem in heterogeneous federated optimization	2020	NeurIPS
Model fusion via optimal transport	2020	NeurIPS
Bayesian nonparametric federated learning of neural networks	2019	ICML
Learning private neural language modeling with attentive aggregation	2019	IJCNN
Communication-Efficient Learning of Deep Networks from Decentralized Data	2017	AISTATS

Model Merging in Zero-shot/Few-shot Learning

Model Merging for Cross-task Generalization in Zero-shot Learning

Paper Title	Year	Conference/Journal	Remark
Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis	2026	Arxiv
Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Models	2025	NeurIPS Workshop	LLAMA-3.1-8B-INSTRUCT
Investigating Task Arithmetic for Zero-Shot Information Retrieval	2025	SIGIR	LLama-2-7b
Retraining-Free Merging of Sparse Mixture-of-Experts via Hierarchical Clustering	2024	Arxiv	Qwen 60x2.7B, Qwen 45x2.7B, Qwen 30x2.7B, Mixtral 8x7B, Mixtral 6x7B, Mixtral 4x7B
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models	2024	Arxiv	LLAMA 3.1 8B
Learning to Route Among Specialized Experts for Zero-Shot Generalization	2024	ICML
Towards Modular LLMs by Building and Reusing a Library of LoRAs	2024	ICML	Mistral-7B
Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities	2024	ACL	LLaMA-2 13B, Chinese-LLaMA-13B, Chinese-Alpaca-13B, Mistral-7B, llama-2-ko-7b
Unlocking the Potential of Model Merging for Low-Resource Languages	2024	Arxiv	Llama-2-7B
Diffusion Soup: Model Merging for Text-to-Image Diffusion Models	2024	Arxiv
No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement	2024	Arxiv
MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models	2024	Arxiv
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging	2024	Arxiv	Llama2-7b
Model Composition for Multimodal Large Language Models	2024	Arxiv	Vicuna-7B-v1.5
Exploring the Benefits of Training Expert Language Models over Instruction Tuning	2023	ICML
Token-Level Adaptation of LoRA Adapters for Downstream Task Generalization	2023	Arxiv	Llama-2-7b
Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization	2023	Arxiv	PaLM 2-S

Model Merging for Cross-task Generalization in Few-shot Learning

Paper Title	Year	Conference/Journal	Remark
Task Arithmetic with Support Languages for Low-Resource ASR	2026	Arxiv
Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs	2025	CVPR
LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks	2024	ACL	Llama-2- 7B
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition	2024	COLM	Llama-2-7B, Llama-2-13B
LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild	2024	ACL
Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy?	2024	Arxiv
MerA: Merging pretrained adapters for few-shot learning	2023	Arxiv
Multi-Head Adapter Routing for Cross-Task Generalization	2023	NeurIPS

Model Merging in Adversarial Learning

Model Merging as an Attack

Paper Title	Year	Conference/Journal	Remark
Backdoor Vectors: a Task Arithmetic View on Backdoor Attacks and Defenses	2025	Arxiv
Merge Now, Regret Later: The Hidden Cost of Model Merging is Adversarial Transferability	2025	Arxiv
Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy	2025	ACL	Llama-3.2-3b-it, Gemma-2-2b-it, Qwen-2.5-3b-it, and Phi-3.5-mini-it
Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models	2025	Arxiv	LLaMA3.1-8B
From Purity to Peril: Backdooring Merged Models From “Harmless” Benign Components	2025	Arxiv	LLaMA2-7B-chat, Mistral-7B-v0.1
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging	2025	Arxiv
Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy	2025	Arxiv
LoBAM: LoRA-Based Backdoor Attack on Model Merging	2024	Arxiv
BadMerging: Backdoor Attacks Against Model Merging	2024	CCS
LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario	2024	ACL	Llama-2-7B

Model Merging as a Defense or Intellectual Property Protection

Paper Title	Year	Conference/Journal	Remark
Defending against Backdoor Attacks via Module Switching	2026	ICLR
Making Models Unmergeable via Scaling-Sensitive Loss Landscape	2026	Arxiv
Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models	2026	Arxiv	Llama2-7B and Qwen3-8B
Do Not Merge My Model! Safeguarding Open-Source LLMs Against Unauthorized Model Merging	2026	AAAI	LLaMA-2-13B, WizardLM-13B, WizardMath-13B, LLaMA-2-13B-Code Alpaca
Defending Unauthorized Model Merging via Dual-Stage Weight Protection	2025	Arxiv
Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing	2025	Arxiv
POSTER: Investigating Transferability of Adversarial Examples in Model Merging	2025	ASIA CCS
RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Merging	2025	Arxiv
MergeGuard: Efficient Thwarting of Trojan Attacks in Machine Learning Models	2025	Arxiv
BadJudge: Backdoor Vulnerabilities of LLM-As-A-Judge	2025	Arxiv	Mistral-7B-Instruct-v0.2, Meta-Llama3-8B
Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracy	2025	ICCV
Large Language Models Merging for Enhancing the Link Stealing Attack on Graph Neural Networks	2024	Arxiv	Vicuna-7B, Vicuna-13B
Strong Copyright Protection for Language Models via Adaptive Model Fusion	2024	ICML	LLaMa2 7B, StarCoder 7B
Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models	2024	Arxiv
REEF: Representation Encoding Fingerprints for Large Language Models	2024	Arxiv	Evollm-jp-7b, Shisa-gamma-7b-v1, Wizardmath-7b-1.1, Abel-7b-002, Llama-2-7b, Openllama-2-7b, Mpt-7b, Internlm2-chat-20b, Mixtral-8x7b-instruct, Qwen-1.5-chat-72b
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace	2024	Arxiv
MergePrint: Robust Fingerprinting against Merging Large Language Models	2024	Arxiv	LLaMA-2-7B, WizardMath-7B-V1.0, LLaMA-2-7B-CHAT
Avoiding Copyright Infringement via Machine Unlearning	2024	Arxiv	Llama3-8B
Merging Improves Self-Critique Against Jailbreak Attacks	2024	Arxiv	Mistral-7B, Mixtral-8x7B
Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging	2024	Arxiv	LLaMA-2-7B, LLaMA-2-7B-CHAT, WizardMath-7B-V1.0
Here’s a Free Lunch: Sanitizing Backdoored Models with Model Merge	2024	ACL
Revisiting adapters with adversarial training	2023	ICLR
Seasoning model soups for robustness to adversarial and natural distribution shifts	2023	CVPR

Other Applications

Paper Title	Year	Conference/Journal	Remark
Securing the Floor and Raising the Ceiling: A Merging-based Paradigm for Multi-modal Search Agents	2026	Arxiv
ACE-Brain-0: Spatial Intelligence as a Shared Scaffold for Universal Embodiments	2026	Arxiv
Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis	2026	Arxiv
Less Finetuning, Better Retrieval: Rethinking LLM Adaptation for Biomedical Retrievers via Synthetic Data and Model Merging	2026	Arxiv	Qwen3-0.6B, Gemma-2B, Phi4-3.8B
When Domain Pretraining Interferes with Instruction Alignment: An Empirical Study of Adapter Merging in Medical LLMs	2026	Arxiv	14B-parameter LLM
MergeRec: Model Merging for Data-Isolated Cross-Domain Sequential Recommendation	2026	KDD
Rare Word Recognition and Translation Without Fine-Tuning via Task Vector in Speech Models	2025	Arxiv
System Report for CCL25-Eval Task 10: Prompt-Driven Large Language Model Merge for Fine-Grained Chinese Hate Speech Detection	2025	Arxiv	Qwen2.5-7B-Instruct
Group-Aware Partial Model Merging for Children’s Automatic Speech Recognition	2025	Arxiv
Subtract the Corruption: Training-Data-Free Corrective Machine Unlearning using Task Arithmetic	2025	Arxiv
RecCocktail: A Generalizable and Efficient Framework for LLM-Based Recommendation	2025	AAAI	Llama-3.1-8B
A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs	2025	Arxiv	Mistral-7B
WeaveRec: An LLM-Based Cross-Domain Sequential Recommendation Framework with Model Merging	2025	Arxiv	Qwen2-7B
Effect of Model Merging in Domain-Specific Ad-hoc Retrieval	2025	Arxiv
Look the Other Way: Designing ‘Positive’ Molecules with Negative Data via Task Arithmetic	2025	Arxiv
Transferring Visual Explainability of Self-Explaining Models through Task Arithmetic	2025	Arxiv
Distilling a speech and music encoder with task arithmetic	2025	Arxiv
MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation	2025	Arxiv
Oscillation-Reduced MXFP4 Training for Vision Transformers	2025	ICML
Transferring Visual Explainability of Self-Explaining Models through Task Arithmetic	2025	Arxiv
Temporal Information Retrieval via Time-Specifier Model Merging	2025	Arxiv
Generative Representational Learning of Foundation Models for Recommendation	2025	Arxiv
Towards Model Merging for Tabular Telecommunications Data	2025	Arxiv
CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning	2025	Arxiv
U-Net Transplant: The Role of Pre-training for Model Merging in 3D Medical Segmentation	2025	International Conference on Medical Image Computing and Computer Assisted Intervention
CodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving	2025	Arxiv
Mixture of Latent Experts Using Tensor Products	2024	TMLR
In-Model Merging for Enhancing the Robustness of Medical Imaging Classification Models	2025	Arxiv
Self-supervised Normality Learning and Divergence Vector-guided Model Merging for Zero-shot Congenital Heart Disease Detection in Fetal Ultrasound Videos	2025	Arxiv
A Sliding Layer Merging Method for Efficient Depth-Wise Pruning in LLMs	2025	Arxiv	LLaMA-2-7B
Self-supervised Normality Learning and Divergence Vector-guided Model Merging for Zero-shot Congenital Heart Disease Detection in Fetal Ultrasound Videos	2025	Arxiv
MedForge: Building Medical Foundation Models Like Open Source Software Development	2025	Arxiv
Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette	2024	Arxiv
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging	2024	EMNLP	Llama-2-7b
Is Multiple Object Tracking a Matter of Specialization?	2024	NeurIPS
Tracking Universal Features Through Fine-Tuning and Model Merging	2024	Arxiv
HM3: Heterogeneous Multi-Class Model Merging	2024	Arxiv
Emotion Arithmetic: Emotional Speech Synthesis via Weight Space Interpolation	2024	Interspeech
Erasure Coded Neural Network Inference via Fisher Averaging	2024	Arxiv
MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair	2024	Arxiv
Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks	2024	Arxiv	Llama2-7B, Llama2-13B-chat, Mistral-7B-instruct
Scaling Up Personalized Image Aesthetic Assessment via Task Vector Customization	2024	Arxiv
An Attribute Interpolation Method in Speech Synthesis by Model Merging	2024	Arxiv
Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition	2024	Arxiv
MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Tasks	2024	Arxiv
Experts Weights Averaging: A New General Training Scheme for Vision Transformers	2023	Arxiv
One Student Knows All Experts Know: From Sparse to Dense	2022	Arxiv
Meta-Learning PAC-Bayes Priors in Model Averaging	2019	AAAI

---

Star History

---

Contact

We welcome all researchers to contribute to this repository 'model merging in foundation models or machine learning'.

If you have a related paper that was not added to the library, please contact us.

Email: ennengyang@qq.com / ennengyang@gmail.com