Publications | Jiaying He

Mitigating Hallucination of Large Vision-Language Models via Dynamic Logits Calibration

Thu, 01 May 2025 00:00:00 +0000

Note: This collaborative work was completed during my undergraduate studies at ZJUT, focusing on addressing hallucination issues in vision-language models.

Overview

Dynamic Logits Calibration (DLC) is a novel framework designed to mitigate hallucination in Large Vision-Language Models (LVLMs) by dynamically aligning text generation with visual evidence during inference. By leveraging CLIP to evaluate semantic associations between input images and generated text sequences, DLC adaptively adjusts output token probability distributions, ensuring a more accurate and contextually grounded generation process.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated significant advancements in multimodal understanding, yet they are frequently hampered by hallucination-the generation of text that contradicts visual input. Existing training-free decoding strategies exhibit critical limitations, including the use of static constraints that do not adapt to semantic drift during generation, inefficiency stemming from the need for multiple forward passes, and degradation of detail due to overly rigid intervention rules.

To overcome these challenges, this paper introduces Dynamic Logits Calibration (DLC), a novel training-free decoding framework designed to dynamically align text generation with visual evidence at inference time. At the decoding phase, DLC step-wise employs CLIP to assess the semantic alignment between the input image and the generated text sequence. Then, the Relative Visual Advantage (RVA) of candidate tokens is evaluated against a dynamically updated contextual baseline, adaptively adjusting output logits to favor tokens that are visually grounded.

Furthermore, an adaptive weighting mechanism, informed by a real-time context alignment score, carefully balances the visual guidance while ensuring the overall quality of the textual output. Extensive experiments conducted across diverse benchmarks and various LVLM architectures (such as LLaVA, InstructBLIP, and MiniGPT-4) demonstrate that DLC significantly reduces hallucinations, outperforming current methods while maintaining high inference efficiency by avoiding multiple forward passes.

Key Innovations

Dynamic Visual Alignment: Unlike prior methods that rely on static constraints, DLC continuously evaluates the semantic alignment between the image and the generated text at each decoding step. This dynamic approach allows for real-time adjustments that better reflect the evolving context of the generated sequence.
Relative Visual Advantage (RVA): DLC introduces the concept of RVA, which measures the visual relevance of candidate tokens relative to a dynamically updated contextual baseline. This ensures that token adjustments are context-sensitive, enhancing the model’s ability to generate visually grounded text.
Adaptive Weighting Mechanism: To balance visual grounding with overall text quality, DLC employs an adaptive weighting strategy based on a context alignment score. This mechanism ensures that the influence of visual evidence is modulated according to the confidence in the current textual context, preventing over-reliance on visual cues that could compromise fluency.

Citation

@article{chen2025mitigating,
 title={Mitigating Hallucination of Large Vision-Language Models via Dynamic Logits Calibration},
 author={Chen, Jiahe and He, Jiaying and Shao, Qian and Chen, Qiyuan and Ying, Jiahe and Xu, Hongxia and Chen, Jintai and Zheng, Jianwei and Wu, Jian},
 journal={arXiv preprint arXiv:2506.21509},
 year={2025}
}

BioGraphFusion: Graph Knowledge Embedding for Biological Completion and Reasoning

Sat, 01 Mar 2025 00:00:00 +0000

Note: This work was completed during my undergraduate studies and represents my early research experience in graph neural networks and biological knowledge graphs.

Overview

Motivation: Biomedical knowledge graphs (KGs) are crucial for drug discovery and disease understanding, yet their completion and reasoning are challenging. Knowledge Embedding (KE) methods capture global semantics but struggle with dynamic structural integration, while Graph Neural Networks (GNNs) excel locally but often lack semantic understanding.

Results: We introduce BioGraphFusion, a novel framework for deeply synergistic semantic and structural learning. BioGraphFusion establishes a global semantic foundation via tensor decomposition, guiding an LSTM-driven mechanism to dynamically refine relation embeddings during graph propagation. Experiments across three key biomedical tasks demonstrate BioGraphFusion’s superior performance over state-of-the-art KE, GNN, and ensemble models.

Key Contributions

CP decomposition-based global semantic modeling that establishes a semantic foundation to guide subsequent dynamic structural reasoning
LSTM-driven context-aware relation refinement mechanism that dynamically fuses entity contexts through gating systems
Query-guided subgraph construction with biological relevance filtering for precise and efficient reasoning

Citation

@article{lin2025biographfusion,
 title={BioGraphFusion: Graph Knowledge Embedding for Biological Completion and Reasoning},
 author={Lin, Yitong and He, Jiaying and Chen, Jiahe and Zhu, Xinnan and Zheng, Jianwei and Tao, Bo},
 journal={Bioinformatics},
 pages={btaf408},
 year={2025},
 publisher={Oxford University Press}
}

C3S3: Complementary Competition and Contrastive Selection for Semi-Supervised Medical Image Segmentation

Sun, 01 Dec 2024 00:00:00 +0000

Note: This work was completed during my undergraduate studies and represents my early research experience in medical image processing and semi-supervised learning.

Overview

Semi-supervised medical image segmentation (SSMIS) offers a promising solution for the challenge of insufficiently annotated samples in the medical field. Despite achieving impressive results in delineating primary target areas, most current methodologies struggle to precisely capture the subtle details of boundaries, often leading to significant diagnostic inaccuracies.

Abstract

For the immanent challenge of insufficiently annotated samples in the medical field, semi-supervised medical image segmentation (SSMIS) offers a promising solution. Despite achieving impressive results in delineating primary target areas, most current methodologies struggle to precisely capture the subtle details of boundaries. This deficiency often leads to significant diagnostic inaccuracies. To tackle this issue, we introduce C3S3, a novel semi-supervised segmentation model that synergistically integrates complementary competition and contrastive selection. This design significantly sharpens boundary delineation and enhances overall precision. Specifically, we develop an Outcome-Driven Contrastive Learning module dedicated to refining boundary localization. Additionally, we incorporate a Dynamic Complementary Competition module that leverages two high-performing sub-networks to generate pseudo-labels, thereby further improving segmentation quality. The proposed C3S3 undergoes rigorous validation on two publicly accessible datasets, encompassing the practices of both MRI and CT scans. The results demonstrate that our method achieves superior performance compared to previous cutting-edge competitors. Especially, on the 95HD and ASD metrics, our approach achieves a notable improvement of at least 6%, highlighting the significant advancements.

Key Innovations

Dynamic Complementary Competition (DCC): A dual-network dynamic game mechanism that generates high-confidence pseudo-labels, breaking dependency on single-model teacher frameworks
Outcome-Driven Contrastive Learning (ODCL): Combined with dual-space intersection-union contrastive loss functions to precisely enhance target boundary representation capabilities
Superior Performance: 6%+ improvement in boundary-sensitive metrics (95HD, ASD) on left atrial MRI and pancreatic CT datasets compared to SOTA methods

Citation

@article{he2025c3s3,
 title={C3S3: Complementary Competition and Contrastive Selection for Semi-Supervised Medical Image Segmentation},
 author={He, Jiaying and Lin, Yitong and Chen, Jiahe and Xu, Honghui and Zheng, Jianwei},
 journal={arXiv preprint arXiv:2506.07368},
 year={2025}
}