SMART: Semantic Merging Adaptive Regional Transformer for Image Captioning

Image captioning requires integrating accurate object recognition with a deep understanding of semantic relationships to generate natural descriptions. Existing methods primarily rely on grid-based visual features for image understanding, which suffer from semantic fragmentation and fail to form coherent region-level representations. Furthermore, many approaches do not effectively integrate semantic prior information from the textual modality,
resulting in coarse and inaccurate descriptions. To address these challenges, we propose the Semantic Merging Adaptive Regional Transformer (SMART) model, which incorporates three novel modules: a Spatial Continuous Region Aggregation Module (SCRAM) to mitigate fragmentation by aggregating grid
features into complete object regions; a Semantic-aware Region Enhancement Module (SREM) to align textual semantic priors with these region representations; and an Adaptive Region-to-Grid Feature Fusion Module (ARGFFM) that dynamically selects key regions to enhance feature capacity. Finally, these enhanced features are subsequently leveraged by a transformer-based encoder-decoder architecture to generate semantically robust and accurate captions. Extensive experimental results on the MS COCO dataset establish that the SMART model achieves state of-the-art performance, confirming the efficacy of the SMART.