Towards generalizable fault diagnosis via LLM-driven hierarchical cross-modal alignment

Recent advances in generative artificial intelligence (AI) have led to transformative breakthroughs across multiple fields. As the core of generative AI, large language models (LLMs) possess strong generalization and reasoning capabilities, offering the potential to establish a unified framework for fault diagnosis. Such a framework can overcome the limitations of existing methods when faced with challenges, including multiple datasets, cross-condition transfer, and data imbalance. However, owing to the significant modality gap between continuous vibration signals and discrete text, effective LLM utilization to process time-series data for fault diagnosis remains a critical challenge. In this study, an LLM-driven hierarchical cross-modal alignment method, termed HCMA_GPT, is proposed to overcome this challenge. First, a signal-text synergistic representation module is constructed to generate domain-knowledge-enriched text descriptions from raw vibration signals. Then, these descriptions are jointly encoded with the original signals to enhance the diversity and semantic expressiveness of fault features. Secondly, a hierarchical cross-modal feature alignment and fusion module is designed to achieve feature alignment and fusion between signals and text data from shallow to deep layers. By minimizing the distributional discrepancy between the two modalities, this module enhances the applicability of semantically trained LLM to time-series data. Finally, a novel unified fault diagnosis framework is developed by leveraging the generalization and reasoning abilities of LLM. Extensive experiments conducted on multiple datasets, cross condition and data imbalance scenarios indicate that HCMA_GPT effectively improves diagnostic accuracy and exhibits superior robustness and generalization capabilities.