Gene regulatory network (GRN) provides critical insights into the molecular mechanisms that govern cellular processes and disease pathogenesis, facilitating the identification of key regulatory factors and the discovery of potential therapeutic targets. Although numerous methods have been proposed to infer GRN from single-cell RNA sequencing (scRNA-seq) data, GRN inference remains challenging due to the inherent sparsity of scRNA-seq data and the naturally sparse connectivity of GRN. To address these challenges, this study proposes the Multimodal Adaptive GRN Inference Constructor (MAGIC), a method that improves GRN inference by aligning and integrating gene expression data, sequence information, and semantic features. Specifically, gene expression features reflect gene activity within cells, gene sequence features offer structural insights at the DNA level, and gene semantic features encapsulate functional meaning by leveraging biological knowledge bases. Furthermore, a consensus similarity network is constructed from multimodal gene similarity networks and integrated with known GRN to form a dual-topology network. To address the issue of sparse connectivity in GRN, a shared graph attention weight alignment module is employed. Following this, a Knowledge-Aware Multimodal Fusion Module is introduced to effectively integrate multimodal features by leveraging prior knowledge, thereby alleviating the inherent sparsity of scRNA-seq data. Finally, the fused features are used to infer GRNs. MAGIC achieved an average AUROC of 0.839 across seven scRNA-seq data sets using four types of ground-truth networks, outperforming other state-of-the-art models. Further analysis of two spatial transcriptomic data sets, bladder and breast cancer, demonstrates the robustness of MAGIC and its ability to uncover potential associations between transcription factors (TFs) and their target genes. MAGIC is publicly available at https://github.com/ydkvictory/MAGIC.