Cancer remains a major threat to human health. Tumor heterogeneity often leads to differences in tumor growth rate, invasion capacity, drug sensitivity, and prognosis, which complicates treatment strategies. Currently, drug responses are often verified through time-consuming and costly biological experiments, hindering the development of anticancer drug and precision medicine. With advancements in deep learning, various models for drug response prediction have been proposed. However, few of them take into account the impact of molecular topological properties on drug feature extraction and drug response prediction. In this study, we present DeepExpDR, a deep expert framework designed for drug response prediction. We first pretrain a self-supervised clustering model to group drugs based on their molecular scaffold similarities and then assign each drug group to a specialized substructure-aware expert. Each expert incorporates a substructure sensing network, which predicts drug response information from substructure sequences, cancer cell transcriptional gene expression values, and drug response correlation matrices. Finally, the predicted responses from experts are weighted summed to generate the final IC50 value. Experimental results demonstrate that DeepExpDR achieves state-of-the-art performance in both warm and cold settings, across regression and classification tasks. Our case study further verifies the effectiveness of DeepExpDR for detecting unknown cancer drug responses. Data and codes are available on https://github.com/ZYPssss/DeepExpDR.