The three-dimensional organization of chromatin is essential for gene regulation and cellular function, with epigenome playing a key role. Hi-C methods have expanded our understanding of chromatin interactions, but their high cost and complexity limit their use. Existing models for predicting chromatin interactions rely on limited ChIP-seq inputs, reducing their accuracy and generalizability. In this work, we present a computational approach, EpiVerse, which leverages imputed epigenetic signals and advanced deep learning techniques. EpiVerse significantly improves the accuracy of cross-cell-type Hi-C prediction, while also enhancing model interpretability by incorporating chromatin state prediction within a multitask learning framework. Moreover, EpiVerse predicts Hi-C contact maps across an array of 39 human tissues, which provides a comprehensive view of the complex relationship between chromatin structure and gene regulation. Furthermore, EpiVerse facilitates unprecedented in silico perturbation experiments at the "epigenome-level" to unveil the chromatin architecture under specific conditions. EpiVerse is available on GitHub: https://github.com/jhhung/EpiVerse .