Evolutionarily conserved signaling pathways are essential for early embryogenesis, and reducing or abolishing their activity leads to characteristic developmental defects. Classification of phenotypic defects can identify the underlying signaling mechanisms, but this requires expert knowledge and the classification schemes have not been standardized. Here we use a machine learning approach for automated phenotyping to train a deep convolutional neural network, EmbryoNet, to accurately identify zebrafish signaling mutants in an unbiased manner. Combined with a model of time-dependent developmental trajectories, this approach identifies and classifies with high precision phenotypic defects caused by loss of function of the seven major signaling pathways relevant for vertebrate development. Our classification algorithms have wide applications in developmental biology and robustly identify signaling defects in evolutionarily distant species. Furthermore, using automated phenotyping in high-throughput drug screens, we show that EmbryoNet can resolve the mechanism of action of pharmaceutical substances. As part of this work, we freely provide more than 2 million images that were used to train and test EmbryoNet.