Building Energy prediction has emerged as an active research area due to its potential in improving energy efficiency in building energy management systems. Essentially, building energy prediction belongs to the time series forecasting or regression problem, and data-driven methods have drawn more attention recently due to their powerful ability to model complex relationships without expert knowledge. Among those methods, artificial neural networks (ANNs) have proven to be one of the most suitable and potential approaches with the rapid development of deep learning. This survey focuses on the studies using ANNs for building energy prediction and provides a bibliometric analysis by selecting 324 related publications in the recent five years. This survey is the first review article to summarize the details and applications of twelve ANN architectures in building energy prediction. Moreover, we discuss three open issues and main challenges in building energy prediction using ANNs regarding choosing ANN architecture, improving prediction performance, and dealing with the lack of building energy data. This survey aims at giving researchers a comprehensive introduction to ANNs for building energy prediction and investigating the future research directions when they attempt to implement ANNs to predict building energy demand or consumption.