The identification of emotions conveyed by faces as genuine or not is a topic under-explored. While some controversial insights are available for happiness (where genuine happiness is supposed to create crow's feet around the eyes), nothing is known regarding other emotions. This topic is important as human beings are known to perform around the chance level to identify genuine emotions. It is thus pivotal to identify the relevant features for the correct identification of a facial emotion as genuine. To this aim, we capitalized on explainable artificial intelligence (XAI), characterized by a superior ability to detect subtle patterns in the data. Two different XAI models were created and applied to a dataset including 50 participants displaying both genuine and not genuine emotions. Results revealed that ML models achieved high accuracies in genuineness discrimination (78 % of average) while being robust and interpretable. The robustness of the results, ensured by their stability across different models and experimental conditions, is critical to improving the generalizability of the results. Our XAI algorithms provided interpretable results as they correctly identified facial muscle movements that are critical for the classification of emotions as genuine or fake.