Recently, the increasing spread of Online Social Networks (OSNs) provided an unprecedented opportunity of analysing online traces of human behaviour to get insight on individuals and society. Among the others, the possibility of predicting users' political orientation relying on data extracted from OSNs received growing attention. In this study, we introduce and make publicly available a dataset composed of 6.685 unique Twitter users and 9.593.055 Tweets. Differently from most of the dataset currently available in the literature, here, each user was manually labeled according to their political orientation by a pool of human judges, using strict inclusion criteria. Further, we address the feasibility of the automatic classification of Italian Twitter users' political orientation based on their Tweets content. Our analysis focuses first on implementing a series of classifiers with the aim of predicting users' political preference as right- or left-oriented. The built models were then evaluated for inferring the political orientation of those users supporting 'Movimento 5 Stelle' (M5S), an Italian political party with a still unclear political leaning. Results show high performances on the left-right classification task, with accuracy rates up to 93%. Finally, classification performances obtained on M5S supporters and possible applications of our findings are discussed.