In the research reported on here, we focus on evaluating measures of distinctiveness through an analysis based on synthetic texts. Recent work has shown the importance of both frequency and dispersion of words for keyness analysis (Gries, 2021). By conducting keyness analysis using synthetically created datasets and through inserting an artificial word with precisely-manipulated frequency and dispersion into the synthetic dataset, we aim to systematically uncover the characteristics of different measures. Our goal is to determine the sensitivity of each measure to variations in frequency and dispersion.