Salt marshes are valuable ecosystems that provide important ecosystem services. Given the global scale of marsh loss due to climate change and coastal squeeze, there is a pressing need to identify the critical extrinsic (wind exposure and foreshore morphology) and intrinsic factors (soil and vegetation properties) affecting the erosion of salt marsh edges. In this study, we quantified rates of cliff lateral retreat (i.e., the eroding edge of a salt marsh plateau) using a time series of aerial photographs taken over four salt marsh sites in the Westerschelde estuary, the Netherlands. In addition, we experimentally quantified the erodibility of sediment cores collected from the marsh edge of these four marshes using wave tanks. Our results revealed the following: (i) at the large scale, wind exposure and the presence of pioneer vegetation in front of the cliff were the key factors governing cliff retreat rates; (ii) at the intermediate scale, foreshore morphology was partially related to cliff retreat; (iii) at the local scale, the erodibility of the sediment itself at the marsh edge played a large role in determining the cliff retreat rate; and (iv) at the mesocosm scale, cliff erodibility was determined by soil properties and belowground root biomass. Thus, both extrinsic and intrinsic factors determined the fate of the salt marsh but at different scales. Our study highlights the importance of understanding the scale dependence of the factors driving the evolution of salt marsh landscapes.