The increasing deployment of robots in urban spaces calls for design strategies to ensure their adaptation and to mitigate potential disruptions to complex urban contexts. Our research aims to initiate the discussion of contextual adaptability issues of urban robots by exploring everyday scenarios where their presence would appear out of place. We created a design probe for people to carry in their daily lives, facilitating them to envision the robot's presence and capture scenarios where a robot seems to be disruptive. We collected data by distributing the probes among the research team and conducting a city walk activity using the probe at a workshop. This paper presents factors arising from the collected scenarios, encompassing temporal, spatial, cultural, and social dynamics, as well as various stakeholders that robots need to adapt to. These findings provide a blueprint and potential research directions for future research into robot contextual adaptability in urban environments.