With the development of internet of vehicles and automated driving, individual-based trajectory control at intersections becomes possible. Trajectory planning and coordination for connected and automated vehicles (CAVs) have been studied at isolated “signal-free” intersections and in “signal-free” corridors under the fully CAV environment in the literature. Most existing studies are based on the definition of approaching and exit lanes. The route a vehicle takes to pass through an intersection is determined by its movement. That is, only the origin and destination arms are included. This study proposes a mixed-integer linear programming (MILP) model to optimize vehicle trajectories at an isolated “signal-free” intersection without lane allocation, denoted as “lane-allocation-free” (LAF) control. Each lane can be used as both approaching and exit lanes for all vehicle movements including left-turn, through, and right-turn. A vehicle can take a flexible route by way of multiple arms to pass through the intersection. In this way, the spatial–temporal resources are expected to be fully utilized. The interactions between vehicle trajectories are modeled explicitly at the microscopic level. Vehicle routes and trajectories (i.e., car-following and lane-changing behaviors) at the intersection are optimized in one unified framework for system optimality in terms of total vehicle delay. Considering varying traffic conditions, the planning horizon is adaptively adjusted in the implementation of the proposed model to make a balance between solution feasibility and computational burden. Numerical studies validate the advantages of the LAF control in terms of both vehicle delay and throughput with different demand structures and temporal safety gaps.