Switching and routing are critical functionalities for a reconfigurable bandwidth-dense optical network, and great efforts had been made to accommodate mode-division multiplexing technology. Although the reconfigurable routing for spatial-mode groups between different optical paths was realized recently, a demultiplexing-switching-multiplexing process is necessary. Here we present a simplified and compact on-chip 2×2 multimode switch that can be easily upgradable to a larger scale. Fully and reconfigurable routing between not only optical paths but also spatial modes is achieved. To obtain a low loss multimode processing, a novel structure free from demultiplexing and re-multiplexing operations is adopted. The switch enables minimum and maximum insertion losses of 0.3 and 1.2 dB, with a compact footprint of 433 μm×433 μm and low crosstalk of <−16.6 dB for all channels. It is further extended to two types of 4×4 switch fabrics with cross-bar and ring-bus architectures, as demonstrations of high-level integration. System characterization with 32 Gb/s high-speed modulated signals is also carried out, reaching up to 256 Gb/s aggregate throughput. These results verify a general solution of 2×2 multimode switch for reconfigurable inter-mode and inter-path routing applicable in large-scale and high-density multimode optical network.