The recent introduction of \texttt{ SAM3} has revolutionized Open-Vocabulary Segmentation (OVS) through \textit{ promptable concept segmentation}, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift (\textit{ data drift}) or conditional label distributions evolve (\textit{ concept drift}) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present \textsc{ ConceptBank}, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach (\textit{ i}) anchors target-domain evidence via class-wise visual prototypes, (\textit{ ii}) mines representative supports to suppress outliers under data drift, and (\textit{ iii}) fuses candidate concepts to rectify concept drift. We demonstrate that \textsc{ ConceptBank} effectively adapts \texttt{ SAM3} to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.
Categories: cs.CV