The brain consists of a large number of structured regions responsible for diverse functions. Detailed region annotations with stereotaxic coordinates are rare, prompting the need of using one or very few available annotated results of a specific brain section to label images of broadly accessible brain section samples. Here we develop a one-shot learning approach to segment regions of mouse brains. Using the highly ordered geometry of brains, we introduce a reference mask to incorporate both the anatomical structure (visual information) and the brain atlas into brain segmentation. Using the UNet model with this reference mask, we are able to predict the region of hippocampus with high accuracy. We further implement it to segment brain images into 95 detailed regions augmented from the annotation on only one image from Allen Brain Atlas. Together, our one-shot learning method provides neuroscientists an efficient way for brain segmentation and facilitates future region-specific functional studies of brains.