Teaching AI to Understand Historical Maps Using Legends

Projekt: Semantic segmentation of historical maps

Forschende: Yunshuang Yuan, Frank Thiemann, Monika Sester

Projektidee: Historical maps are treasures of the past — they record landscapes, settlements, and land use patterns that no longer exist today. To unlock this valuable information, researchers are turning to artificial intelligence (AI). This project aims to train AI models to automatically recognize and locate different elements (like forests, rivers, or towns) on historical maps using a technique called semantic segmentation. In simple terms, semantic segmentation teaches an AI model to look at every pixel in an image and decide what that pixel represents.

Why It’s Challenging

Traditional AI models for semantic segmentation are trained on specific datasets — each with a fixed set of categories, such as “road,” “building,” or “water.” However, historical maps vary widely in their styles, colors, and symbols. Because of this diversity, a model trained on one map collection often performs poorly on another.

Creating large training datasets with detailed “ground truth” labels for every map is extremely time-consuming and expensive. To overcome this problem, our team adapts a foundation model known as SAM (Segment Anything Model) — an advanced AI model originally trained on everyday photographs — and fine-tunes it for historical maps.

Adapting SAM for Historical Maps

Although SAM is powerful, it doesn’t automatically understand the meanings behind what it segments — it simply separates shapes and regions based on visual cues. It also needs user input (like clicks or boxes) to tell it what to segment. This type of input is called a prompt.

To make SAM more useful for maps, we developed a new, fully automatic and semantic-aware version of the model. Instead of relying on clicks, our model uses a legend-like prompt — a small image-label pair that serves as an example.

• The image is a small patch from the map (for instance, 384×384 pixels).
• The label is a simple mask showing which part of that patch represents the target feature (for example, woodland or grassland).

The AI then uses this legend-like example to recognize the same feature elsewhere on the map — even in regions it hasn’t seen before.

How the Model Works

As shown in Figure 1, the model takes two inputs:

1. A target map image (the area we want to analyze).
2. A source example (the image-label pair, acting like a digital map legend).

Using these, the model predicts which areas of the target map match the example.

It does this through three main modules:

• Image Encoder: Converts images into a compact “latent” representation. This part is borrowed from the pretrained SAM.
• Prompt Encoder: Combines the example image and its label into a guiding signal for the model.
• Mask Decoder: Produces the final segmented output — highlighting the target class (e.g., forest or field) in the target map.

To make training more efficient, we also use DoRA, a low-rank adaptation method that reduces the number of parameters that need fine-tuning — saving both time and computing resources.

Experiments and Results

We tested our approach on several historical map collections, including Hameln and Donauwörth. Some examples of the qualitative results are shown in Figure 2.

The model successfully learned to segment maps based on different prompts. For example, when given a sample of Grassland from one part of a Donauwörth map, it could automatically find and highlight all Grassland areas in another part of the same map. Similarly, when prompted with Woodland, it segmented forested areas accurately.

This demonstrates that our model can flexibly recognize and distinguish different land types across diverse historical maps — guided only by simple, legend-style examples.

Conclusion

By teaching AI to “read” historical maps using legend-like examples, we bridge the gap between semantic understanding and automated segmentation. This approach not only reduces the need for expensive manual labeling but also opens new possibilities for large-scale analysis of cartographic archives — helping historians, geographers, and researchers explore the past more efficiently than ever before.