A key objective of our institute is the development of innovative concepts, processes, and technologies that strengthen the resilience of terrestrial infrastructures. In the department "Digital Twins for Infrastructures”, we leverage digital twin technology to support the monitoring and evaluation of critical infrastructure and thus the continuous enhancement of their resilience. Our work focuses particularly on the automated generation of digital twins, simulation-based VR applications for training first responders and analyzing threat scenarios, and the cloud-based deployment of digital twins to ensure a robust, location-independent use in crisis situations.

What to expect 
Digital twins and virtual replicas of physical entities, processes, or systems have a broad range of applications across various industries, including crisis management, evacuation planning, and emergency response training. Their ability to integrate physical environments with digital representations enables organizations to simulate, monitor, and optimize real-world assets and operations. However, creating detailed and functional digital replicas of outdoor environments often requires extensive manual effort, particularly when open data sources lack essential metadata including structural dimensions or facade features of buildings. These limitations hinder the potential of digital twins in high-stakes scenarios (e.g., rescue operations) where spatial accuracy and contextual detail are critical. To address this challenge, leveraging Vision-Language Models (VLMs) in combination with diverse sensor data offers a promising solution. These models can facilitate automated extraction and enrichment of building information, significantly reducing manual workload and enhancing the utility of digital twins in operational and planning contexts.
Therefore, the objective of this work is to investigate how VLMs can be employed to automatically infer and enrich missing building information from sensor data to support the scalable creation of high-fidelity digital twins for safety-critical applications.

Your tasks

• evaluating the ability of state-of-the-art VLMs to interpret and semantically reason over sensor inputs (RGB images) within the context of buildings
• developing a VLM-based pipeline to infer and integrate missing structural and functional building components into incomplete digital models using available sensor data and partial prior knowledge.
• assessing the practical effectiveness of the proposed pipeline in representative use cases

Your profile

• currently enrolled as a Master student in computer science, mathematics, optical engineering or similar major
• experience in deep learning, especially in terms of image-based object detection or semantic segmentation
• basic Python skills (ideally with respect to deep learning applications, e.g., PyTorch or Tensorflow)
• experience with VLMs or foundation models in general is beneficial, but not required
• knowledge of the German language is preferable, but not required

 We look forward to getting to know you!

If you have any questions about this position (Vacancy-ID 2398) please contact:

Tobias Koch 
Tel.: +49 2241 20148 55