The house of the former Yugoslavian embassy in the Rauchstraße was erected on the grounds of a former villa of the Jewish industrialist Paul Mendelssohn Bartholdy, grandson of the famous composer Felix Mendelssohn Bartholdy. The building was designed by Werner March, the designer of the Berlin Olympic Stadium, and was erected between 1939 and 1940. March was obligated to follow specifications of Albert Speer, who was also General Building Inspector for Berlin. The sculpture elements were designed by Arno Breker, another prominent artist of the Nazi era. Initially, the building served as a residence of the Yugoslavian embassy. After the German invasion of Yugoslavia and Greece in spring 1941, the Yugoslavian state ceased to exist, and the various Reich and NSDAP departments set up their offices inside.

In May 1941, Alfred Rosenberg, one of the chief ideologists of the NSDAP, also established his office in the building, which would later become the Reich Ministry of the Eastern Occupied Territories (Reichsministerium für die besetzten Ostgebiete). The ministry received a task to concentrate experts and prepare plans for the occupational policy in the Eastern territories, which would be conquered by the Wehrmacht after the beginning of Operation Barbarossa. Nine days after the infamous Wannsee Conference, on 29 January 1942, the building was used for the following conference, which included sixteen participants from the Rosenberg Ministry, the Wehrmacht, RSHA, Party Chancellery and Ministry of Justice and was arranged to discuss implementation of the decisions of the Wannsee Conference. Nevertheless, the ministry moved its headquarters to the former Soviet embassy in the Unter den Linden street in summer 1942.

The building in the Rauchstraße was subsequently used as a guest house of the ‘Greater Germanic Reich’. The building was one of two buildings on the Rauchstraße that survived the air raids. From 1945 to 1953, it was used by the communist Yugoslavian military mission, and between 1953 and 1990 the Oberstes Rückerstattungsgericht für Berlin (Supreme Restitution Court for Berlin) had its office inside. Since 1995, the building is used by the German Council on Foreign Relations (Deutsche Gesellschaft für Auswärtige Politik).