DH^CH@ISR 2021 / Presentations

Multimodal interrelations have been analyzed across disciplines to gain insights into the contextual appearance and meanings of signs or semiotic artifacts. In this project a computational approach to the analysis of interrelation structures in image-text compositions is suggested. For this purpose computer vision and Natural Language Processing methodology are combined with a background in art history and studies of multimodal semiotics. A case study in this respect are early modern emblems, which offer an allegorical composition of text – a motto and a description – with an image. Visual, as well as textual features of this data will be investigated concerning their contextual appearances. Furthermore, metadata units are taken into account, which provide a fruitful basis for research in data-relevant fields. The project is planned to range from the development of a feature extraction framework that is both general, as well as adaptable to the nature of a defined variety of image-text compounds over the analysis of their distributions to analyses and visualizations of the results.

Based on the artworks made in a research-creation PhD, and weaving links between computer science, art history, cognitive sciences, philosophy and psychoanalysis, my talk will try to unveil new visions of Artificial Intelligence (A.I.).
A.I. is at the center of many fantasies and polemics, often contradictory. As a machine, it appears as a computer system both neutral and logical; and yet, it is mainly a mysterious
black box full of biases. As a technological advance, it holds the promise of a safe and wonderful future; but it is also a threat for our freedom as a tool for extended surveillance. As an external intelligence, it offers to unburden our brains and increase their limited abilities; but for now, AI mostly brings out our worst tendencies... There might be a common cause for these apparent inconsistencies: A.I. surely imitates
some aspects of our reasoning, – but mainly its instinctive, non-reflective thinking. In other words, more precisely than an abstract “intelligence”, A.I. would be better described as an
artificial unconscious. And it questions our own paradoxes of (ir)rational beings. Which “unconscious” are we talking about here? What does A.I. say about us, humans? What relationships can we foster with this other we are obsessed about?
To consider these questions as a whole, rational thinking might insufficient: in meeting another unconscious, we could stay in a dead-end if we do not open our own. Therefore, as a
domain whose object is our feelings, and a thinking that involves more than mere reason, Art appears as the most relevant field to tackle these challenges.
In this talk I will explore the unreasonable fantasies of the machine – and of its creators. I will bring the audience to a journey through the dreams of the machine, its fears, its
paradoxes; I will try to open a bridge between organic and metallic neurons.

This presentation addresses the potential for computational practices within archival and museological domains by charting a positional shift occurring in digital archives; from working with an object orientation and containment – to computing with a dimension orientation, segmentation, analytics and visualization. Computational museology allows us to conceive new trajectories that link all forms of cultural materiality: objects, knowledge systems, representation and participation. These topics will be explored through research at the Laboratory for Experimental Museology. It is a transdisciplinary initiative at the intersection of cultural heritage, imaging technologies, immersive visualisation, visual analytics and digital aesthetics. eM+ engages in research from scientific, artistic and humanistic perspectives and promotes a post-cinematic multisensory engagement using experimental platforms. The research explores the ways in which mechanistic descriptions of database logic can be replaced and computation can become ‘experiential, spatial and materialized; embedded and embodied’, a landscapes for the senses. The title for this talk ‘computational museology’ is a framework that unites machine intelligence with data curation and ontology with visualization and immersion.

The talk will focus on the Swiss photographer Werner Nefflen (1919-2014). However, it is not about his professional work and his commissioned works, but about photographs taken in his spare time. Nefflen was close to nature and often hiked in the mountains in the summer. In the process, he photographed numerous mountain farmers at making hay and building haystacks.
Werner Nefflen's work is now kept in the historic museum of Baden and catalogued in an archival information system. A large part of the images have been digitized and are publicly accessible. As is common in archives today, the images can be searched via text queries, but this often leads to unsatisfactory results.
In my talk I will apply the new ontology "Records in Contexts" RiC-O (https://github.com/ICA-EGAD/RiC-O) to contextualize Nefflen's vacation images from different perspectives, both in terms of content, geography, but also in terms of publication. Using example images, I will show how a photographer's legacy can be described and accessed in new ways.

Digital Image Archives, Max Frischknecht investigates how visualizations and interfaces in digital cultural collections support participatory use. The research focuses on three aspects: The participation and involvement of researchers at universities and the general public in enhancing the collection through the interface, the use of visualizations in exploring and presenting themes in the collection, and the use of machine learning in communicating the cultural collection on a visual and content level. The results of the study will be incorporated into the development of a new participatory image archive for the Swiss Folklore Society.

There are millions of images from the past that we have never seen, but whose digital availability forces us to ask the same question that was asked about texts fifteen years ago: "What Do you Do with a Million Books?" We all agree on this: digital technology can help us to "do" something with these millions of images that have been seen by others and not by us. Of course, in order to talk about images that we have not seen, we need to know what computers can “say” about them—or, rather, we need to know what kind of information from images is given and can be given to computers. Two elements can be given to computers: Textual metadata (indirectly linked to the image, such as the author and spatio-temporal context, or specific to it, such as a title, the medium, or a description of content), and computationally analyzable pixels. Does this allow us to talk about images, or about anything else? That is where the positions are not necessarily the same as to the computational approach to images: computation contravenes the ethics of art history according to which one does not speak of what one has not seen, and where it is ill-advised to assume that one is speaking of a reproduction and not of an original. “Distant viewing" is then easily relegated to the sidelines of social history, naive cultural analytics, art geography, or heritage engineering. My argument will be to defend the interest of historical and visual studies that speak of what they have not seen. Beyond the simple contribution of discussing forgotten images and artists and reconstructing the power dynamics in social and cultural exchange —i.e a contribution to the social and the global history of art—the computational approach encourages us to revisit questions that art history had put aside, such as the birth and development of style. Above all, it forces us to consider the political stakes of an aesthetic theory of images that would no longer prefer to discuss images in the singular. In the age of machine surveillance, building a theory about images that we have not seen, will not see, and that no one else will, is of dire importance.

In the context of Computer Vision applied to Digital humanities and Art history, we currently miss a proper training dataset of artworks for body/object recognition that addresses the diversity and uncertainty linked to the large spectrum of non-naturalistic artistic depictions.
In this talk, I introduce and discuss the possibility of using Data Augmentation on an existing labeled dataset of real images in order to generate loo-alike paintings as an alternative path. Additionally, I aim to engage the conversation on the creation of a labelled dataset of artworks for object recognition, if such a model should require a specific ontology and what could be he benefits for this field of research.

This talk presents a novel digital edition of Gottfried Semper's magnum opus Style, comprising manuscripts and other sources identified as forming main parts of its evolution and contextualizing it within its intellectual milieu both within and outside of Semper’s corpus. The Edition’s implementation combines the established paradigm of TEI XML with the new modes of representing text made possible by labeled property graph databases. The talk summarizes the Edition's workflow and explains its technical processes to enable critical engagement of readers and scholars. It also presents the current and projected content of the Edition and the outlook for its technical and functional development.

Intangible cultural heritage (ICH) as a site for digital efforts has grown significantly since 2003 as UNESCO affirmed the significance of preserving "the practices, representations, expressions, knowledge, skills". Unlike tangible heritage which is conventionally protected via legislation and object-based conservation, intangible cultural expressions are usually embodied, enacted and continuously recreated. It is thus crucial for us "digital custodians" to find appropriate ways to transform the living knowledge from ephemeral nuances to tangible cultural entities. As such, this presentation will showcase a machine-enhanced approach and its application to retrieving and representing motion as meaning from multimodal cultural datasets.

As a real store of knowledge, the museum's collections require continuous research and accessibility. The digitalization and networking of this knowledge have opened up entirely new possibilities for museums with respect to both exhibitions and online offerings. Fundamental to successful knowledge transfer is the digitalization of the collections and the curation of digital content. With the help of concrete examples, students receive insights into the diverse and complex content that the world of objects offers for the Digital Humanities.

One of the biggest challenges of the 21st century is how to handle and take advantage of the massive and ever-growing amount of data. Cultural heritage institutions are increasingly converting their collections into the digital space as the pressure towards digitalization and open access rises, which was extremely accelerated during the COVID-19 pandemic.
By transitioning into the digital space, cultural heritage institutions are facing similar challenges as research institution when it comes to data quality, long-term storage and life-cycle management. Interoperability and therefore data standards, semantics and ontologies are important factors to enable open access for cultural heritage collections and to link data across institutional borders.
Data curation practices address these issues, but simply digitizing and permitting online access to cultural heritage collections does not ensure visibility and long-term interest. This talk aims to discuss and explore new theoretical and practical approaches to digital curation using the example of the Digitales Schaulager, Basel.

The presentation focuses on:
1) the digital workflow for long-term preservation and the web dissemination of the Vatican Digital Library.
2) the Library's IIIF initiatives and the implementation of the "Thematic Pathways on the Web: IIIF annotations of manuscripts from the Vatican collections".

Within the multidisciplinary project "Participatory Knowledge Practices in Analogue and Digital Image Archives" funded by the Swiss National Science Foundation (SNSF), we aim to design a visual interface with machine learning-based tools.

This interoperable platform will make it easy to annotate, contextualise, organise and link both images and their metadata to deliberately encourage the participatory use of the archives from the Swiss Society for Folklore Studies (Schweizerische Gesellschaft für Volkskunde -SGV).

One of the most important components that will support this effort is to leverage the APIs developed by the International Image Interoperability Framework (IIIF), not only to facilitate external exchange but also the interaction between the various in-house software packages that will be deployed.

The presentation will be an opportunity to present the data and intertoperability model of the project, currently in the conceptualization phase.

The presentation draws upon an ongoing collaborative research project Ways of Machine Seeing that first developed at University of Cambridge (2016), to explore the entanglement of machines and their ways of seeing in the light of developments in machine vision, and most recently in a special issue of AI & Society (2021). Central to this work has been Berger’s assertion (in Ways of Seeing) that: “The relation between what we see and what we know is never settled.” If, broadly, these relations are even further unsettled by developments in algorithmic learning and machine vision, then how to understand the ways in which images act in the world, and their effects? To further explore these issues we take a close look at ImageNet, a visual dataset of more than 14 million images, used to train AI algorithms, and ask how it operates in terms of social reproduction. Berger begins his book with the statement: "Seeing comes before words. The child looks and recognises before it can speak." Whilst we might be uneasy with aspects of this, it is quite common to draw analogies between intelligence in machines and cognitive development in humans, especially in children. This is broadly a (constructivist) idea of learning as something informed by, and learning from, experiencing the world. Yet this can also appear superficial, as for instance, in the example of Fei-Fei Li describing her insight into teaching a machine to see, and informing the development of ImageNet. Berger looks to children for different reasons, to expose some of the ways in which seeing and knowledge come together, and by extension, not for machines to learn to see like humans, but to see differently, to unlearn to see in ways that are "trained". The recent breakthroughs in computer vision rely on the development of machine learning, a technology that learns to detect regularities from large collections of samples. For machines to learn how to see, datasets must provide visual data whose variations reflect those encountered by algorithms when they are used in production (in the “real” world). Here scale matters and state of the art datasets use millions of images. These datasets define the boundaries of what machines will be able to parse in scenes or images. They are also annotated (images are labelled, tagged and organised according to taxonomies) which makes Berger's questions about the relation between words and images a problem computer scientists have to resolve on a daily basis. Millions of images are described around the clock to provide the AI industry with the annotated data it needs to operate. The dataset can be said to be the algorithm's worldview, which raises the question of what this worldview is, how it comes into being, and more importantly how it can be challenged. Our proposal is to develop a dataset based on the situated knowledge of annotators. By introducing changes in the process of creation and annotation of datasets, we want to explore the potential for divergence in machine vision. Through small changes in the parameters of the annotation process, we seek to discover how machine behaviour is affected. What happens when the annotator does not conform to the implicit expectation of a disembodied neutral observer? If annotation is made by children (or another disenfranchised group), how does it impact what the algorithm is able to recognise and its ways of making sense of the world? How might other ways of training elicit other ways of seeing, and in ways that are socially transformative?

Digital collections of art museums are a data source (images and work information) for creating interactive visualizations. The content of data visualizations and the arrangement of images depend on metadata. This presentation aims to address different roles of metadata regarding visualizations and their relationship to images. Metadata is used to represent both object information and the underlying structure of visualizations, and can be extracted from image files by algorithms or enriched with information and linked to each other via vocabularies and ontologies. Examples of collection visualizations from the GLAM domain will be used to explain types of metadata inherent in digital collections and the purposes they serve. Furthermore, fundamental design concepts such as interaction with digital collections, generative visualizations, and the generous user interface will be considered. Interactive exploration of visualizations with changing appearance serves as a prerequisite for interpretation and gaining knowledge. Metadata-based collection visualizations are discussed in light of their usefulness and limitations as a digital method for accessing and researching digital collections in digital humanities, art history, and museums.

The research Project “Piranesi and his workshop”, started in 2017 in collaboration between the Kunsthalle Karlsruhe and the Università della Svizzera italiana. One of its missions is to publish an online catalog presenting the recently rediscovered drawings used in workshop work, mostly during the last twenty years of the career of Giovanni Battista Piranesi (1720-1778) and afterwards by his son Francesco (1758?-1810). The choice of using a digital format for this catalog, which is still under development, has brought out interesting issues regarding the presentation modalities of the material and the results of the research. In other respects, the use of some of the digital tools available online has proven indubitably helpful for the study of the drawings, as for example the high-resolution digitalization of piranesian prints or a facilitated access to the collections of museum, libraries, or archives through online databanks. These databases have been used to trace a great number of artistic personalities in relation to the context and reconnect different drawings that are nowadays scattered between various collections in Europe and in the United States. Even though an online visualization of the drawings, lacking the material qualities and intrinsic nature of the 17th century document, cannot compare to the real-life version, the digital image can nonetheless offer many presentation possibilities and useful solutions for an efficient scientific divulgation, taking account of the various scales and levels of the analysis of the drawings. The talk offers to explore the issues and presentation choices met in the creation of this catalog, and the advantages and limits of digital tools in this context.