Artesan.IA
Year: 2023
Type: Interactive Experience
Core Skills:
Research Coding 3D Design Prototyping Visual Design Machine LearningProject Objective
This project invites viewers to reflect on the future of artisan
craftsmanship and its possible evolving relationship with AI. It presents a contrast between
AI-generated artisan artefacts and authentic Colombian handmade pieces, providing insights into
the original meanings embedded in traditional patterns.
It is also important to acknowledge that many of the artisans in Colombia are part of Indigenous
groups, and therefore in talking about their objects this project is also talking about them.
This opened interesting questions about data appropriation and the ethics of talking about vulnerable
groups.
The Problem
With the rapid increase in generative AI’s accuracy and capacity to blur the line between real and generated, this project delves into the encoding of cultural meaning within the context of artisan artefacts, specifically Colombian artefacts. What happens when a generative model is capable of understanding what the “Balay basketry” from the Cubeo artisans is and produces designs with the right colours and materials? Or, could weaving techniques be passed down to generations with the help of generative AI?
Combined with the idea of awareness and education, ‘Artesan ia’ is intended to be an interactive experience in which people can learn about patterns that usually accompany Colombian artisan artefacts and question the role of generative AI in the context of cultural heritage objects. This project has three different elements intended to work in conjunction: a set of patterns displayed in cards, a box in which a computer vision model reads the patterns, and finally, a Processing sketch that reacts to each pattern.
Research
Artesan-ia has been inspired by different Interactive projects that seek to fuse cultural heritage knowledge with technology as a medium to explore concepts such as meaning, co-creation, and misrepresentation. In an attempt to dismantle the perception of apparent friction between what is ‘traditional’ and ‘technological’, Ars Electronica ran the Electronic Indigenous Art project in 2018. This project brought together indigenous and non-indigenous people to produce interactive creative work.
One of the outcomes, ‘The Earth that is Us’ (Electronica, 2019), is a project that plays with the patterns and meaning of body painting for the Brazilian Karapotó Plak-ô Indigenous community. For this project, the community would first draw the patterns that would later be projected onto their bodies, and on those of the audience.
Knots of Code
As another example, Paola Torres Núñez is a Peruvian transdisciplinary practitioner who explores the use of technology with ancestral knowledge (João Gabriel Ribeiro & Ribeiro, 2022). In her work Knots of Code, Paola explores the Quipu (a system of knotted tally cords used by the Inca Civilization to encode information) through computer vision. This work aimed to use technology to decode the possible meaning of the system by looking at the amount of knots, and the distance between them. However, it also meant an understanding of how the Incas used the system, which also implies the use of AI for archaeological research (Torres, 2020).
Development
As mentioned, ‘Artesan-ia’ was inspired by projects that fostered interactions in combination with the exploration and learning of cultural heritage content. The generated interaction involves a user, an input, and an output; and was very much influenced by the Yoto mini box, a product designed to play stories when a card is placed in the box. Each card triggers a story that is read out loud and which can be easily switched if the card is changed.
Following this principle, I sketched an interaction that initially aimed to communicate the real meaning of a pattern, v.s the distorted notions they have acquired. To achieve this, I had sketched two different boxes, one ‘transparent box’ that would tell me the real meaning of a pattern v.s an ‘opaque box’ that would tell me the distorted notion of the pattern. However, the interaction was simplified into a single box that receives input from a card (the pattern) and outputs expanded information about that card. The development of this project is divided into the design of the box, the design of the Processing sketch and the cards, and finally the design of the interaction (all elements working together).
The box
As a result of reading about Artificial Intelligence and the ethical considerations behind it, I stumbled upon the concept of the Black Box models. These models are created from data, with minimum input from humans, and often produce results that are complex to explain and are often opaque (Rudin & Radin, 2019). However, I found it more relevant to focus on the counterpart of these obscure models, that is, Interpretable models. In the overall interactive experience, the box is the part that receives the pattern from the user and produces an output. The way I chose to represent an interpretable model is by literally building a box from which all the training of my model is happening. All the elements that go inside it (the camera and the cards with patterns) are visible to everyone.
First Iteration
The purpose of the flowchart is to understand the overall user journey and guide the code structure of the app. By identifying main screens and interactions, I was able to start thinking programmatically and roughly understand the components and the file structure that make up the app. This flow chart came in handy when I needed to create the routing of the app and the different navigations during the code development.
Second Iteration
In the second iteration, I decided to add a pattern to the box. This design choice was initially aimed to help light come through the box, and therefore support the detection of the patterns by the camera. Later, it also became an analogy for the artisan artefacts I was researching.
Processing Sketch Design
Colombia is a multicultural country with more than 100 indigenous ethnic groups and about 65 different spoken native languages. Its geographical richness has allowed the country to be geographically divided into 6 distinct natural regions, each with its very distinctive climate, fauna and flora, customs, and people (Convention on Biological Diversity, 2015).
The Processing sketch intends to communicate the richness of the country and its makers. Based on the 5 mainland regions in which Colombia is divided, I identified and mapped the main artisan makers for each region and selected the ones that were better supported by veridic information. These communities were found through a collection of resources, such as books (Villegas & Jiménez, 2000) and online government archives (Artesanías de Colombia SIART, 2014).
Mapping the communities
After selecting the maker communities I organised them into a chart, and then located them in the Colombian map. For each region, I chose representative fauna and flora, buildings, or industries, and kept the geographical division clear through the variation of colous and clear division lines.
Interactive elements
The interaction of the Processing sketch relies on the X and Y positioning of an element reaching specific areas of the sketch. When the camera reads a pattern, an element will move to a specific X and Y position within the map, and trigger the opening of a new window.
To achieve this, I created specific sections within the sketch that covered a range (red squares on the map). The elements that trigger the pop-up windows when entering a given range are the yellow butterflies, which move across the map according to the pattern the user enters in the box. I used yellow butterflies mainly because it is an important reference to Colombian writer Gabriel Garcia Marquez's magic realism and his book “One Hundred Years of Solitude”.
Map states demo
When the butterflies reach a specific position, they seamlessly initiate the opening of an overlay window within the sketch. This window not only provides additional information about the introduced pattern in the box but also encourages the reader to contemplate the role of AI by showcasing AI-generated objects. Below, you'll find three examples of these overlay windows in action which talk about the Amazon, Andean, and Pacific regions.
The Pattern Cards
As part of the interaction, I designed 5 different cards displaying patterns that covered the 5 different regions in which Colombia is geographically divided. The main objective of selecting these specific patterns was to show the multiculturality of the country, as well as its richness in natural resources. Additionally, it is important to understand that the areas that artisans inhabit inform their use of materials, techniques, colours, and ultimately the artefact’s patterns.
Decoding the Pattern
The purpose of the cards with patterns is to enable the interaction between the user and the Processing sketch. The interaction between the butterflies and the cards is managed through Wekinator, an open-source tool compatible with Processing that allows the training of neural networks for image recognition.
Using this software, I mapped each pattern to a specific area in the map. This means that when the camera identifies certain colours and configurations in the image, the butterflies will move to the section assigned to that specific pattern. To test this principle, I sketched some quick patterns and ran a Processing sketch that reduced the camera input to 100 pixels. Reducing the number of pixels the camera needs to analyse helped the model training to be faster.
Model training demoCards
After mapping the patterns with their corresponding geographical area in the map of Colombia, I refined the design of the cards. The pattern would cover most of the space since the camera needs to interpret the image and trigger the interaction. At the top of the card, I added the name of the Artisan artefact, then I added information about the Region and location in which the artefacts are usually made and the name of the main community maker. When the card is inserted in the box, the user can still see the name of the artisan artefact, but the pattern will trigger a location change in the map, and show more information about the artefact.
Outcome
The project outcome is 'Artesan.ia', an Interaction that brings together encoding of meaning and computer vision to promote awareness and interest in artisan artefacts. The 'medium is the message' (McLuhan, 1995) and in this project, the medium is the computer vision model. It is the making of the model and its constraints that have informed the making of the project. The three stages of the process for this interaction are:
- 1. Input: a card enters the box.
- 2. Training interface: the camera reads the image, and through Wekinator, a position is mapped to match the pattern with its corresponding location. Then the model is trained to match different patterns to different locations.
- 3. Output: the Processing sketch reacts to the card inserted, primarily by changing the position of the butterflies in the map and opening information popups along the way.
Reflection
This project took me to different places. Starting with the encoding of meaning it turned into an understanding of the inside workings of Artificial Intelligence models. The role of machine learning is to process information, in this case, this information has cultural meaning to us, but in reality, the model can’t acknowledge its historical importance. We consciously grant meaning to that information and data.
Finally, this project also attempts to inform people and create an interactive experience. If further developed, this project could be improved, starting with the actual training of the computer vision model. It could be enhanced to capture better details of the patterns and ‘resist’ changes in light. The flow between the elements could also be improved, with changes in the box's architecture, and the different screens' coding.
Bibliography
- Artesanías de Colombia SIART, – Sistema de Snformación para la Artesanía (2014) Artesanias de Colombia, Artesanias de Colombia – Artesanías de Colombia. Available at: https://artesaniasdecolombia.com.co/PortalAC/General/template_index.jsf (Accessed: 21 December 2023).
- Convention on Biological Diversity (2015) Colombia – Main Details, Convention on Biological Diversity. Available at: https://www.cbd.int/countries/profile/?country=co (Accessed: 20 December 2023)
- Electronica, A. (2019) The earth that is us, Out of the Box. Available at: https://ars. electronica.art/outofthebox/en/theearththatisus/ (Accessed: 18 January 2024).
- João Gabriel Ribeiro, R.P. and Ribeiro, J.G. (2022) Paola del Prado USA ia para mapear possibilidades Além Do Eurocentrismo, Shifter. Available at: https://shifter.pt/2022/07/ paola-del-prado-artista-ia-human-entities/ (Accessed: 18 January 2024).
- McLuhan, M. (1995) ‘The medium is the message’, in Understanding media: The extensions of man. MIT-Press: Cambridge, Mass., pp. 1–18.
- Rudin, C. and Radin, J. (2019) Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition, Harvard Data Science Review. Available at: https://hdsr.mitpress.mit.edu/pub/f9kuryi8/release/8 (Accessed: 01 January 2024).
- Torres, P. (2020) Knots of code, Knots of Code. Available at: https://khipucamayoc.github. io/ (Accessed: 18 January 2024).
