EXPANDED SENSORIUM
Discussion participants:
Harry Meadows, artist and lecturer
Pedro Latas, multimedia artist, composer, and sound artist
Zikai Peng, industrial designer
Harry Meadows, artist and lecturer
Pedro Latas, multimedia artist, composer, and sound artist
Zikai Peng, industrial designer
Cover image: 'Touch Intelligence' by Zikai Peng
Interviewer: Ksenia Kopalova
In 2023, Marjolijn Boterenbrood drove her van from Amsterdam to Bishkek, carrying a stack of old Soviet maps that she reworked, incorporating elements of the Dutch landscape into them. In Kyrgyzstan, she invited artists to draw, paint, and add elements on top to reflect on the local landscape and their relationship with it. In an exhibition Connecting Geographies #5, she shows edited maps of 31 artists from Central Asia, her project Letter to a Silk Road, and new works resulting from her trip to Kyrgyzstan. In this interview Marjolijn and 3 of the artists participating in the project - Munara Abdukaharova, Zulya Esentaeva, and Malika Umarova - talk about their collaboration, mapping presence, and visualising personalised location narratives.
'Touch Intelligence' by Zikai Peng
Finally I found a way to talk about it: to make work not about Kyrgyzstan, but about our connection
Ksenia
What comes to mind when you think of “expanded sensorium” and “digital bodies”?
Pedro
When I think of digital systems, I also inevitably have to think about all the manners of translation of information that you’re obliged to engage with – like when you’re on a Zoom call, the camera picks up the light, transforms it into binary data and electrical impulses transmitted by fibre cables under the ocean. And then there’s a reverse process: the light becomes images on the screen – images of a person you’re talking to almost in real time. Our “meat bodies” undergo so many processes of transformation here. Some of our identities are indeed embodied in these digital forms. That’s fascinating to me – all these layers of translation, as well as the fact that we are translatable through media in principle.
Zikai:
The first thing I thought about in relation to ‘digital bodies’ as such was the difference between the digital and real. With physical touch, the signals come from our skin, whereas with digital touch it’s not the case: some sensors provide information much richer than what our skin can provide – with all the information on humidity, temperature, pressure, etc. Thus digital touch could give humans new ways to understand the physical world – a richer feedback system.
What comes to mind when you think of “expanded sensorium” and “digital bodies”?
Pedro
When I think of digital systems, I also inevitably have to think about all the manners of translation of information that you’re obliged to engage with – like when you’re on a Zoom call, the camera picks up the light, transforms it into binary data and electrical impulses transmitted by fibre cables under the ocean. And then there’s a reverse process: the light becomes images on the screen – images of a person you’re talking to almost in real time. Our “meat bodies” undergo so many processes of transformation here. Some of our identities are indeed embodied in these digital forms. That’s fascinating to me – all these layers of translation, as well as the fact that we are translatable through media in principle.
Zikai:
The first thing I thought about in relation to ‘digital bodies’ as such was the difference between the digital and real. With physical touch, the signals come from our skin, whereas with digital touch it’s not the case: some sensors provide information much richer than what our skin can provide – with all the information on humidity, temperature, pressure, etc. Thus digital touch could give humans new ways to understand the physical world – a richer feedback system.
I tend not to separate physical and digital – they enhance each other. They are designed together and made to emulate each other.
Ksenia
How does digital interaction influence your physical experience of the world? Do you feel your real-life interactions are shaped by digital processes?
Pedro:
Digital systems are made as a reflection of human experience – it’s a mirror of how we perceive the world. This technological mirror gets closer to the real world, but it also feeds back into us: it’s a weird feedback loop, that’s reflected even in our language. I tend not to separate physical and digital – they enhance each other. They are designed together and made to emulate each other.
Zikai:
In ‘Touch Intelligence’, pressing differently textured buttons triggers AI-generation of music that expresses different emotional states. In this sense, the physical touch and the digital have the same aim – conveying emotion. Touch can change emotion: a rough button can change the feeling to ‘tension’, for instance, while a smooth one can evoke the sensation of calmness and relaxation, and the generated music can express it back – that’s how the physical and the digital are linked in my project.
How does digital interaction influence your physical experience of the world? Do you feel your real-life interactions are shaped by digital processes?
Pedro:
Digital systems are made as a reflection of human experience – it’s a mirror of how we perceive the world. This technological mirror gets closer to the real world, but it also feeds back into us: it’s a weird feedback loop, that’s reflected even in our language. I tend not to separate physical and digital – they enhance each other. They are designed together and made to emulate each other.
Zikai:
In ‘Touch Intelligence’, pressing differently textured buttons triggers AI-generation of music that expresses different emotional states. In this sense, the physical touch and the digital have the same aim – conveying emotion. Touch can change emotion: a rough button can change the feeling to ‘tension’, for instance, while a smooth one can evoke the sensation of calmness and relaxation, and the generated music can express it back – that’s how the physical and the digital are linked in my project.
Harry:
Digital touch is mediated by machines. We are in a position where we have to rely on these extensions of our senses to make sense of the world beyond us. A lot of my work has been around climate sensing and its apparatus. This mediation has inherent bias, since it’s selection and omission of information needs to be focussed for a particular product . Instead, my research is looking into a multiplicity of ways of sensing. Apart from standard modes of sensing, I’m looking at amateurs and enthusiasts who set up their own climate sensing systems that are intuitive or experimental and offer alternative ecological maps.
For instance, ‘Personal Ecologies: The Community Gardener’ is a work I made with a community gardener who carefully raises tortoises in climate-controlled fridges, with a complex system of lamps and heat control systems for their hibernation. This is outside of standard practice of scientific sensing because she combines standard climate sensors with wider experimental systems of plants, animals and rather radical social organisation. I’m interested in how we as artists can celebrate sensing outside of its standard modes. How do the laboratory modes of sensing combine with intuition and social engagement? These combinations reveal a multiplicity of environmental sensing, and complexity in the relationship between machines, plants, animals and geology as sensors.
There are two examples of ecological mapping I’m particularly interested in: one is eDNA, which helps to identify all the species that might live in, let’s say, a lake, or a forest – through relatively tiny little samples of water or soil.
Another example is the Food and Agriculture Organization of the United Nations, who have developed methods of encoding forestation/deforestation in parts of the world that export food. Their process acknowledges the relationship between the digital map and ground truths of the people who live and work in this ‘digital map’.
So I’m interested in how art can explore ground truths and the social reality of mechanical sensing.
Digital touch is mediated by machines. We are in a position where we have to rely on these extensions of our senses to make sense of the world beyond us. A lot of my work has been around climate sensing and its apparatus. This mediation has inherent bias, since it’s selection and omission of information needs to be focussed for a particular product . Instead, my research is looking into a multiplicity of ways of sensing. Apart from standard modes of sensing, I’m looking at amateurs and enthusiasts who set up their own climate sensing systems that are intuitive or experimental and offer alternative ecological maps.
For instance, ‘Personal Ecologies: The Community Gardener’ is a work I made with a community gardener who carefully raises tortoises in climate-controlled fridges, with a complex system of lamps and heat control systems for their hibernation. This is outside of standard practice of scientific sensing because she combines standard climate sensors with wider experimental systems of plants, animals and rather radical social organisation. I’m interested in how we as artists can celebrate sensing outside of its standard modes. How do the laboratory modes of sensing combine with intuition and social engagement? These combinations reveal a multiplicity of environmental sensing, and complexity in the relationship between machines, plants, animals and geology as sensors.
There are two examples of ecological mapping I’m particularly interested in: one is eDNA, which helps to identify all the species that might live in, let’s say, a lake, or a forest – through relatively tiny little samples of water or soil.
Another example is the Food and Agriculture Organization of the United Nations, who have developed methods of encoding forestation/deforestation in parts of the world that export food. Their process acknowledges the relationship between the digital map and ground truths of the people who live and work in this ‘digital map’.
So I’m interested in how art can explore ground truths and the social reality of mechanical sensing.
'Personal Ecologies: The Community Gardener' by Harry Meadows
Research process behind the project
What if the local knowledge is more important and valuable that the more universal one?
Ksenia
This makes me think about the inevitable mistakes and gaps that happen in these translations from mechanical sensing to the lived experience and vice versa. Pedro, in your project ‘Can we feel touch when we're made of light’ you seem to explore these gaps – could you talk about that?
Pedro:
I was thinking about the impossibility of fully translating human touch into data. Machines can only create flawed representations, since in the process of translation some data is inevitably lost – machines can’t capture everything. Our screens do not even have enough pixels; it’s impossible to be aware of all the molecules making up someone’s body. I’m interested in the comfort of error, the impossibility of perfect interaction.
My performances explore “non-touch” feedback – like a microphone’s feedback loop. Two un-touching bodies, separated by a curtain, reaching for each other through technology.. It’s a metaphor for digital connection: veiled, partial, yet creating something new.
When Harry mentioned climate sensing, what came to mind was Timothy Morton and the idea of hyperobjects: we can only access a sliver of information in the mass of the world. As human beings we are not designed to understand phenomena in their fullness, because if we were we would probably explode! Even though we have the AI power, it’s very important to understand the failures at the edges of destruction.
Harry:
I love your use of a mechanical process like ‘feedback’ as a metaphor. Digital encoding should be seen as a process of mapping, but it can be confused with the territory. Encoding/mapping can be seen as the creation of a metaphor as much as a measurement. As artists, we use metaphors to create images to make meaning from things like clouds. In my work I use the metaphor of a ‘codec’: something being coded and decoded, but within that system, there’s a black box in the middle. I’m trying to put a human figure in this middle ground. For instance, I’m encoding an ecologist’s practice by creating a video game, which I’m showing to them and asking: is this what you do? Is this accurate? I’m trying to put a human conversation in the place of the machine to do the decoding.
This makes me think about the inevitable mistakes and gaps that happen in these translations from mechanical sensing to the lived experience and vice versa. Pedro, in your project ‘Can we feel touch when we're made of light’ you seem to explore these gaps – could you talk about that?
Pedro:
I was thinking about the impossibility of fully translating human touch into data. Machines can only create flawed representations, since in the process of translation some data is inevitably lost – machines can’t capture everything. Our screens do not even have enough pixels; it’s impossible to be aware of all the molecules making up someone’s body. I’m interested in the comfort of error, the impossibility of perfect interaction.
My performances explore “non-touch” feedback – like a microphone’s feedback loop. Two un-touching bodies, separated by a curtain, reaching for each other through technology.. It’s a metaphor for digital connection: veiled, partial, yet creating something new.
When Harry mentioned climate sensing, what came to mind was Timothy Morton and the idea of hyperobjects: we can only access a sliver of information in the mass of the world. As human beings we are not designed to understand phenomena in their fullness, because if we were we would probably explode! Even though we have the AI power, it’s very important to understand the failures at the edges of destruction.
Harry:
I love your use of a mechanical process like ‘feedback’ as a metaphor. Digital encoding should be seen as a process of mapping, but it can be confused with the territory. Encoding/mapping can be seen as the creation of a metaphor as much as a measurement. As artists, we use metaphors to create images to make meaning from things like clouds. In my work I use the metaphor of a ‘codec’: something being coded and decoded, but within that system, there’s a black box in the middle. I’m trying to put a human figure in this middle ground. For instance, I’m encoding an ecologist’s practice by creating a video game, which I’m showing to them and asking: is this what you do? Is this accurate? I’m trying to put a human conversation in the place of the machine to do the decoding.
'Personal Ecologies: The Community Gardener' by Harry Meadows
Ksenia
Zikai’s work, on the contrary, seems to emphasize efficiency and expanding the sensorium—new ways of interacting with the world—rather than the weirdness or mistakes. How do you see the role of error?
Zikai:
Mediated touch can make digital data more understandable to humans. However, our brain cannot directly understand the input from the sensors, so that can create mistakes in interpretation. In my AI music project, the machine-generated parts have that sense of detached ‘perfection’, while adding human touch – literally – introduces uniqueness. Each person could choose different “buttons” or textures, shaping their own experience. Everyone would feel something different.
Pedro:
It’s interesting that your machine is ‘assemblable’ and ‘disassemblamble’, since the majority of devices that we get are ‘black boxes’: you cannot modify them, this is what you get, full stop. It’s interesting how the device becomes a reflection of one’s identity.
Harry:
I think it’s important to acknowledge our own biases when we’re setting these systems up. Zikai’s work is very different from sonification systems, which turn data into music to understand it, or similarly – visualisation systems. There’s always a responsibility in defining what data creates what sound, because they are assigned meaning. So to me works like that are a great opportunity to reveal our own biases in how we think about the relationship between sensing and coding. These come from a cultural place, a set of ideas about how the world should be. It’s great when that’s explicit.
Zikai’s work, on the contrary, seems to emphasize efficiency and expanding the sensorium—new ways of interacting with the world—rather than the weirdness or mistakes. How do you see the role of error?
Zikai:
Mediated touch can make digital data more understandable to humans. However, our brain cannot directly understand the input from the sensors, so that can create mistakes in interpretation. In my AI music project, the machine-generated parts have that sense of detached ‘perfection’, while adding human touch – literally – introduces uniqueness. Each person could choose different “buttons” or textures, shaping their own experience. Everyone would feel something different.
Pedro:
It’s interesting that your machine is ‘assemblable’ and ‘disassemblamble’, since the majority of devices that we get are ‘black boxes’: you cannot modify them, this is what you get, full stop. It’s interesting how the device becomes a reflection of one’s identity.
Harry:
I think it’s important to acknowledge our own biases when we’re setting these systems up. Zikai’s work is very different from sonification systems, which turn data into music to understand it, or similarly – visualisation systems. There’s always a responsibility in defining what data creates what sound, because they are assigned meaning. So to me works like that are a great opportunity to reveal our own biases in how we think about the relationship between sensing and coding. These come from a cultural place, a set of ideas about how the world should be. It’s great when that’s explicit.
Ksenia
In discussions about sensing, there’s often a divide between embodied sensing and digitally mediated sensing, often referred to as a ‘digital divide’. What conceptual frameworks guide your work and help you navigate this ‘divide’?
Harry:
I draw from epistemics – critical analysis of how knowledge is created through laboratory processes – particularly, Hans-Jörg Rheinberger and Bruno Latour. They talk about systems of technology as designed technical objects that produce results that are often, more-or-less, unsurprising. When these objects are just invented though, they are a mess of processes: for instance, with photography it was initially a chaos of devices, chemicals, etc. Only later did cameras become reliable, stable, and impenetrable ‘black boxes’.
What interests me is that cycle, when new experiments get tightened into technical objects that stop giving us new knowledge and only give us expected knowledge, with these systems eventually breaking them open again. A part of this ‘breaking open’ is connecting with wider experimental systems. When it happens, not only does it reveal the internal workings of the device itself, but also the social and corporeal relations behind its existence. I guess one of the challenges here is to get this disrupted stage seen and recognised by a wider audience, since it’s so much easier to get people to use a nice consumable technical product rather than a disassembled experiment.
In discussions about sensing, there’s often a divide between embodied sensing and digitally mediated sensing, often referred to as a ‘digital divide’. What conceptual frameworks guide your work and help you navigate this ‘divide’?
Harry:
I draw from epistemics – critical analysis of how knowledge is created through laboratory processes – particularly, Hans-Jörg Rheinberger and Bruno Latour. They talk about systems of technology as designed technical objects that produce results that are often, more-or-less, unsurprising. When these objects are just invented though, they are a mess of processes: for instance, with photography it was initially a chaos of devices, chemicals, etc. Only later did cameras become reliable, stable, and impenetrable ‘black boxes’.
What interests me is that cycle, when new experiments get tightened into technical objects that stop giving us new knowledge and only give us expected knowledge, with these systems eventually breaking them open again. A part of this ‘breaking open’ is connecting with wider experimental systems. When it happens, not only does it reveal the internal workings of the device itself, but also the social and corporeal relations behind its existence. I guess one of the challenges here is to get this disrupted stage seen and recognised by a wider audience, since it’s so much easier to get people to use a nice consumable technical product rather than a disassembled experiment.
'Personal Ecologies: The Community Gardener' by Harry Meadows
Screenshots from the game
In this regard, could you talk a bit about the ways you involve the audience directly in your performances?
Zikai:
For Touch Intelligence, my audience are not just users but co-creators. Even during the design phase, their direct participation was essential—they experienced buttons with different textures, felt the emotional changes triggered by touch, and evaluated the relationships between musical styles and rhythms. Through their feedback, the AI could learn the mapping between “touch—emotion—sound,” continuously refining its outputs to better align with human experience. Touch Intelligence was showcased in a small-scale exhibition at the London Design Festival, where many visitors actively engaged with the installation. Anyone can perform using ‘Touch Intelligence’ if they want to. They choose a button, press it, and generate music. What I did notice though is that people do not always like the AI-generated result – partly because everyone’s interpretation of tactile sensations would be different, and the AI-generated pieces won’t necessarily match these.
Zikai:
For Touch Intelligence, my audience are not just users but co-creators. Even during the design phase, their direct participation was essential—they experienced buttons with different textures, felt the emotional changes triggered by touch, and evaluated the relationships between musical styles and rhythms. Through their feedback, the AI could learn the mapping between “touch—emotion—sound,” continuously refining its outputs to better align with human experience. Touch Intelligence was showcased in a small-scale exhibition at the London Design Festival, where many visitors actively engaged with the installation. Anyone can perform using ‘Touch Intelligence’ if they want to. They choose a button, press it, and generate music. What I did notice though is that people do not always like the AI-generated result – partly because everyone’s interpretation of tactile sensations would be different, and the AI-generated pieces won’t necessarily match these.
The fact that the movements that I’m performing have been done by these historical people as well creates a relation with history, when a body becomes a part of it.
Pedro:
I’ve done that a couple of times, but lately I’ve been more interested in scenarios when the audience accompanies me, not necessarily having an active role. For example, in Latent Space / Body: limbs [2023] I’ve tried to approach performance as creating a place of vulnerability—where the audience can witness my exploration. It is all about the fragility of the body and the body as inherently containing histories, the past, and the struggles of the past. In the ‘Body: limbs’ I was using MYO sensors that track muscle movement data, which, interestingly enough, were originally developed for corporate use, but have eventually only been used by artists. Right when I started using these sensors I started practicing Voguing, a dance practice within the Ballroom Scene coming from the 70s developed to celebrate POC trans and gay bodies. It was a chance encounter, but these two interests started intersecting and affecting each other. I noticed how the gestural vocabulary of Voguing influences me, and how its histories – the histories of people who were brutally killed and jailed – become embodied in my own body. The fact that the movements that I’m performing have been done by these historical people as well creates a relation with history, when a body becomes a part of it. Of course I was not comparing my own struggles with these historical struggles, but this practice definitely created a sense of connection and radical appreciation for these movements. In this sense, the digital systems allowed me to more fully and radically appreciate the physical body: feeling all the muscles, bones, tensions, movements, and the history of such bodily expression.
Also, of course it is an attempt to foreground how much we’re bound by European history, and to celebrate the histories of people of non-European origin. For me it was about being outside of the usual knowledge-making bubble. For instance, what if any language other than English was used as a basis for coding, for instance? What would the syntax look like?
I’ve done that a couple of times, but lately I’ve been more interested in scenarios when the audience accompanies me, not necessarily having an active role. For example, in Latent Space / Body: limbs [2023] I’ve tried to approach performance as creating a place of vulnerability—where the audience can witness my exploration. It is all about the fragility of the body and the body as inherently containing histories, the past, and the struggles of the past. In the ‘Body: limbs’ I was using MYO sensors that track muscle movement data, which, interestingly enough, were originally developed for corporate use, but have eventually only been used by artists. Right when I started using these sensors I started practicing Voguing, a dance practice within the Ballroom Scene coming from the 70s developed to celebrate POC trans and gay bodies. It was a chance encounter, but these two interests started intersecting and affecting each other. I noticed how the gestural vocabulary of Voguing influences me, and how its histories – the histories of people who were brutally killed and jailed – become embodied in my own body. The fact that the movements that I’m performing have been done by these historical people as well creates a relation with history, when a body becomes a part of it. Of course I was not comparing my own struggles with these historical struggles, but this practice definitely created a sense of connection and radical appreciation for these movements. In this sense, the digital systems allowed me to more fully and radically appreciate the physical body: feeling all the muscles, bones, tensions, movements, and the history of such bodily expression.
Also, of course it is an attempt to foreground how much we’re bound by European history, and to celebrate the histories of people of non-European origin. For me it was about being outside of the usual knowledge-making bubble. For instance, what if any language other than English was used as a basis for coding, for instance? What would the syntax look like?
'Personal Ecologies: The Community Gardener' by Harry Meadows
We’ve got to have conversations about the meaning of encoding and decoding, conversations that go beyond the technical.
Harry:
You mentioned ‘vulnerability’, and I think it is very important when working with others: for instance, when I was working with that community gardener, I learned a lot about not being in a hierarchical position of explaining somebody’s practice. When I first showed the work in an exhibition, the game I made was my own interpretation of her practice, and she was not given a final say in it. So, after the opening, I rethought this position and what I eventually made the main event in the second part of the show was her playing this game and commenting on it. I recorded the session, her screen and her commentary on our co-creation. That way, she had the last word—she could critique what I’d done and put distance between what I'd done and what her own experience of it was. Her voice was eventually the most present voice in the show, rather than mine, which hopefully shifted the dynamic.
It was the result of a year-long relationship, it wasn’t a case of her saying ‘oh, how dare your video game represent me in this way’, it was more like, ‘your account has omitted some important things that I’d like to tell you about’. So my key takeaway from that was that we’ve got to have conversations about the meaning of encoding and decoding, conversations that go beyond the technical.
You mentioned ‘vulnerability’, and I think it is very important when working with others: for instance, when I was working with that community gardener, I learned a lot about not being in a hierarchical position of explaining somebody’s practice. When I first showed the work in an exhibition, the game I made was my own interpretation of her practice, and she was not given a final say in it. So, after the opening, I rethought this position and what I eventually made the main event in the second part of the show was her playing this game and commenting on it. I recorded the session, her screen and her commentary on our co-creation. That way, she had the last word—she could critique what I’d done and put distance between what I'd done and what her own experience of it was. Her voice was eventually the most present voice in the show, rather than mine, which hopefully shifted the dynamic.
It was the result of a year-long relationship, it wasn’t a case of her saying ‘oh, how dare your video game represent me in this way’, it was more like, ‘your account has omitted some important things that I’d like to tell you about’. So my key takeaway from that was that we’ve got to have conversations about the meaning of encoding and decoding, conversations that go beyond the technical.