EXPANDED SENSORIUM
Discussion participants:
Harry Meadows, artist and lecturer
Pedro Latas, multimedia artist, composer, and sound artist
Zikai Peng, industrial designer
Harry Meadows, artist and lecturer
Pedro Latas, multimedia artist, composer, and sound artist
Zikai Peng, industrial designer
Cover image: 'Touch Intelligence' by Zikai Peng
Interviewer: Ksenia Kopalova
How does technology reshape embodied experience, our relations with our own bodies and with each other? Machines can heighten sensory feedback, but there is a human responsibility in translation of senses into digital outputs. How can we address it, and how can the mistakes in this process become sites of productive uncertainty? Touch becomes a way of thinking — an active embodied exchange between human and non-human sensory systems.
What does it mean to sense through technology rather than despite it? And how might digital systems open space for new forms of intimacy, awareness, and connection across the human and more-than-human?
What does it mean to sense through technology rather than despite it? And how might digital systems open space for new forms of intimacy, awareness, and connection across the human and more-than-human?
'Touch Intelligence' by Zikai Peng
What comes to mind when you think of “expanded sensorium” and “digital bodies”?
Some sensors provide information much richer than what our skin can provide, [creating] a more complex and diverse feedback system
Pedro:
When I think of digital systems, I also inevitably have to think about all the manners of translation of information that you’re obliged to engage with – like when you’re on a Zoom call, the camera picks up the light, transforms it into binary data and electrical impulses transmitted by fibre cables under the ocean. And then there’s a reverse process: the light becomes images on the screen – images of a person you’re talking to almost in real time. Our ‘meat bodies’ undergo so many processes of digital transformation here. Part of our identities are indeed embodied in these digital forms. This of course also brings up the question of how much is lost in this transformation - perfect translation is impossible. That’s fascinating to me – all these layers of translation, as well as the fact that we are translatable through digital media in principle.
Zikai:
The first thing I thought about in relation to ‘digital bodies’ as such was the difference between the digital and real. With physical touch, the signals come from our skin, whereas with digital touch it’s not the case: some sensors provide information much richer than what our skin can provide – with all the information on humidity, temperature, pressure, etc. Thus digital touch could give humans new ways to understand the physical world – a complex and diverse feedback system.
When I think of digital systems, I also inevitably have to think about all the manners of translation of information that you’re obliged to engage with – like when you’re on a Zoom call, the camera picks up the light, transforms it into binary data and electrical impulses transmitted by fibre cables under the ocean. And then there’s a reverse process: the light becomes images on the screen – images of a person you’re talking to almost in real time. Our ‘meat bodies’ undergo so many processes of digital transformation here. Part of our identities are indeed embodied in these digital forms. This of course also brings up the question of how much is lost in this transformation - perfect translation is impossible. That’s fascinating to me – all these layers of translation, as well as the fact that we are translatable through digital media in principle.
Zikai:
The first thing I thought about in relation to ‘digital bodies’ as such was the difference between the digital and real. With physical touch, the signals come from our skin, whereas with digital touch it’s not the case: some sensors provide information much richer than what our skin can provide – with all the information on humidity, temperature, pressure, etc. Thus digital touch could give humans new ways to understand the physical world – a complex and diverse feedback system.
How does digital interaction influence your physical experience of the world? Do you feel your real-life interactions are shaped by digital processes?
I tend not to separate physical and digital – they enhance each other. They are designed together and made to emulate each other
Pedro:
Digital systems are made as a reflection of human experience – it’s a mirror of how we perceive the world. This technological mirror gets closer to the real world, but it also feeds back into us: it’s a weird feedback loop. Think for example how programming languages are designed to have a clear and correct syntax structure as if it were a spoken language: ‘If this then that’. I tend not to separate physical and digital – they enhance each other. They are designed together and made to emulate each other.
Zikai:
In ‘Touch Intelligence’, pressing differently textured buttons triggers AI-generation of music that expresses different emotional states. In this sense, the physical touch and the digital have the same aim – conveying emotion. Touch can change emotion: a rough button can change the feeling to ‘tension’, for instance, while a smooth one can evoke the sensation of calmness and relaxation, and the generated music can express it back – that’s how the physical and the digital are linked in my project.
Digital systems are made as a reflection of human experience – it’s a mirror of how we perceive the world. This technological mirror gets closer to the real world, but it also feeds back into us: it’s a weird feedback loop. Think for example how programming languages are designed to have a clear and correct syntax structure as if it were a spoken language: ‘If this then that’. I tend not to separate physical and digital – they enhance each other. They are designed together and made to emulate each other.
Zikai:
In ‘Touch Intelligence’, pressing differently textured buttons triggers AI-generation of music that expresses different emotional states. In this sense, the physical touch and the digital have the same aim – conveying emotion. Touch can change emotion: a rough button can change the feeling to ‘tension’, for instance, while a smooth one can evoke the sensation of calmness and relaxation, and the generated music can express it back – that’s how the physical and the digital are linked in my project.
Harry:
Digital touch is mediated by machines. We are in a position where we have to rely on these extensions of our senses to make sense of the world beyond us. A lot of my work has been around climate sensing and its apparatus. This mediation has inherent bias, since it’s selection and omission of information needs to be focussed for a particular product . Instead, my research is looking into a multiplicity of ways of sensing. Apart from standard modes of sensing, I’m looking at amateurs and enthusiasts who set up their own climate sensing systems that are intuitive or experimental and offer alternative ecological maps.
For instance, ‘Personal Ecologies: The Community Gardener’ is a work I co-created with a community gardener who carefully raises tortoises in climate-controlled fridges, with a complex system of lamps and heat control systems for their hibernation. This is outside of standard practice of scientific sensing because she combines standard climate sensors with wider experimental systems of plants, animals and rather radical social organisation. I’m interested in how we as artists can celebrate sensing outside of its standard modes. How do the laboratory modes of sensing combine with intuition and social engagement? These combinations reveal a multiplicity of environmental sensing, and complexity in the relationship between machines, plants, animals and geology as sensors.
There are two examples of ecological mapping I’m particularly interested in: one is eDNA, which helps to identify all the species that might live in, let’s say, a lake, or a forest – through relatively tiny little samples of water or soil.
Another example is the Food and Agriculture Organization of the United Nations, who have developed methods of encoding forestation/deforestation in parts of the world that export food. Their process acknowledges the relationship between the digital map and ground truths of the people who live and work in this ‘digital map’.
So I’m interested in how art can explore ground truths and the social reality of mechanical sensing.
Digital touch is mediated by machines. We are in a position where we have to rely on these extensions of our senses to make sense of the world beyond us. A lot of my work has been around climate sensing and its apparatus. This mediation has inherent bias, since it’s selection and omission of information needs to be focussed for a particular product . Instead, my research is looking into a multiplicity of ways of sensing. Apart from standard modes of sensing, I’m looking at amateurs and enthusiasts who set up their own climate sensing systems that are intuitive or experimental and offer alternative ecological maps.
For instance, ‘Personal Ecologies: The Community Gardener’ is a work I co-created with a community gardener who carefully raises tortoises in climate-controlled fridges, with a complex system of lamps and heat control systems for their hibernation. This is outside of standard practice of scientific sensing because she combines standard climate sensors with wider experimental systems of plants, animals and rather radical social organisation. I’m interested in how we as artists can celebrate sensing outside of its standard modes. How do the laboratory modes of sensing combine with intuition and social engagement? These combinations reveal a multiplicity of environmental sensing, and complexity in the relationship between machines, plants, animals and geology as sensors.
There are two examples of ecological mapping I’m particularly interested in: one is eDNA, which helps to identify all the species that might live in, let’s say, a lake, or a forest – through relatively tiny little samples of water or soil.
Another example is the Food and Agriculture Organization of the United Nations, who have developed methods of encoding forestation/deforestation in parts of the world that export food. Their process acknowledges the relationship between the digital map and ground truths of the people who live and work in this ‘digital map’.
So I’m interested in how art can explore ground truths and the social reality of mechanical sensing.
'Personal Ecologies:
The Community Gardener'
The Community Gardener'
Research process behind the project by Harry Meadows
This makes me think about the inevitable mistakes and gaps that happen in these translations from mechanical sensing to the lived experience and vice versa. Pedro, in your project ‘Can we feel touch when we're made of light’ you seem to explore these gaps – could you talk about it?
Digital encoding should be seen as a process of mapping, but it can be confused with the territory. Encoding/mapping can be seen as the creation of a metaphor as much as a measurement
Pedro:
I was thinking about the impossibility of fully translating human touch into data. Machines can only create flawed representations, since in the process of translation some data is inevitably lost – machines can’t capture everything. Our screens do not even have enough pixels; it’s impossible to capture all the molecules making up someone’s body. I’m interested in the comfort of error, the impossibility of perfect interaction.
My performances explored “non-touch” feedback – in this piece, metaphorically represented by an audio feedback loop. Two un-touching bodies, separated by a curtain, reaching for each other through technology. It’s a metaphor for digital connection: veiled, partial, yet creating something new and exciting, redefining touch and compassion.
When Harry mentioned climate sensing, what came to mind was Timothy Morton and the idea of hyperobjects: we can only access a sliver of information in the mass of the world. As human beings we are not designed to understand phenomena in their fullness, because if we were we would probably explode! We might be able to access these complex systems through advanced data processing technology like AI, but I would argue it's far more interesting to be comfortable with (and work through) the inability to get a hold of all these hyperobjects that surround us. It's a special and beautiful kind of humility.
Harry:
I love your use of a mechanical process like ‘feedback’ as a metaphor. Digital encoding should be seen as a process of mapping, but it can be confused with the territory. Encoding/mapping can be seen as the creation of a metaphor as much as a measurement. As artists, we use metaphors to create images to make meaning from things like clouds. In my work I use the metaphor of a ‘codec’: something being coded and decoded, but within that system, there’s a black box in the middle. I’m trying to put a human figure in this middle ground. For instance, I’m encoding an ecologist’s practice by creating a video game, which I’m showing to them and asking: is this what you do? Is this accurate? I’m trying to put a human conversation in the place of the machine to do the decoding.
I was thinking about the impossibility of fully translating human touch into data. Machines can only create flawed representations, since in the process of translation some data is inevitably lost – machines can’t capture everything. Our screens do not even have enough pixels; it’s impossible to capture all the molecules making up someone’s body. I’m interested in the comfort of error, the impossibility of perfect interaction.
My performances explored “non-touch” feedback – in this piece, metaphorically represented by an audio feedback loop. Two un-touching bodies, separated by a curtain, reaching for each other through technology. It’s a metaphor for digital connection: veiled, partial, yet creating something new and exciting, redefining touch and compassion.
When Harry mentioned climate sensing, what came to mind was Timothy Morton and the idea of hyperobjects: we can only access a sliver of information in the mass of the world. As human beings we are not designed to understand phenomena in their fullness, because if we were we would probably explode! We might be able to access these complex systems through advanced data processing technology like AI, but I would argue it's far more interesting to be comfortable with (and work through) the inability to get a hold of all these hyperobjects that surround us. It's a special and beautiful kind of humility.
Harry:
I love your use of a mechanical process like ‘feedback’ as a metaphor. Digital encoding should be seen as a process of mapping, but it can be confused with the territory. Encoding/mapping can be seen as the creation of a metaphor as much as a measurement. As artists, we use metaphors to create images to make meaning from things like clouds. In my work I use the metaphor of a ‘codec’: something being coded and decoded, but within that system, there’s a black box in the middle. I’m trying to put a human figure in this middle ground. For instance, I’m encoding an ecologist’s practice by creating a video game, which I’m showing to them and asking: is this what you do? Is this accurate? I’m trying to put a human conversation in the place of the machine to do the decoding.
Can we feel touch when we're made of light [2021] by Pedro Latas,
for 2 performers and improviser or pre-recorded electronics
for 2 performers and improviser or pre-recorded electronics
Zikai’s work, on the contrary, seems to emphasize efficiency and expanding the sensorium—new ways of interacting with the world—rather than the weirdness or mistakes. How do you see the role of error?
Zikai:
Mediated touch can make digital data more understandable to humans. However, our brain cannot directly understand the input from the sensors, so that can create mistakes in interpretation. In my AI music project, the machine-generated parts have that sense of detached ‘perfection’, while adding human touch – literally – introduces uniqueness. Each person could choose different “buttons” or textures, shaping their own experience. Everyone would feel something different.
Pedro:
It’s interesting that your machine is ‘assemblable’ and ‘disassemblamble’, since the majority of devices that we get are ‘black boxes’: you cannot modify them, this is what you get, full stop. It’s interesting how the device becomes a reflection of one’s identity by how one will physically change it.
Harry:
I think it’s important to acknowledge our own biases when we’re setting these systems up. Zikai’s work is very different from sonification systems, which turn data into music to understand it, or similarly – visualisation systems. There’s always a responsibility in defining what data creates what sound, because they are assigned meaning. So to me works like that are a great opportunity to reveal our own biases in how we think about the relationship between sensing and coding. These come from a cultural place, a set of ideas about how the world should be. It’s great when that’s explicit.
Mediated touch can make digital data more understandable to humans. However, our brain cannot directly understand the input from the sensors, so that can create mistakes in interpretation. In my AI music project, the machine-generated parts have that sense of detached ‘perfection’, while adding human touch – literally – introduces uniqueness. Each person could choose different “buttons” or textures, shaping their own experience. Everyone would feel something different.
Pedro:
It’s interesting that your machine is ‘assemblable’ and ‘disassemblamble’, since the majority of devices that we get are ‘black boxes’: you cannot modify them, this is what you get, full stop. It’s interesting how the device becomes a reflection of one’s identity by how one will physically change it.
Harry:
I think it’s important to acknowledge our own biases when we’re setting these systems up. Zikai’s work is very different from sonification systems, which turn data into music to understand it, or similarly – visualisation systems. There’s always a responsibility in defining what data creates what sound, because they are assigned meaning. So to me works like that are a great opportunity to reveal our own biases in how we think about the relationship between sensing and coding. These come from a cultural place, a set of ideas about how the world should be. It’s great when that’s explicit.
In discussions about sensing, there’s often a divide between embodied sensing and digitally mediated sensing, often referred to as a ‘digital divide’. What conceptual frameworks guide your work and help you navigate this ‘divide’?
Harry:
I draw from epistemics – critical analysis of how knowledge is created through laboratory processes – particularly, Hans-Jörg Rheinberger and Bruno Latour. They talk about systems of technology as designed technical objects that produce results that are often, more-or-less, unsurprising. When these objects are just invented though, they are a mess of processes: for instance, with photography it was initially a chaos of devices, chemicals, etc. Only later did cameras become reliable, stable, and impenetrable ‘black boxes’.
What interests me is that cycle, when new experiments get tightened into technical objects that stop giving us new knowledge and only give us expected knowledge, with these systems eventually breaking them open again. A part of this ‘breaking open’ is connecting with wider experimental systems. When it happens, not only does it reveal the internal workings of the device itself, but also the social and corporeal relations behind its existence. I guess one of the challenges here is to get this disrupted stage seen and recognised by a wider audience, since it’s so much easier to get people to use a nice consumable technical product rather than a disassembled experiment.
I draw from epistemics – critical analysis of how knowledge is created through laboratory processes – particularly, Hans-Jörg Rheinberger and Bruno Latour. They talk about systems of technology as designed technical objects that produce results that are often, more-or-less, unsurprising. When these objects are just invented though, they are a mess of processes: for instance, with photography it was initially a chaos of devices, chemicals, etc. Only later did cameras become reliable, stable, and impenetrable ‘black boxes’.
What interests me is that cycle, when new experiments get tightened into technical objects that stop giving us new knowledge and only give us expected knowledge, with these systems eventually breaking them open again. A part of this ‘breaking open’ is connecting with wider experimental systems. When it happens, not only does it reveal the internal workings of the device itself, but also the social and corporeal relations behind its existence. I guess one of the challenges here is to get this disrupted stage seen and recognised by a wider audience, since it’s so much easier to get people to use a nice consumable technical product rather than a disassembled experiment.
'Personal Ecologies: The Community Gardener'
Screenshots from the game by Harry Meadows
Screenshots from the game by Harry Meadows
In discussions about sensing, there’s often a divide between embodied sensing and digitally mediated sensing, often referred to as a ‘digital divide’. What conceptual frameworks guide your work and help you navigate this ‘divide’?
Zikai:
For ‘Touch Intelligence’, my audience are not just users but co-creators. Even during the design phase, their direct participation was essential — they experienced buttons with different textures, felt the emotional changes triggered by touch, and evaluated the relationships between musical styles and rhythms. Through their feedback, the AI could learn the mapping between “touch—emotion—sound,” continuously refining its outputs to better align with human experience.
‘Touch Intelligence’ was showcased in a small-scale exhibition at the London Design Festival, where many visitors actively engaged with the installation. Anyone can perform using ‘Touch Intelligence’ if they want to. They choose a button, press it, and generate music. What I did notice though is that people do not always like the AI-generated result – partly because everyone’s interpretation of tactile sensations would be different, and the AI-generated pieces won’t necessarily match these.
For ‘Touch Intelligence’, my audience are not just users but co-creators. Even during the design phase, their direct participation was essential — they experienced buttons with different textures, felt the emotional changes triggered by touch, and evaluated the relationships between musical styles and rhythms. Through their feedback, the AI could learn the mapping between “touch—emotion—sound,” continuously refining its outputs to better align with human experience.
‘Touch Intelligence’ was showcased in a small-scale exhibition at the London Design Festival, where many visitors actively engaged with the installation. Anyone can perform using ‘Touch Intelligence’ if they want to. They choose a button, press it, and generate music. What I did notice though is that people do not always like the AI-generated result – partly because everyone’s interpretation of tactile sensations would be different, and the AI-generated pieces won’t necessarily match these.
Latent Space / Body: limbs [2023] by Pedro Latas,
solo live-electronics performance with MYO sensors and AI
solo live-electronics performance with MYO sensors and AI
The digital systems allowed me to engage fully and radically appreciate the physical body: feeling all the muscles, bones, tensions, movements, and the history of such bodily expression
Pedro:
I’ve done that a couple of times, but lately I’ve been more interested in scenarios when the audience accompanies me, not necessarily having an active role. For example, in Latent Space / Body: limbs [2023] I’ve tried to approach performance as creating a place of vulnerability—where the audience can witness and accompany me in my exploration. It is all about the fragility of the body and the body as inherently containing histories, the past, and the struggles of the past.
In the ‘Latent Space / Body: limbs’ I was using MYO sensors that track muscle movement data, which, interestingly enough, were originally developed for corporate use, but have eventually only been used by artists. Right when I started using these sensors I started practicing Voguing, a performance practice within the Ballroom Scene coming from the 70s in New York especially created to celebrate POC trans and queer bodies. It was a chance encounter, but these two interests started intersecting and affecting each other. I noticed how the gestural vocabulary of Voguing influenced me, and how its histories – the histories of those POC bodies that, at the cost of their physical wellbeing, paved way for the contemporary struggle for queer liberation and body autonomy – become embodied in my own body. These are not only "my" movements, but carry with them a historical context. Of course I was not comparing my own struggles with these historical struggles, but this practice definitely created a sense of connection and radical appreciation for those people - the movements (or poses) being the vehicle.
In this sense, the digital systems allowed me to engage fully and radically appreciate the physical body: feeling all the muscles, bones, tensions, movements, and the history of such bodily expression.
I’ve done that a couple of times, but lately I’ve been more interested in scenarios when the audience accompanies me, not necessarily having an active role. For example, in Latent Space / Body: limbs [2023] I’ve tried to approach performance as creating a place of vulnerability—where the audience can witness and accompany me in my exploration. It is all about the fragility of the body and the body as inherently containing histories, the past, and the struggles of the past.
In the ‘Latent Space / Body: limbs’ I was using MYO sensors that track muscle movement data, which, interestingly enough, were originally developed for corporate use, but have eventually only been used by artists. Right when I started using these sensors I started practicing Voguing, a performance practice within the Ballroom Scene coming from the 70s in New York especially created to celebrate POC trans and queer bodies. It was a chance encounter, but these two interests started intersecting and affecting each other. I noticed how the gestural vocabulary of Voguing influenced me, and how its histories – the histories of those POC bodies that, at the cost of their physical wellbeing, paved way for the contemporary struggle for queer liberation and body autonomy – become embodied in my own body. These are not only "my" movements, but carry with them a historical context. Of course I was not comparing my own struggles with these historical struggles, but this practice definitely created a sense of connection and radical appreciation for those people - the movements (or poses) being the vehicle.
In this sense, the digital systems allowed me to engage fully and radically appreciate the physical body: feeling all the muscles, bones, tensions, movements, and the history of such bodily expression.
'Personal Ecologies: The Community Gardener' by Harry Meadows
We’ve got to have conversations about the meaning of encoding and decoding, conversations that go beyond the technical
Harry:
You mentioned ‘vulnerability’, and I think it is very important when working with others: for instance, when I was working with that community gardener, I learned a lot about not being in a hierarchical position of explaining somebody’s practice. When I first showed the work in an exhibition, the game I made was my own interpretation of her practice, and she was not given a final say in it. So, after the opening, I rethought this position and what I eventually made the main event in the second part of the show was her playing this game and commenting on it. I recorded the session, her screen and her commentary on our co-creation. That way, she had the last word—she could critique what I’d done and put distance between what I'd done and what her own experience of it was. Her voice was eventually the most present voice in the show, rather than mine, which hopefully shifted the dynamic.
It was the result of a year-long relationship, it wasn’t a case of her saying ‘oh, how dare your video game represent me in this way’, it was more like, ‘your account has omitted some important things that I’d like to tell you about’. So my key takeaway from that was that we’ve got to have conversations about the meaning of encoding and decoding, conversations that go beyond the technical.
You mentioned ‘vulnerability’, and I think it is very important when working with others: for instance, when I was working with that community gardener, I learned a lot about not being in a hierarchical position of explaining somebody’s practice. When I first showed the work in an exhibition, the game I made was my own interpretation of her practice, and she was not given a final say in it. So, after the opening, I rethought this position and what I eventually made the main event in the second part of the show was her playing this game and commenting on it. I recorded the session, her screen and her commentary on our co-creation. That way, she had the last word—she could critique what I’d done and put distance between what I'd done and what her own experience of it was. Her voice was eventually the most present voice in the show, rather than mine, which hopefully shifted the dynamic.
It was the result of a year-long relationship, it wasn’t a case of her saying ‘oh, how dare your video game represent me in this way’, it was more like, ‘your account has omitted some important things that I’d like to tell you about’. So my key takeaway from that was that we’ve got to have conversations about the meaning of encoding and decoding, conversations that go beyond the technical.