Interaction in digital art

Interaction may use... any kind of means. Here a demonstration of Ass-ball at Siggraph Asia (Singapore, 2008).

Last revised Jan. 11, 2018. Reurn to Major concepts. See beings behavior, attention, time.
- Our posts and references.  

Recent references:
- In 2017, Jonas Lund goes as far as possible in interaction, notably with his Critical Mass piece.
- In 2014,see
- our post about the Art session in Laval Virtual, chaired by Alain Lioret, and a second look about it.
- our post about Le cinéma interactif, by Marie-Laure Cazin.

1. Introduction. 2. Developments. 3. Artists and persons. 4. References.

1. Introduction

[Wardrip-Frouin] defines interaction as "changement in the state of the work, coming from outside the work". This "outside" may mean the workd in general (for instance weather events which were perceived by the Liège tower of Nicolas Schoeffer), but more frequently the intervention of human actors.

Our feeling is that only games and transmedia (broadly speaking) are really looking for engaging and compelling interaction. In the other forms of digital art, interaction is always minimal, frequently reduced to a simple on/off switch.. See comments about several expositions in spring 2013: Bruxelles and in Paris, Le Parc.

Interaction is considered one of the major affordances of digital technology to arts. It appears that, eventually, interaction is generally rather limited, for two reasons:
- it demands appropriate hardware, which may be costly and difficult to integrate to the physical system as well as to the software; besides, the conservation of interactive works on the long run is sensibly more difficult than passive ones : interfaces are fragile, subject to inapproriate (if not malevolent) actions; and interactive software is more rapidly obsolescent than static files of sound or images;
- few artists do sincerely desire it, in spite of grand statements; they easily consider interaction as belonging to games much more that to art.

In our report about Le Parc exhibition in Paris (Spring 2013), we noted the contrast between his 1968 strong mottos and the reality of his works (and art in general) in our days.

Two kinds of arts have successfully given a prime importance to interaction: games and transmedia.

Images from Dynamic intermediate models for audiographic synthesis, by Vincent Goudard, Vincent, Hugues Genevois, Boris Doval and Emilien Ghomi.

2. Developments

2.1. Interaction and the four poles of creativity process

Instead, or in complement of dealing with assets as parts of S, of the work itself,  we can see them as communication abilities with our four poles of the creative process: matter, spirit, public and author.

Matter

With “matter”, interaction is the “incarnation” of the work into the natural world, and a capacity to perceive the promising, constraining and sometimes threatening… or simply artistically interesging features of places and times. You can find that in some kind of robotic arts, or even in network applications like Google Earth, if you take the formatting and presentation in the browser as a (not so) rough form or art. Great hopes, and limited results have been tied to connection with weather and general time conditions, for instance the Cybernetic tower of Pierre Schoeffer in Liège (Belgium). It is also a facet of the Moben's multimedia work Mechanics of emotions based on web data transformation into 3D objects.

Spirit

With “spirit”, interaction has a meaning if the sponsor of the work (or the artist himself), is put in position to see the realization progress and can give more details, change some points, of his/her original “idea”. That applies, in particular, to architectural projects or to events preparation.

The artist

With the artist, interactivity has two different kind of implementations and operation :
- in plastical arts (painting and sculpture) interaction may be taken simply as part of the  Wysiwyg (What you see is what you get) design interface, a way to progress more effectively in the creation process; but the interaction is not properly part of the work;
- in performance arts as VJing for instance, and so more in collective arts (music, choreography), the work may interact with some or all of the performers and impact strongly on their virtuosity or expressive creativity.

The public

But of course, it is with the public that interactivity in art has been mostly sought for. Its evidently a constitutive part of games, but until now, games stay out of the proper Art gamut. Public reactions  have always been important in performance arts, and every actor will insist on the importance of audience reaction to the quality of the play, would it be only the quality of its silence.

In the sixties interactivity could even be desired as a form of more “democratic art”, breaking the wall between the dictatorship of authors and actors on one side and the submissivity of the lay spectators, heretofore called to be “spectactors”. That can apply not only to on site performances  but on TV and so more on blogs and social networks. The results seem more convincing in technical discussion groups than in collective art.

Did the digital communication add really much to the grand traditional meetings (religious and sports events in particular) where the public is indeed called to take his part, and possibly a large part, but along well marked tracks and traditions?  A lot of interactive works of Art, shown in digital art events, do not give to the public a real free space to take its part, and look more like a way of showing the creativity of the artist than to imply actually the public into the creation.

Anyway, the desire to “take part” is not shared by everybody nor a permanent attitude. Art is most of the time taken as an entertainment, be it a well deserved rest before the TV screen after a days work, or a collective pleasure to be shared with friends along with social behaviours before and after the show, like hugging, dining … “and more if affinity”. Did not Artists have excessive expectations from this side of the creation ?

Things are perhaps changing rapidly now. Some years ago, high level interfaces were simplistic (joysticks at best) or  heavy and costly (tables, large screens, markers for motion capture, plus high level professional software to make proper use of the signals).

Since the 2010's , WII, Kinect, markerless capture, and probably a lot more to come, have entered everybody’s homes, along with the appropriate software for games, anyway using powerful machines). On the other way round, sophisticated interfaces like immersive goggles and caves or haptic devices can be dispensed with, since our mind is powerfully gifted to take us fully into action. Just look at kids involved in a game around the home computer!  

As for global audiences and, so to say, collective or cooperative playability, its importance and development are stressed on and taken into account by innovative curators (see the book of Graham and Cook [Graham 2010]) as well as theater professionals (see the collection of texts gathered by Garbagnati and Morelli [Garbagnati 2006]).  

What has Art to do now is to climb to another level.. To take a step above the basic instincts of games: kill or die, win or lose, step on the top scale of the podium. Art requires less animality and more soul.  How shall we achieve that?

2.2. Arts and games

A game is interactive by construction. But artists will consider that a game is not a work of art, since it includes a motivation to "win", which is foreign to the contemplative nature or art proper.

Nevertheless, we can quote games where the artistic dimension is important, be it for image quality, story structure or even music and sounds.

See games.

2.3. The transmedia

See transmedia.
The second screen and transmedia, since the beginning of the 21th century, gives a new dimension to interaction.

Here also, since transmedia properly speaking is based upon a television emission or series, the World or Art tends to consider that to popular to be called art.

But, during the 2010 decade, the transmedia hype became obsolete. Perhaps because, properly taken as a project, it requires a too hea y investment.

2.4. Levels interactions. Ends and means

2.4.1. Ends

Some artists say they expect an attitude of contemplation by the audience. In this case, interaction becomes a psychological process between the spectator and the representations he builds from the work.

Just adapt to public presence, attitudes, or engage into participation.
The minimal position, sometimes shamelessly called interaction, is a binary detection of a spectator, be it by their action on a switch, or by some sensor like a photoelectric cell.
There is no maximal position. You can always dig deeper into the spectators profiling and reactions analyzing.

Artists also can interact with the work:
- during the creation process.
- during a performance (DJ, VJ, real time live).

2.4.2. Means

They are not different of HCI (human computer interaction) devices. Since the end of the 20th century, powerful devices are offered to the public at an affordable price, and with software interfaces (API) which are not out of reach for even an amateur programmer.

Interaction is bi-directional. The work must have "expression" means, and the spectator's actions or attitudes must be perceived.

What can the work (and the artist in the background) perceive of the public reactions. Some first examples
- attention time: may be a first look, then a second, then live with. That is an information rather easy to gather, manually with a chronometer, or automatically with sensors
- positions, presence of faces, possibly of expression (smiles), with now widely available cameras and vision software, and of course WII and kinect.

With higher ambitions, these observations would have to be correlated with the work's behavior.

2.4.3. Augmented tactility.

See Digitalarti. "Escales Tactiles Performance chorégraphique et tactilité augmentée", on aug. 5 2011, at Spatial station Mir, Cité de l'espace (Nuit des Etoiles), Toulouse. Choreography : Anne Holst & Jean-Marc Matos. Interactive scenography, sound and lights, Grégory Lasserre & Anaïs met den Ancxt (Scenocosme), dancers-performers: Aude Miyagi & Julien Lecuziat. This performance explores the body tactility and the body-technology convergence.

Augmented skin electricity and communicating clothes affords the contact sensations between two dancers to be translated into sound and light, and so perceived by the audience. The different touch qualities generate and modulate sound and light matters. Expressive and bodily conventions are reconstituted and our sense amplified, then our imaginations, relations to the world and communication moods.

2.4.4. Reciprocal learning and co-creation.

To become really interesting, interactivity would call for a recirprocal learning: the spectator learns how to play with the work, but symmetrically, the work learns how to satisfy the needs and desires of the spectator. This kind of process is foreign to art, even if, in the 2000's, its has becomme common in marketing and on line services, particularly Google.

Somehow, that implies that the work, when delivered, is not "finisthed". It will continue to grow with its interactions, acquiring knowledge of more and more spectators. And, possibly, for instance with some kind of retropropagation (neural networks), and why not with some kind of automatic development, keep growing independently of its author. Artists don't like that !

This process of co-creation could be progressive, step by step. The work, for example a "painter" like Roxame, would propose developments (drawing, colors... ) and orient the creationa ccording to the "spectators" notes.

2.5. Artists, producers, spectators: the fight for control

Not everybody is in quest of power or even of liberty. But, in practice, there is a border issue somewhere the artist intention, their ego and more or less transcendent message, and the desire of rich and free engagement by the spectator. And to make it still less imple :
- artists may act individually or in group, cooperate on a peer-to-peer basis or on more structured distribution of parts,
- spectators do not behave the same way when alone and when taken in a small or large group,
- between spectators and artists come the producers, with their financial or political weigh and responsibilities, and their short and long term strategies.

The scene has reached a new height of complexity with transmedia, where an original story is told diffrent ways through different medias, then completed by user generated contents and giving feedback to the production through metadata (these latter becoming sometimes the real financial value and by itself a complex process operating (partly) in real time.

Frequently, interactive works are disappointing for spectators as well as for artists.
The spectators find very limited affordances, and a lot of difficulties to see how act effectively.
The author is disappointed by the poor behavior of spectators. He smirks "they don't understand' and "they go off when they think they have discovered the trick"

A first easy metaphor is to say that, with a work or art, artists open spaces where spectators can play. This space may be
- strictly constrained, for instance in cinema, where the spectators have no other choice than to remain seated and look/hear.
- widely opened if the work is just a score, let's just a tune, which spectators can use passively in a concert, or actively if they sing it for themselves, with the whole set of parts than are taken between the original author and the final spectator: interprets, directors, sound engineers, dancers, participating spectators on a dance floor.

3. Artists and persons

Artists . See our index or artists specially interested in interaction.

Other artitss, not specially centered on interaction, but looking for it sometimes:
- Agam, in 1955 proposed a set of formes that the spectator could move, within the limites set by the artist.

Other persons:
Nicolas Nova. scientist and consultant,
Etienne Mineur. Cofounder and art director of Incandescence studio,
Yves Rinato, president, Intactile Design

4. References

< Wikipedia about interactive art.
< Praticable. From Participation to Interaction in Contemporary Art, edited by Samuel Bianchini and Erik Verhagen. MIT Press 2016. 930 pages,not very oriented towards digital art, but with a lot of digital pieces.
< Images interactives. Art ccontemporain, recherche et création numérique. Sous la direction de Jean-Paul Fourmentraux. La lettre volée, Bruxelles, 2016.
- ACM Sigchi (Special interest group on computer human interaction) website.
- Techniques : see [Garrelts], [Processing], [Jouable], [Berger-Lioret 2012] pp. 99 sq. , 186 sq.
- You can test a textual form of interaction on Cleverbot, in the spirit of Eliza (See Weizenbaum).
- A lot in Digital Storytelling. A creator's guide to interacdtive entertainment. [Miller, 2014].
- AIAS (Academy of Interactive Arts and Sciences). Wikipedia. Mainly awards for games.

< Cinéma, interactivité et société, edited by Jean-Maarie Dallet. VDMC, Bruxelles, 2013. See a note by Sliders Lab.

< Le design d'interaction, Edited by Luca Marchetti & Emmanuelle Quinz. Editorial coordination by Lucie Wullschleger. Contributions by Anna Bernagozzi (Ensad Paris), Christophe Talled, Experientia ( (Mark Vanderbeeke and Joes Voels), Nicolas Gaudron, Virginia Cruz (IDSL), Intersezioni (Luca Buttafava et Alessandro Confalonieri). Online. It includes an important number of references.

< Aesthetics of Interaction in Digital Art. by Katja Kwastek. MIT Press 2013.
< Music and Human-Computer Interaction. by S. Holland, K. Wilkie, P. Mulholland and A. Seago (Eds.). Springer Series on Cultural Computing. Springer 2013.
< Sonde 04#12 (2012), experimentation about interaction by Emmanuel Guez, Christian Giriat and Xavier Boissarie. They explain their work in [Moindrot], pp.269-284.
- Interactor In interaction design see [Janet Murray 2012].
< Virtual Reality in Medicine, by Robert Riener and Matthias Harders. Springer, London 2012. A lot of data about input periphery, visual aspects, haptic aspects, auditory aspects, olfactory and gustatory aspects.
- The symposium Siana 2011 held a session "Au doigt et à l'oeil", with three speakers :
< The living art, [Aziosmanoff 2009].
< Inventing the Medium. Principles of Interaction Design as a Cultural Practice, by Janet H. Murray. MIT Press 2011
.< Composing for the Interactive Medium; by Taylor Robyn, Schofield Guy, Shearer John, Wallace Jayne, Wright, Peter, Boulanger Pierre, andOlivier Patrick/ Communication at Vric 2011. Online.
< Creating interactivity of our heritage stored in Museums Experimentation with Château des Ducs de Bretagne History Museum of Nantes, by Florent Laroche and Jean-Louis Kerouanton.Communication at Laval Virtual 2010.
< Interactive landscapes, by Daan Roosegaarde. Nai Uitgevers 2010.
< Principles of Interaction Programming, by Harold Thimbleby. MIT Press 2010. (Different on the above one?)
- Metaphors of Experience and Self in Interactive Systems. by Roman Danylak. Communication at Laval Virtual 2010.
< Agents conversationnels animés. Greta, une plate-forme d'agent conversationnel expressif. by J.C. Martin. Hermès/Lavoisier 2010.
< Interactive architecture, by Michael Fox and Miles Kemp. Pricenton Architectural Press, 2009
< Programming interactivity, by Joshua Noble. O'Reilly 2009. With Processing, Arduino and openFrameworks.

< Human-Computer Interaction. Fundamentals. by Andrew Sears and Julie A. Jacko (eds.). CRC Press 2009. A big book. Some of the chapters:
. Perceptual-Motor interaction
. Emotion
. Cognitive architecture
. Human-Error identification
. Wearable computers.

- Body degree zero. Anatomy of an interactive performance, by Alan Dunning and Paul Woodrow. 16 pages in [Adams 2008]
- Behind the screen: installations for the interactive future, by Ted Hiebert. 18 pages in [Adams 2008]
- It is forbidden not to touch : some remarks on the (forgotten parts of the) history of interactivity and virtuality, par Peter Weibel. 21 pages dans [Grau, 2007]
- Twin-Touch-Test-Redux: Media archelogical approach to Art, interactivity and tactility, by Erkki Huhtamo. 31 pages in [Grau 2007]. With rather important references to the Italian Futurists and to Marcel Duchamp.
< Press On. Principles of Interaction Programming. by Harold Thimbleby. MIT Press 2007.
- A whole chapter in [Popper 2007] (Interactive digital installations).
< Qu'apporte l'interactivité à la littérature numérique ? by Philippe Bootz. Olatz 2006. Online.
- Building interactive world in 3D. Virtualisation sets and pre-visualization for games, films and the web. by Jean-Marc Gauthier. Elsevier 2005.
- A note (in French) by Pierre Berger in 2004
- Interagir avec les nouvelles technologies. Nouvelles de Danse Contredanse, Bruxelles 2004. The book includes a CD-Rom showinw works and providing software.
< La notion de figure dans les arts interactifs, by Jean-Marie Dallet. PhD thesis directed by Jean-Louis Boissier. 2003. A review of literary figures and their transcription in cinema and interaction.
< Windows and mirrors: interaction design, digital art and the myth of transparency, by Jay David Bolter and Diane Gromala. MIT Press 2003.
< Twisty little passages. An approcach to interactive fiction. by Nick Montfort. MIT Press 2003.
< De l'interactivité à l'autonomie comportementale d'une création. by Michel Bret. Brouillard Précis, Marseille 2001.
< Interactifs. Fonctions et usages dans les musées. by Bernadette Goldstein. Direction des musées de France. 1996.
< Computer interface design. by Brenda Laurel (ed.). Addison-Wesley 1990.

DICCAN'S PARTNERS:

Varia to be edited

 

Aufofinality.
The work must want itself a a work of art, then seduce.
Not necessarily interactive. ?? Evolutive ? If not evolutive, its finality is external.
The basis is a loop : see/evaluate/act.
Must show itself as interesting. ex. to light on during nihgt

Expression
Show that it has its perception of the spectator
Show its own mood, what it is doing

Autonomy.
- Have its own temporality. Different from human social conventions? But tuned with them.
- Aptitude to make its own choices, its agency

There is a dynamic in presence of somebody:
- detect the presence of somebody
- if evaluated as interesgint, call its attention, show that it is perceived, offer some agency
- make hypotheses on what the spectator wants
- when it acts, answer to this agency
- evaluate its actions (including mouse/keyboard), but also its mood, moves, eattention.

A rich interior life
P. desire to seduce. conserve presence/attention.
H. exchange intensity
P. duration of pf excja,ge/ Summa of exchanges during the recent times

Standard model of a control surface

On/Off switch . Globally puts on/off. Indicates that somebody is willing to interact

On/Off button. According to the place of the button, and possibly a related caption, generally sosets some parameter of binary value (e.g. AM/FM)
indicates taht somebody is here, the caption could be more "mgeneral", imperatif e.g. ask the user to qualify themselves.

Continuous button (circular or linear). Give a value within some intervfal. Sets some pece to this value e.g. wavelengtth
MMay be... may describe the user, may assign tome general wish parameter

The precededn interface inputs can be used multiple times, a series of on/off, couble clicks, or some kind of musical bits
Here the "interpretation" includes some kind of globalizatin. H.meter, neural synapse, musical beat, rythm

if continuous inmut
- tthe sppec of change can be used (the user is nervous, or very active)
- recognition of patterns, typically the converging mode of somebody on a radio set aiming to a given emitter
then this pattern could push to a helping process to reach more rapidly the target frequency

All these simple inputs will see their "meaning" interpreted/expanded according to the states of the system, of the user state, of which usder

Then we get a Q/R (stimulus response S) which is not determinitic, or at leat includes much more parameters that what the user thinks. Pleasant/unpleasant

As the input are not "commandes" but "wishes", S may apply its own L criteria, but also ethic rules, commercial aims, requirement to pay.
Not commandes perhaps, but in action, OK upon the aims, no ambiguityu

Possible haptic repsonse

Elaboré, le S va faire la distinction entre les demandes/intentions explicites de l'utilisateur et l'utilisation
- de son comportemnet plus ou moins contrôlé/inconscient
- de sa personnalité connue
- avec des modèles généraux.

 

- - -

C'est la raison pour laquelle je suis extrêmement réservé sur les systèmes complexes interactifs. Faute d'un apprentissage qui peut être parfois long et fastidieux (violon), on ne peut que se cantonner dans le lieu commun. Donc à l'opposé de l'art.

 

C'est la raison pour laquelle je suis extrêmement réservé sur les systèmes complexes interactifs. Faute d'un apprentissage qui peut être parfois long et fastidieux (violon), on ne peut que se cantonner dans le lieu commun. Donc à l'opposé de l'art.

Bien d’accord.

Il me semble qu’il faut distinguer deux cas.

1. C’est l’artiste qui interagit. Le système interactif est un dispositif de création comme d’autres.

D’une certaine façon, la boucle du peintre : œil qui voit, main qui pose une touche, œil qui voit la touche, conduit au jugement et de là à la touche suivante, est un dispositif interactif. Et là, dans n’importe quel art, il faut une formation, un entraînement.

En peinture, il y a des exceptions. Parfois un très beau dessin d’enfant. Plus étonnant, les deux petites toiles faites par mon frère Jean-Marie avec ma peinture à l’huile (je ne trouve plus l’autre). Elles sont à la fois originales, bien construites et bien équilibrées quant aux couleurs. Or il n’avait jamais touché une brosse avant et n’a jamais recommencé par la suite.

C’est un peu vrai pour le chant aussi.

Mais cela rejoint ta formule « systèmes complexes interactifs ». La peinture est un des plus simples systèmes, comparé par exemple à n’importe quel instrument de musique. Surtout sur de petites toiles ou papiers. Car dès que c’est grand, l’œil se fait beaucoup plus exigeant.

2. (cas plus rare) L’artiste demande au public de réagir.
Et en général ça ne va pas loin, sauf si par exemple un bon danseur s’en mêle. Disons, un autre genre de professionnel.

 

 

 

 

Galerie Charlot An important supporter of digital art.