Semiotica 202 (2014), 631-640.
Abstract
In this study, we present a model of change in visual perception that is complimentary to a prior semiotic model of static visual perception. As in the previous study, a sign is defined to be a triadic relation that is derived from universal categories of thought, as proposed by Charles S. Peirce. This semiotic model is based upon the empirical findings of Benjamin Libet in the neurophysiology of sensory awareness. He has shown that sensory awareness is a physiologically discontinuous process although it is subjectively experienced as being continuous. Libet has demonstrated that, successive moments of sensory awareness are separated by brief intervals of unconscious signal processing. We propose that this unconscious processing of sensory signals includes the formation of unconscious associations that mediate between successive moments of sensory awareness. Furthermore, vision science provides empirical evidence that the formation of unconscious associations is governed by the Peircean categories of thought. On this basis, categorical typologies are proposed for both veridical and illusory visual change.
Keywords: Peircean categories; semiotic modeling; perception of change; illusions of motion.
Semiotic models of static visual perception and static visual illusions have been presented in a prior study (Cantor 2014). These semiotic models are based on a triadic concept of sign that is derived from the universal categories of thought, as proposed by Charles S. Peirce (1992 [1868]). The same fundamental principles are applied in this study. We denote the Peircean categories as Attribution (1st Category), Opposition (2nd Category) and Mediation (3rd Category). For the purposes of this study, a sign is defined to be a triadic relation comprised of a binary opposition between a Representamen and an Object that is mediated by an Interpretant. With this sign concept, the ground of an Interpretant is always in the mind of an interpreter while the ground of the other relates may be in the physical world or in the mind of an interpreter. A categorical typology of signs that is based on the concept of sign as representation (after Peirce), which was anticipated historically in the classical principles of mental association, will be expressed as relations of Similarity (1st Category), Contiguity (2nd Category) and Conventionality (3rd Category) between Representamen and Object (see Cantor 2011, for historical references). In this typology, similarity refers to a commonality of attributes, contiguity refers to a commonality of place in space and time, and conventionality refers to a commonality in thought or action acquired by repetition for an individual (by habit) or by agreement for a group (by custom). Furthermore, we shall use the categorical typology of Detection (1st Category), Localization (2nd Category) and Identification (3rd Category) to model the phenomenology of illusory motion (cf. Cantor 2014, on static illusions). Neurophysiological studies of the conscious human brain by Benjamin Libet (2004) provide an empirical basis for our semiotic model of change in visual perception. For convenience of reference, all the examples of illusory change of place or motion may be found in Palmer (1999).
Our understanding of visual perception is based upon both objective and subjective knowledge. Objective knowledge involves both optic (preretinal) and physiological (postretinal) processes. Subjective knowledge involves both conscious and unconscious processes.
2.1. Objective visual processes
In monocular vision, the opened eye is directed toward a region of interest in the visual world. Light emanating from that direction is focused and admitted into the eye by the cornea-pupil-lens system. The focusing process forms a light-cone that impinges upon the retina, which is a layer of photosensitive cells that covers the posterior surface of the ocular chamber. A cross-section of this light-cone constitutes a virtual or optic image of the visual world. At any moment, the initial physiological event in the process of visual perception is the interaction of an optic image with the retinal photoreceptors. In this event, the light-sensitive cells are collectively stimulated to produce a two-dimensional distribution of electrochemical signals termed a retinal image. A retinal image may be regarded as a physiological representation of an optic image. The stimulation of a photoreceptor produces a train of discrete electrochemical signals that are transformed and transmitted along the visual pathway extending from the eye to the primary visual cortex (see Cantor 2012, for semiotically pertinent details and references). The correspondence between the distribution of signals in a retinal image and the distribution of signals that are relayed to the primary visual cortex is termed a retinotopic mapping since it preserves the relation of spatial proximity (Palmer 1999:149). By a process that is roughly analogous to the formation of visual images, the aperture–lens mechanism of a camera collects and focuses light to form a light-cone containing optic images that interact with a light-sensitive film to produce a photographic image.
2.2. Subjective visual processes
Benjamin Libet has demonstrated in conscious neurosurgical patients that there is an approximately one-half second interval between the time of arrival of sensory signals in the cerebral cortex, as indicated by an initial cortical response, and the time of onset of sensory awareness (Libet 2004:88). He has also demonstrated that sensory signals may be modified during this unconscious interval (Libet 2004:201). To account for the fact that we are not conscious of any temporal gaps in our awareness of sensory stimuli, i.e., that sensory experience is subjectively continuous, Libet infers that there must be an unconscious mental process by which sensory awareness is subjectively experienced or interpreted as beginning at the same time as the initial cortical response to a stimulus. It is as if sensory awareness is initiated unconsciously and after a brief interval is referred backwards in subjective time (Libet 2004:79, 86). Consequently, sensory experience is subjectively continuous while being objectively discontinuous (Libet 2004:113).
We have seen that sensory perception is a physiologically discontinuous process that is subjectively experienced as continuous. Hence, a series of discrete retinal images produces a series of discrete cortical activations that produce a series of brief intervals of visual awareness and this series is unconsciously interpreted as a continuous process of visual awareness (cf. Libet 2004:112). Each moment of visual awareness may be regarded as the temporal experience of a subjective image which is a mental representation of a retinal image. By the work of Libet, we know that successive subjective images are separated by brief intervals during which visual signals are processed unconsciously. It is likely that during these intervals of unconscious signal processing, correspondences are established between the features of successive subjective images which result in the experience of spatial continuity. By an analogous process, a succession of static images that are presented to the eye at an appropriate rate produces the perception of a continuously moving or cinematic image (cf. Palmer 1999:474). It is likely that correspondences between features in successive subjective images that produce the illusion of visual continuity are recognized in the brief unconscious intervals that mediate intervals of visual awareness. We postulate that these correspondences entail the same categorical principles of mental association that operate in the perception of static images (cf. Cantor 2012). Furthermore, it is likely that a process of unconscious valuation also occurs in the unconscious intervals between moments of visual awareness (cf. Cantor 2012, on visual valuation). Such a process would determine whether or not visual signals that arrive in the cortex will result in visual awareness (cf. Libet 2004:72).
In visual perception, each subjective image and its successor constitutes a binary opposition that is marked by temporal order and is mediated by an unconscious interpretant. Such triadic relations constitute mental signs and a series of such signs involves a type of sign action or semiosis. In general, if Ak-1, Ak and Ak+1, are three consecutive subjective images that are separated by two consecutive intervals in which unconscious interpretants Ik and Ik+1 are formed, we may define two consecutive triadic signs as Ak-1
Ak and Ak
Ak+1, where the representamen of the first sign is the object of the second sign and the interpretant of the second sign is dependent upon the interpretant of the first. We refer to this process as visual semiosis, which may be symbolically represented as
AkAk+1 . . . In this representation of serial sign action, the temporal order of events is from left to right, while the direction of reference is from right to left, as indicated by the arrows. Note that, in visual semiosis a representamen at one stage becomes an object in the next stage. This is in contrast with Peircean semiosis where an interpretant at one stage becomes a representamen in the next stage.
The concept of subjective visual change may be defined in terms of binary relations between successive moments of visual awareness. When there is a perceivable difference between corresponding features of subjective images, the later image is marked as changed while the earlier image is unmarked (cf. Cantor 2014, on categories of markedness). A series of such binary oppositions produces the subjective experience of visual change. When the degree of difference is appropriate, the change will be subjectively experienced as continuous. When there is no detectable difference between subjective images, there will be no subjective experience of visual change. Hence, continuous change in the visual world is represented in the mind by a series of subjective images which is interpreted as continuous change in a visual scene. Change may be perceived in relations between successive images in the subjective present as determined by short term memory or in relations between images in the subjective present and in the recalled past as determined by long term memory.
The Peircean categorical typology for signs as representations may be applied to each sign in the process of visual semiosis. Hence, the categorical relations between successive subjective images in visual awareness are Similarity, Contiguity and Conventionality where:
In a previous study, we have demonstrated that the principles of perceptual organization for static images are based upon the Peircean universal categories of thought (Cantor 2014). Hence, it is likely that the perception of visual change that emerges from the presentation of consecutive subjective images is determined by categorical types of phenomenal change over time. The categorical types of change for objects are:
Empirical studies have confirmed that the correspondence between successive static images in cinematic displays that results in the perception of continuous object motion is primarily due to proximity (contiguity) of corresponding features and the rate of presentation of the images which is a physiological convention or habit (cf. Palmer 1999, on the correspondence problem for apparent motion). It has also been empirically demonstrated that the perception of continuous object motion is to a lesser degree influenced by similarity of corresponding features, e.g. size, shape, and possibly color.
The perception of veridical change in objects of perception, as opposed to apparent change due to extrinsic factors, depends upon the use of unconscious conventions or habits that have been established by repeated visual experience. Such conventions may be regarded as assumptions of perceptual constancy (cf. Palmer 1999:312). These unconscious assumptions operate in all the phenomenal categories of visual perception, i.e., Detection, Localization and Identification (cf. Cantor 2014). In what follows:
Hence, Detection, Localization and Identification are 1st, 2nd and 3rd Category perceptual processes, respectively.
7.1. Change in detection
The initial event in the visual detection of objects, is the perception of light reflected from their surfaces. Color and brightness are phenomenal attributes of objects. In the process of detection, unconscious assumptions of color constancy and lightness constancy maintain veridical perception of these attributes despite change in the spectrum or intensity of the source of illumination (cf. Palmer 1999:125, 133).
7.2. Change in localization
In the process of object localization, the unconscious assumption of motion constancy maintains veridical positioning and orientation despite changing extrinsic factors such as eye, head and body movements of the viewer (cf. Palmer 1999:487). Furthermore, the unconscious assumption of size constancy maintains veridical perception of object distance despite expansion or contraction of the object image due to movement of the object or viewer along the line of sight (Palmer 1999:315).
7.3. Change in identification
In the process of object identification, the unconscious assumption of shape constancy (the rigidity heuristic) maintains veridical identification despite the shape changes that result from changing perspectives due to movements of the object or viewer (cf. Palmer 1999:490).
We have seen that continuous change in the visual world is initially represented by a succession of retinal images and subsequently represented by a succession of discrete subjective images that is unconsciously interpreted as temporally continuous change in a visual scene. Furthermore, at any moment of awareness, a subjective image, which represents the activation pattern of a two-dimensional array of discrete photoreceptors, is unconsciously interpreted as a spatially continuous visual scene, i.e., without gaps. It has been suggested that the perception of unitary objects in an integrated visual scene is due to attentional actions that bind features into objects and objects into configurations (Palmer 1999:557). From a semiotic perspective, we may add that the categorical principles of correspondence for the perception of visual change are agents of a syntagmatic binding process. Similarly, we may regard the categorical principles of perceptual organization for static visual perception as agents of a paradigmatic binding process (cf. Cantor 2014). In this way, continuous and integrated visual perception would be a result of concurrent syntagmatic and paradigmatic binding processes that are governed by the Peircean categories of thought.
Categorical illusions of visual motion involve the illusory detection, localization or identification of a change of place.
9.1.Illusory detection of motion
Illusions of detection include motion aftereffects and the autokinetic effect. Motion aftereffects are produced by prolonged viewing of constant motion (Palmer 1999:470). For example, the Waterfall Illusion is produced when a viewer looks at a waterfall without moving the eyes and then looks at the adjacent landscape. This results in a sensation of upward movement in the stationary landscape. The direction of this movement is determined by the direction of movement in the inverted retinal image. The autokinetic effect is produced by prolonged fixation of a stationary spot of light in total darkness that prevents direct perception of the absence of relative motion (Palmer 1999:471). Under these conditions, the spot of light appears to move in a constant direction. This effect is thought to originate in signals produced by the extraoccular muscles. Both of these examples involve paradoxical motion, i.e., the perception of subjective motion in the absence of objective motion.
9.2.Illusory localization of motion
Illusory localization of motion is due to the unconscious assumption in figure-ground segregation that the ground is to be perceived as stationary. This convention results in illusions of relative motion: induced object-motion and induced self-motion. Induced object-motion refers to the illusion in which a stationary object is perceived as moving when its image is surrounded by the image of a larger moving object, e.g., when a large cloud passes in front of the Moon, it may look as though the Moon is moving through the cloud (Palmer 1999:501). Induced self-motion refers to the illusory sensation of motion experienced by a stationary viewer that is provoked by a global pattern of motion in the visual field. For example, this illusion is produced when a viewer looks through a window of a stationary train and the entire visual field is occupied by the image of an adjacent moving train (Palmer 1999:504).
9.3.Illusory identification of motion
Identification entails reference to an individual or a type by means of a name or verbal convention. Illusory identification occurs in motion adaptation and apparent motion. In motion adaptation, an illusory identification of motion in terms of its speed is produced by prolonged viewing of things moving at a constant speed in the same direction, which makes them seem to slow down (Palmer 1999:470). In apparent motion, a visual signal that is discontinuous in time will create the illusion of continuous motion when presented to the viewer at an appropriate rate (Palmer 1999:474).
In this study, we have constructed a semiotic model of visual change on the basis of experimental findings in the neurophysiology of sensory awareness. These findings prove that subjectively continuous visual perception is produced by an objectively discontinuous physiological process, i.e., successive intervals of visual awareness are separated by brief nonconscious intervals. It is likely that the nonconscious intervals between successive moments of awareness allow time for the unconscious processing of visual signals. Hence, visual perception may be regarded as a temporal series of triadic relations, where each consists of a binary opposition between a moment of visual awareness and its successor that is mediated by a time interval in which visual signals are unconsciously interpreted. In other words, visual perception is a process of visual semiosis. On this basis, continuous visual change is perceived when successive visual signs are marked by corresponding differences. We propose that the types of correspondence that are unconsciously recognized in the intervals between moments of visual awareness are derived from the Peircean categories of thought. On this basis, we have constructed categorical typologies for veridical visual change and for visual illusions of motion. These results are complimentary to the findings of a prior study that presents semiotic models of static veridical visual perception and static visual illusions. Hence, semiotic modeling supports the claim that visual perception is fundamentally a semiotic process that is governed by universal categories of thought.