Semiotica 192 (2012), 65-76.
In this paper, we demonstrate an empirical basis for Roentgen semiotics in vision science. We first observe that the triad of phases in the process of Roentgen diagnosis, previously termed detection, localization and identification, originate at the cellular level in well described visual pathways in the brain. We then demonstrate that the phenomenology of local Roentgen signs is derived from the physiology of cells in the detection, localization and identification pathways. Furthermore, we infer that a fundamental duality principle in the interpretation of local Roentgen signs may be derived from a similar duality principle that governs the physiology of cells in the detection pathway. In this way, perceptual phenomenology of the mind is determined by cellular physiology of the brain.
Keywords: Roentgen semiotics; vision science; Peircean categories; visual pathway; double opponency; duality principle.
In a series of papers, the author has studied the phenomenology of Roentgen diagnosis from the perspective of Peircean semiotics. (Cantor 2000 to the present). The aim of this paper is to demonstrate an empirical basis for Roentgen semiotics in vision science. Our approach to Roentgen semiotics is based upon the Peircean concept of sign as representation, i.e., a Roentgen sign is a triadic relation among a representamen, an object and an interpretant such that
The interpretation or “reading” of an image depends upon recollection of collateral knowledge by the interpreter. This includes knowledge of the conditions under which the image was formed and the context in which the sign functions. On this basis, we distinguish between material and mental signs. For material signs, the representamen and object are physical entities and the physical world is their relate ground (Cantor 2002). For mental signs, the representamen and object are mental entities and the mind of the interpreter is their relate ground (Cantor 2005). For both types, the interpretant ground is assumed to be the mind of an interpreter (Cantor 2010). These material and mental perspectives are complimentary and either one may be employed at different stages of the argument.
Visual representations or images are carriers of visual signs. Therefore, to understand the formation of visual signs we must review the anatomy and physiology of the brain along those pathways in which visual images are produced and processed. The initial emphasis will be on the semiotic similarities and differences between natural optic images and Roentgen images. Most of the empirical information on vision science may be found in the excellent synthesis by Palmer (1999), with some details drawn from expositions by Hubel (1988), Hubel and Wiesel (2001), Mishkin et al (2001) and Livingstone (2002). In the following sections, we review the interrelated processes involved in the formation of retinal, pictorial and Roentgen images.
In normal binocular vision, the four-dimensional spatiotemporal world is represented by the continuous mental fusion of two three-dimensional spatiotemporal retinal images. Hence, a retinal image is a process occurring in two spatial dimensions. In each eye, a retinal image is formed by the continuous sampling of ambient light reflected from the surfaces of opaque environmental objects. This reflected light enters the chamber of the eye through a variable aperture and is focused by the cornea and lens on a variable point behind the lens. It is known that a cross-section of the light cone that emerges from the focal point at any moment constitutes a completely inverted virtual image of the visual scene (cf. the notion of focal point as a ‘virtual pinhole’ in Palmer 1999:21). This mode of image formation is called perspective projection. When such virtual images interact with the retina, a structure covering the back of the eye that contains light-sensitive cells or photoreceptors, they produce two-dimensional spatial distributions of electrochemical signals that are continually transmitted by nerve cells or neurons to higher levels of the brain for processing. Signals produced by retinal photoreceptors are graded events while signals generated by neurons are discrete events or impulses. Neuronal signals may have excitatory or inhibitory effects, i.e., the reception of signals may provoke an increase or decrease in the spontaneous signaling rate of a neuron (Hubel and Wiesel 1988:55). The three dimensional spatiotemporal distribution of electrochemical signals generated by the retina is a physical representation of the visual scene called the retinal image. Hence, the retinal image is a process. The ‘code’ by which visual experience is represented over time by electrochemical signals is unknown. In binocular vision, signals that constitute the retinal images of both eyes are merged to produce stereoscopic vision. Since Roentgen images are static, we shall simplify the argument by ignoring the temporal dimension (cf. Palmer 1999: 79). The temporal phenomenology of Roentgen semiotics is examined in Cantor (2010).
A realistic pictorial image of a scene in three-dimensional space, e.g., a drawing, painting or photograph, is semiotically equivalent to a momentary retinal image, i.e., pictorial and retinal images present the same static visual signs (cf. the concept of informational equivalence in Palmer 1999:79). Both the realist artist and the photographer create material images that are perceived as perspective projections, in the same way as retinal images (Livingstone 2002:100). Such material images become the representamina of material signs when they are observed by an interpreter. A binocularly viewed pictorial image is semiotically equivalent to a monocularly viewed image, although with a wider field of view (cf. Palmer 1999:229, on pictorial information). Clearly, perceived pictorial images may be chromatic or achromatic, while perceived retinal images of natural scenes are necessarily chromatic.
Roentgen images differ from pictorial images both in the way they are formed and in the signs they present. While light is reflected (actually, absorbed and then emitted) by most biological surfaces, diagnostic x-rays are either absorbed or transmitted. Hence, pictorial images represent material properties and surface contours of objects while Roentgen images represent material properties and interior parts of objects, i.e., things that are inaccessible to natural vision. Roentgen images are produced by the projection of a divergent beam of x-rays through a region of the body. Interaction of the beam with materials of different densities along its path results in a differential attenuation of the beam. Hence, a cross-section of the transmitted beam is a two-dimensional distribution of radiation intensity that constitutes a virtual Roentgen image. The instantaneous interaction of a virtual Roentgen image with the detector of an imaging system produces a representation in the form of electrochemical signals. This mediating representation is then transformed into a two-dimensional distribution of light intensities that constitutes a visible Roentgen image. While retinal images of natural objects are necessarily chromatic and pictorial images may be chromatic or achromatic, Roentgen images are colorless since the natural color of objects is a perceived property of reflected light. By convention, Roentgen images are formed from ‘white’ light. In the same way as for pictorial images, binocular viewing of Roentgen images is semiotically equivalent to monocular viewing, i.e., the two modes of viewing present the same static visual signs.
On a macroscopic level, the brain is a bilaterally symmetric structure with two intercommunicating cerebral hemispheres. Each hemisphere is subdivided into intercommunicating occipital, parietal, temporal and frontal lobes, in progression from back to front. The cerebral cortex is an approximately two millimeter thick layered structure that forms the highly convoluted surface of the cerebral hemispheres. On a microscopic level, the cortex is a vast complex network of approximately ten billion neurons. This highly integrated and differentiated network is comprised of innumerable hierarchically organized subnetworks that may be globally or regionally distributed (see Cantor 2009, for an overview from a semiotic perspective and references). These neuronal subnetworks are commonly organized as layered structures. The visual system is comprised of functionally specific and hierarchically organized neuronal networks that intercommunicate through specialized visual pathways. In each cerebral hemisphere, a detection pathway leads from the retina to the thalamus (a processing and relay station in the midbrain) and from there to the striate cortex or primary visual cortex that occupies most of the occipital lobe. Along this pathway, serially connected neurons process signals generated by individual photoreceptors that result in signals generated by specialized striate neurons that represent local visual features (Hubel and Wiesel 2001: 67-85). Such elementary visual features include luminance, color, line, edge and motion. The striate cortex is almost completely surrounded by the prestriate cortex of the occipital lobe. The prestriate cortex is functionally subdivided into areas that selectively process signals representing motion, position, color and shape (Hubel and Wiesel 2001: 95). Two separate pathways lead beyond the pestriate cortex to the parietal and temporal lobes. The occipitoparietal pathway processes signals that represent the motion and position of objects and may be called the localization pathway. The occipitotemporal pathway processes signals that represent the color and shape of objects and may be called the identification pathway (cf. Mishkin et al. 2001; Livingstone and Hubel 2002; Palmer 1999: 43). The aim of this study is to examine the functions of the detection, localization and identification pathways from the perspective of Roentgen semiotics. Since binocular viewing of a Roentgen image is semiotically equivalent to monocular viewing of the image, the discussion that follows may be restricted, without loss of generality, to the visual pathways in a single cerebral hemisphere. Furthermore, since Roentgen images are static and colorless, we shall limit our discussion to those pathways that are not involved in motion or color perception.
The detection pathway
The detection pathway (the “ Is” pathway) leading from the retina to the striate cortex may be schematically subdivided into two serial pathways that lead from the retina to a relay station in the thalamus (the lateral geniculate nucleus) and from there to the striate cortex. The retina, the lateral geniculate nucleus and the striate cortex are all layered structures. This spatial organization facilitates the topographic mapping of the retinal image to the striate cortex, i.e., a mapping that preserves two dimensional spatial relations. The retina is made up of three cell layers: a deep layer of photoreceptors, a superficial layer of ganglion or relay cells and a third layer that mediates between the other two. In monocular vision, signals generated by photoreceptors in one eye are processed by retinal ganglion cells and then relayed to geniculate cells in the thalamus for further processing and again relayed to cells in Layer IV of the striate cortex. From there, signals are distributed to functionally specific cells in each of the six layers of the striate cortex (Hubel and Wiesel 2001). Hence, signals that constitute the retinal image are processed in stages along the detection pathway.
Perceptual detection. Each cell in the detection pathway receives signals from a corresponding focal region of the retina called the receptive field of the cell. Retinal ganglion cells, thalamic geniculate cells and striate Layer IV cells all have circularly symmetric receptive fields with center-surround functional organization (Palmer 1999:147). This means that there are dual functional types of center-surround cells. In one type, the cell responds by excitation to a light signal in the center of its receptive field and by inhibition to a signal in the surround. In the other type, the cell responds by inhibition to a light signal in the center of its receptive field and by excitation to a signal in the surround. Hence, cells with center-surround receptive fields may detect a light spot on a dark field or a dark spot on a light field, i.e., such cells are luminance spot detectors. Since the center-surround functions are antagonistic, focal light signals that overlap both the center and the surround of a receptive field are detected as spots of intermediate luminance. Cells in the detection pathway beyond striate Layer IV have elongate receptive fields. These receptive fields may have bilaterally symmetric or antisymmetric spatiofunctional organization (Hubel and Wiesel 2001). Hence these cells detect lines with specific orientations, i.e., a light line that divides a dark field or a dark line that divides a light field or edges with specific orientations, i.e., a light edge next to a dark field or a dark edge next to a light field. The cells with symmetric receptive fields are line detectors and the cells with antisymmetric receptive fields are edge detectors. Other striate cells detect light or dark line ends or edge ends, i.e., interrupted lines or edges. The same double opponency or spatiofunctional duality holds for color sensitive spot, line, edge and end detectors. For a detailed discussion of this complex issue see Livingstone 2002: 53-63 or Palmer 1999: 120. Hence, a double opponency or oppositional duality determines the spatiofunctional organization of the receptive fields of cells that are specialized for the detection of elementary visual features. It is likely that line and edge detectors in the striate cortex receive their inputs from appropriately aligned spot detectors (Palmer 1999: 151). Note that the phenomenology of oppositional duality may be represented by the Peircean universal categories. With each receptive field subdivided into two components that are functional opposites:
Diagnostic detection. The first phase in the process of visual diagnosis is the detection of abnormality or change. Fundamentally, diagnostic detection is dependent upon the expectations of the viewer. While normality refers to what is expected in a given context, abnormality refers to what is unexpected. Hence, an abnormality in an image may be the presence of the unexpected or the absence of the expected, i.e., an unexpected presence or an unexpected absence (Cantor 2000, 2006a). In Roentgen semiotics, abnormalities may present as local, regional or global signs. A local sign may be perceived in an arbitrarily small neighborhood in an image (Cantor 2002). Local Roentgen signs may present in the interior or on the boundary of a region in an image. A local sign of abnormality or change, or local detector, is perceived as the unexpected presence or absence of an elementary feature in an image. A local interior or density sign is perceived as an unexpected focal area of increased or decreased optical density, i.e., increased or decreased lightness. Since boundaries of objects are represented in Roentgen images as light or dark lines or edges, a local boundary sign of abnormality may involve the unexpected presence or absence of a light or dark line or a light or dark edge. Other local boundary or interior signs of abnormality include the detection of unexpected line or edge interruptions. (cf.Cantor 2003a). Hence, the detection of local signs of abnormality in a Roentgen image is a function of the spot, line, edge and end detectors in the detection pathway.
Oppositional duality. Oppositional duality is a fundamental organizing principle in the phenomenology of thought. By this principle, dual binary relations are derived from two pairs of opposites. We have seen that the receptive fields of neurons in the detection pathway exhibit a spatiofunctional double opponency that is determined by the objective opposites region/complement and excitation/inhibition. Hence, double opponency is an objective oppositional duality. In Roentgen semiotics, the detection of abnormality or change is also derived from an oppositional duality. Dual signs of abnormality or change are determined by the pair of opposites presence/absence and unexpected/expected (Cantor 2006a). In diagnosis, abnormality or change is detected by the dual conditions:
Hence, signs that detect abnormality or change are derived from a subjective oppositional duality.
In Roentgen semiotics, interior density signs and boundary separation signs are local detectors of abnormality. Dual density signs are
Since the boundaries of regions in Roentgen images are represented by lines or edges that separate the inside from the outside, dual boundary signs are
These are the loss and gain of separation signs described by Cantor (2000). Other boundary signs of abnormality are determined by the pair of opposites presence/absence and line/edge. The corresponding dual boundary signs are
These are the exchange of separation signs described by Cantor (2000). In Roentgen images, the boundaries of parts of objects are represented by lines or edges within regions. Therefore, dual interior boundary signs of abnormality are
Other local signs of abnormality at lines or edges are based on the presence or absence of continuity or discontinuity. Important dual signs are
Hence, local boundary and interior signs of abnormality in Roentgen images are derived from oppositional dualities.
Categorical duality. A binary opposition may be thought of as a marked binary relation, where a binary relation is the concurrent awareness of two things and markedness is the perception of a difference between two things. Hence, a binary opposition is a binary relation in which one relate is marked and the other is unmarked. Based on concepts originating in linguistics and the Peircean categories of thought, three categorical types of markedness have been defined for binary relations (Cantor 2006b).
In ordinal markedness, the marked and unmarked relates have opposite attributes and therefore, ordinal markedness entails polar markedness. Clearly, both ordinal and polar markedness entail cardinal markedness. Hence, the categorical types of markedness are ordered by an inclusion rule in the same way as the Peircean categories (cf. Liszka 1996:46). It follows that the categorical types of markedness or difference determine types of opposition, the types of opposition determine types of oppositional duality and the types of oppositional duality determine types of local detectors. Recall that a sign is a triadic relation which may be thought of as a binary opposition between a representamen and an object that is mediated by an interpretant.
For signs of abnormality,
In both cases, the sign is marked by ‘unexpectedness’. Hence, dual interpretants of detectors of abnormality may be expressed as
Therefore, three categorical types of interpretant determine three categorical types of local detectors of abnormality. For example,
The localization pathway.
The localization pathway (the "Where" pathway) leads from the striate cortex to the prestriate cortex and from there to the parietal cortex (Mishkin 1972; Palmer 1999: 43). This occipitoparietal pathway separates into two functionally distinct pathways in which object motion and object depth, i.e. localizing relations, are processed (Livingstone and Hubel 1987). In what follows, we shall ignore the motion pathway since we are concerned only with signs that present in static images. The depth pathway distinguishes between objects and background and the relative distances of objects from the observer (Livingstone 2002:50). Since binocular viewing of a Roentgen image is semiotically equivalent to monocular viewing, we may ignore the effects of stereopsis. Moreover, since Roentgen images are produced by transmitted radiation rather than reflected light, signs of relative depth such as occlusion, shading and shadowing do not occur. Since Roentgen images are formed by central projection from the focal spot of an x-ray tube, contraction in depth, which is a property of perspective projection, is also unavailable as a sign of relative depth. Actually, magnification with increased distance between object and detector is a property of central projection. However, this effect is small in practice and rarely available as a depth localizer. Hence, the only depth localizers that are readily available in Roentgen semiotics are signs of physical contact. These are local boundary signs, e.g., the unexpected loss of a line or edge separating the inside from the outside of a region.
The identification pathway.
The identification pathway (the "What" pathway) leads from the striate cortex to the prestriate cortex and from there to the temporal cortex (Mishkin 1972; Palmer 1999: 43). This occipitotemporal pathway separates into two functionally distinct pathways in which signals representing object color and object shape, i.e. identifying qualities, are processed (Livingstone and Hubel 1987). The perceived color of an environmental object is determined by physical properties of its surface and viewing conditions (Livingstone 2002). Surface properties produce characteristic patterns of reflection of incident light. In this way, perceived color contributes to the discrimination and identification of environmental objects. Since Roentgen images are not produced by reflected light, they do not have natural color. However, the optical density or brightness of a Roentgen image of an object varies directly with its physical density. On a conventional gray scale of optical densities, higher physical densities are represented as lighter and lower densities as darker, e.g., images of structures containing bone, water and fat have light, intermediate and dark optical densities, respectively. Hence, the optical density of a region in an image signifies the material composition of the object that it represents. In this way, density signs contribute to the process of identification. The recognition or identification of objects by shape is the least understood function of the visual system. However, it is known that complex shapes such as faces, may be identified by specialized neurons in the temporal cortex (Palmer 1999: 195). Therefore, it is likely that specialized cells in the temporal cortex initiate the mental process of object recognition (cf. Mishkin et al 2001).
There is strong empirical evidence for the staged processing of images along specialized detection, localization and identification pathways in the brain. It is known that the same functional tricotomy exists in the phenomenology of perception (cf. Cantor 2002). The author has proposed that the ordering of these functions is based on the Peircean categories of thought. We have seen that the signals which constitute a retinal image are processed along the detection pathway by spot, line, edge and end detectors. Furthermore, we have seen that the receptive fields of these feature detectors exhibit a spatiofunctional double opponency or oppositional duality. We infer that the perceptual dualities previously described for local Roentgen signs are produced by functionally specialized cells in the detection pathway. In previous studies, we have examined the role of the Peircean categories in concept formation, i.e, concepts are formed by attribution, opposition and mediation. In this study, we have seen that the same categorical principles operate on the cellular level, in the anatomy and physiology of the brain.