In the fifth century, Socrates said, “The unexamined life is not worth living.” The tools we use to examine our lives have changed throughout history but the underlying desire to capture, store, recall, and share our experiences has not.
Tools for memory storage and recall can range from the simple, such as a piece of paper in a scrapbook, to the technologically complex, such as the uploading of digitally recorded memories. Since the earliest days of computing technology, we have sought to use it to extend and supplement our own cognitive abilities, including memory functions.
Humans have recorded memories in various forms throughout the course of history. Oral stories, texts, photographic images, and videos all capture bits and pieces of an experience for later recall and sharing with others. Historically, the durability of the media used to record memories has effected what we can learn from the past. Present day digitization of texts, images, and videos extends the life of memories stored in mediums that would once have fallen prey to physical decay.
However, none of these memory-aiding devices alone can serve to allow us to truly re-live a prior experience. Our memories of an experience can be brilliantly vivid or frustratingly dull, depending on the importance of the event, how focused we were during the original experience, and how often we recall the event in our minds. Our limited attentional capacity only allows us to capture a small fraction of what occurs during any event. Happily, our cognitive inner workings tend to capture that which was most important to us, yet we are often left wondering what else was happening that we might have missed.
The brain is also decidedly plastic and far from perfect, sometimes leaving us with the unsettling feeling of incomplete memories or with substitution errors when we try to recall an experience. It is little wonder then that we are seeking to use computer technologies to allow us to have a total, almost perfect recall of an experience.
The Message of the Pensieve
Utilizing an immersive, synthetic environment as a digital archive for storing, recalling, and sharing individual memories is an idea present in many books of science fiction and fantasy. For example, in the fourth book of the Harry Potter series, Albus Dumbledore’s Pensieve is described as a device that allows one to add “excess thoughts from one’s mind . . . examine them at one’s leisure . . . [and] spot patterns and links . . . [better] when they are in this form.”
Technology is close to realizing the goal of allowing us to completely capture our everyday experiences and share these archived experiences with others. This paper will examine some of the many potential implications for immersive memory recording applications at both the individual and societal levels. Virtual reality and the World Wide Web will be investigated as potential delivery vehicles for current and future immersive memory archives.
Toward an Immersive Platform for Memory Capture and Recall
The first step towards developing memory for an experience is capture. In cognitive science, attention acts as a filter for incoming sensory information and what is attended to may move to working memory. Information in working memory can then be transferred into permanent long term memory storage, where is it encoded into the network of associations that form our collective record of experiences.
One of the most consistent findings in cognitive science is that both attention and working memory capacity are limited in nature, meaning we cannot possibly hope to capture every detail of an experience. Steve Mann is arguably one of the first to use wearable computing technology to extend his experience capturing capability. His experiments have allowed him to develop relatively unobtrusive cameras to aid in capturing firstperson perspective video and audio of an experience.
With the incorporation of wearable computing technology onto his person, Mann views himself as transitioning toward total symbiosis with his computer equipment, becoming a cybernetic organism, or cyborg. Mann’s supplementation of his memory through wearable computers has not only provided him with the ability to re-play any moment in time through his visual and auditory record, he is also able to change how he experiences reality through computer augmentation of his visual ability.
Microsoft has undertaken a research project with the help of Gordon Bell to develop its own tool for continuous capture of everyday moments called the MyLifeBits project. Using MyLifeBits, Bell is able to compile and search through a record of every detail of his life, including photographs of things he sees, audio recordings of conversations, and digital records of every piece of paper that crosses his desk. These details are combined with a record of his daily activities, his browsing patterns on the Internet, and a comprehensive record of everything he has ever read or viewed.
While the MyLifeBits technology does not emphasize wearable computing per se, with the wearable tools consisting of a still-image camera and microphone, it does appear to be destined as one of the first consumer products for experience capture, as well as a more formal experiment in the type of memory-offloading the general public is participating in through the use of digital devices.
Some users are moving forward with currently available technology to develop their own ‘lifelogging’ or ‘glogging’, short for cyborg log, capability.
Wearable webcams or digital cameras mounted on a hat or to glasses is a popular lowcost option for experience capture among these users. Mobile phone cameras can be used to capture video from a first-person perspective on demand if the user does not want to continuously record video or use wearable computing devices.
Current Web tools allow users to share individually captured experiences with
others, providing a collective record for an event. Social networking sites such as YouTube and flickr demonstrate this kind of collective documentation via
media sharing tools and tagging capabilities. One group on flickr, called the Panopticon, created the group with the sole purpose of documenting images of
surveillance cameras, or in essence documenting those who document. This is a form of sousveillance, or inverse surveillance. Viewing of an event from several perspectives is possible by consolidating the multiple perspectives provided by individuals through photographs on flickr or videos on YouTube. Each of these files records the event from the perspective of one individual and, when viewed collectively, acts as a Pensieve-like archive of the entire event.
One example of this type of memory archive is the photographic record of the Obama presidential inauguration, in which hundreds of 2-D photographs from numerous independent viewers were combined form a 3-D view of the environment (The 44th President Inauguration). Examples of this type of life examination and archiving pervade the Web and demonstrate the desire of users to share and combine digital memory records in the future.
While the audio and video recording capability of current technology is
remarkable, to create a truly Pensieve-like memory device data beyond simple video and audio would need to be captured during the initial event. Capturing peripheral data would allow the user to not only extend their recording capabilities, but also allow for re-playing of an experience from an almost omniscient, third person perspective as is possible with Rowling’s Pensieve.
Visually, 360-degree cameras can be used to capture images beyond what is visible from a first person perspective. Any single event can be viewed from multiple perspectives by allowing personal wearable computers to gather information from the other users wearing computers within the area.
Documenting affective and body state data using wearable body monitoring devices and potentially using electroencephalography to record the user’s mental state can allow data beyond just visual and audio to be included in the re-playing of memories. Location and contextual information can be provided through global positioning system (GPS) devices such as mobile phones. Combining these technologies makes it possible to capture an event visually, audibly, cognitively, emotionally, and contextually, thereby allowing for the creation of a fully immersive memory.
The second step towards developing memory for an experience is storage. We can only imagine the technical challenges involved in storing the massive amount of data that would be captured by a comprehensive lifelogging device. Gordon Bell, even without continuous video logging of his experiences, captures over 1GB of data daily. While progress is being made towards developing storage and retrieval systems for information captured during lifelogging, we have many more cues to store and integrate within a Pensieve-like data storage system.
The format in which memory data is stored primarily depends on how they will be viewed or re-played and shared later. There are many technical challenges that would have to be overcome in this domain to ensure memory records would be available and playable in the long term.
The third step towards developing memory for an experience is retrieval.
Retrieval can mean simply locating the memory, but in this case we also use retrieval to mean re-playing the memory for the purposes of re-experiencing it or sharing it with others. The most likely environment with which to experience the Pensieve’s goal of total immersion within the memory is through some type of virtual or simulated reality. Char Davie’s imagined virtual reality as “a visual/aural spatiotemporal arena wherein mental models or abstract constructs can be given virtual embodiment in three dimensions and then be kinesthetically explored by others through full body immersion and interaction.”
For example, a Cave Automatic Virtual Environment (CAVE) is one in which the
user is surrounded in on all sides by screens on which can be projected any type of image. Sounds, smells, and other environmental data captured during memory recording can then be re-played to recreate the original experience. The CAVE environment realizes the idea of immersion into another persons ‘body’ of memories by completely surrounding them with the captured experience. By translating memories into a 360-degree, three-dimensional view, anyone can place himself/herself into a moment and physically experience another’s reality.
Although the technology is not yet equal to the task, the idea of studying a historical event through immersion into a Pensieve-like environment, with the ability to stroll around, pause, rewind, and examine the scenario from the perspective of each participant in the event is staggering.
Rowling’s Pensieve also allows the user to link memory traces together, much like a networked hypertext environment allows us to link various pieces of text and multimedia together to indicate relationships between them.
At the retrieval stage, it is likely that users may want to indicate relationships between various events through a type of memory hypertext structure. On its own, the recorded memory would serve as a hypomnemata, or context-less collection of data about an event and person within that event. A link structure through a virtual environment would provide a type of elementary narrative through a memory and serve to put various individual events within a larger context. Jaron Lanier suggested that linking of memory traces in virtual environments would be a powerful way to augment memory: “You can
play back your memory through time and classify your memories in various ways. You’d be able to run back through the experiential places you’ve been in order to be able to find people, tools.”
The ultimate re-playing of a memory may one day be direct electrical stimulationof the brain to re-create the pattern of activation recorded while an individual was experiencing an event. To become truly Pensieve-like, this information would eventually have to be augmented with information captured from external environmental sources to allow viewing of the experience from a third-person perspective, if desired.
Personal and Societal Implications
There are numerous personal and societal implications for this type of first-person account of a life. On a personal level, we must examine the impact of virtually recorded memories on the understanding of the self. In general, our understanding of ourselves is framed by autobiographical memory, or our memories for the events of our lives. By maintaining a separate, immersive record of life events our understanding of ourselves can be extended beyond just what we can store while an event is occurring.
Essentially, we free ourselves from attention and memory limitations. However complete the record for an event might be in the archive, as individuals we still view the record through the filter of our personality and worldview. These aspects of ourselves are likely to be changed by viewing the record, much as our understanding of the self might be, but at least initially we may be unable to view the memory from a perspective outside of our own.
When viewing the memory of another individual, the incorporation of experiencesby the engaged viewer into his/her concept of self is subject to personal misinterpretation, regardless of how objectively the data is captured. Consider for instance the idea of a physically challenged individual (e.g., wheelchair bound) as a lifelogger. Through immersion into this person’s memories, would a non-physically challenged individual be able to embody their experiences?
Inscribing the concepts of being wheelchair bound onto the viewer is possible, but for a truly relational experience he/she would need at the least an immersive CAVE-like environment. Depending on the cognitive skills of the engaged individual, the ability to incorporate this altered life style is possible.
We must also understand the impact of the technology itself on individuals. Just as we have come to rely on computer technology as an extension of ourselves, we might also come to rely on memory-aiding wearable computers to capture events for us, allowing us to play them back at will. In other words, we might become even more cognitively lazy than we already are. Gordon Bell has noted his tendency to neglect life in real-time as he comes to rely more and more on his digital store, essentially using his record as a second self.
Some individuals might become consumed by virtually re-living past moments, ignoring the present much as some have become increasingly reliant on the instant access to information that the Internet provides to satisfy various needs.
On a societal level, there is the potential for misunderstandings brought about bythe sharing of immersive virtual memories. One would initially think that being able to ‘live’ another’s experience would serve to increase understanding. However, immersive memory archives are likely to lead to misunderstandings and a biased interpretation of the archival record when viewed by non-participating individuals, despite an immersive physical experience of the event.
As Wolfgang Ernst notes: an “archive does not tell stories,” so the responsibility of conveying the overall value of the memory to the observer falls to the person generating the record of the event. Regardless of the number of times an outsider views the memory, if the information lacks proper narration and
interpretations are left to the viewer, misinterpretation of the event is a likely result.
Finally, we must investigate the implications of a balance in power created by combinations of surveillance and sousveillance systems. When Hayles examines the reports on management styles in various corporations, she points out that often the surveillance systems in place are designed to provide a thorough examination of the workers through a panoptic-like system. In one case study described by Zuboff, the observers in upper-management stopped directly engaging the workers once a menu-like list of tasks that could be performed was established (i.e., an archive of events).
Due to the physical separation now afforded by the task list, upper-managementissued commands from the menu of options without engaging the workers actually conducting the work. This type of distanced interpretation does not allow for an immersive understanding of the workers actual situation. Surveillance alone does not tell the entire story, one reason for the increasing use of ethnography and participant observer techniques in task analysis.
The understanding of an individual’s experience brought about by experiencing it firsthand suggests that a combination of surveillance and sousveillance techniques may increase the possibility for understanding between groups of differing power status.
Limitations, Concerns, and the Sharing of Virtual Memories
Like any other technology, a Pensieve-like system for capturing, storing,
recalling, and sharing memories is not without limitations. Virtual environments still have serious limitations, especially in terms of cost, maintenance, and the technical knowledge needed to run them. To date, they are far from user-friendly. Of particular concern are the possible physical side-effects of virtual environment use, which can include episodes of nausea and dizziness (‘cybersickness’), headaches, and aftereffects from prolonged use.
Given that we are able to successfully overcome these issues with virtual
environments and there is a viable platform with which to re-play memory data, it is likely that users will not only want to re-play their own recorded memories but will also want to share them with others. Steve Mann and other lifeloggers have already taken this step by sharing their recorded video and audio logs online through the Web.
We could imagine a YouTube-like memory sharing website where users can upload all the stored data associated with a recorded memory in a standardized format that can be re-played and experienced in an immersive virtual environment by anyone else in the world.
Obviously, there are serious privacy issues associated with the recording andsharing of memories. Some individuals may not want to have a complete record of their everyday happenings, for one reason or another.
If lifelogging becomes common enough, gaps in a recording may be taken as evidence of wrongdoing or some other unsavory activity occurring. Others may object if their recordings cannot be edited, much as one edits a physical record of memory such as a scrapbook. Selective forgetting must always be possible.
When sharing memory records there must be a way to ensure that those who happen to be associated with someone’s memory recording (e.g., someone in the room at the same time the person recording a memory was) consent to being featured in that memory if it is shared. We must also ensure the integrity of shared memories so that they cannot be altered or tampered with so as to distort the original experience.
Finally, there are other pressing social issues involved with the distribution of memories, such as whether they can be entered as viable evidence in a court of law and how the rights of individuals within recorded memories are protected.
The lifelogging phenomenon exemplifies users’ desires to capture their everydayexperience, be able to play it back, and share it with others. The incorporation ofwearable computing technology to capture more of our everyday experience makes possible the re-playing of our memories in a more immersive way, such as through a virtual environment. Being able to re-live experiences not only from our own, but from other’s perspectives carries numerous implications for our understanding of the self and of others. As we transition from our current form to that of a cyborg, it is likely that our understanding of a memory will shift, perhaps toward more of a collective understanding of an event rather than a single-perspective understanding.
More of our lives will be visible and sharable with others than ever before. It is up to each of us to determine whether the potential increases in understanding and benefits to ourselves and others brought about by a Pensieve-like system outweigh the potential risks and threats to privacy.
As Dumbledore says about the Pensieve: “Curiosity is not a sin [Harry], but weshould exercise caution with our curiosity…yes, indeed.”
- Digital traces
- Gordon Bell
- Quantified Self
- Steve Mann
- Wearable Computing
- Cathal Gurrin