Research Projects

Here is the list of research projects I have been working with or I am currently working on ...................

It is a National Institutes of Health-supported interdisciplinary project, involving researchers from the fields of Communication, Cell Neurobiology, Computer Science, and Physical Therapy. The purpose of the project is to develop virtual therapeutic environments that include different levels of haptic sensory feedback for post-stroke rehabilitation. Various applications have been created within virtual environments using the PHANToM and CyberGrasp (haptic devices), ranging from everyday functional tasks to game-like activities designed to motivate patients and to maximize therapeutic movement with cortical reorganization goals in mind. We finished 12 training sessions for six participants post-stroke with different levels of motor severity over three-week period in April 2006. Methodologies are developed to diagnose patient's current status and evaluate his/her progress of the function of upper extremity via clinical test data. We are about to start clinical tests in Phase 2.

Faculty Mentor: Carolee J. Winstein, Albert Rizzo, Margaret McLaughlin, et al.

Other Student Participants: Jill Stewart, Lei Li, Hyunjin Yoon, Youn Bo Jung, Maureen Whitford, Shuya Chen, et al.

We propose to address a pressing need for novel, pervasive and easily deployable information technology applications in healthcare. Currently there is a shortage of qualified personnel in patient care, and the annual cost increase is outpacing inflation. We plan to address the care segment where outpatient treatment can be a cost-effective alternative. Specifically, we propose to design a flexible platform that allows application builders to rapidly design, create and deploy applications that require the transmission of delay-sensitive media streams such as audio, video, and haptic data. As an initial example we will apply our framework to tele-rehabilitation where a therapist remotely monitors the exercise regimen and progress of a patient who, for example, previously suffered from a stroke. Clinical tests will be conducted in summer 2007.

Faculty Mentor: Carolee J. Winstein, Margaret McLaughlin, Albert Rizzo, Bonnie L. Kennedy, et al.

Other Student Participants: Lei Li, Shuya Chen, Chien-Yen Chang, Youn Bo Jung, Hyunjin Yoon, et al.

Home-based systems need to be affordable and easy to deploy and maintain, while still providing the interactional fidelity required to produce meaningful motor rehabilitation activity that can foster transfer of training to real world challenges. As well, a home-based system requires that the necessary equipment to deliver good 3D interaction does not actually interfere with the motor activity that is to be retrained. For example, “wired” devices may limit the patient's motion zone causing an unnatural interaction with the virtual environment that could limit therapeutic progress. Thus, while there are many benefits that could be accrued with home-based methodologies, there are considerable challenges for creating low-cost, yet effective 3DUI techniques and tools in this clinical domain. We develope an optical tracking device to meet the challenges mentioned. It is designed to meet the therapist’s needs for a usable system for delivering VR tasks for rehabilitation, and the patient's needs for home-based use. The features are: 1) low cost; 2) easy installation and simple operation; 3) wireless operation. It consists of one or two inexpensive color webcams, battery-powered light-emitting diodes (LEDs), and tracking software that extracts spatial information with multiple degrees of freedom. Any webcam with a USB interface and a resolution of 320x240 or higher is adequate for this LED tracking system. The use of a single camera or dual cameras is determined by the need for more degrees of freedom depending on the task requirements. The tracking algorithm applies the mean shift method that tracks multiple targets with different colors in real time. We use LEDs as colored targets for tracking; the algorithm returns the centrallocation of each target in the image frame. To extract exact spatial information, calibrated dual cameras are needed. If only spatial information relative to the virtual environment is required, depth information can be extracted via the change of disparity from dual cameras while vertical and horizontal information is extracted from the change of central location on image frame.

As a method for assessing spatial ability, the Mental Rotation Test is especially attractive for this purpose as there is a vast literature on human mental rotation ability that 3DUI researchers can leverage for interpretation of experimental results. Mental Rotation (MR) is a well-studied visuospatial variable, which can be described as a dynamic imagery process that involves “turning something over in one’s mind” [Shepard and Metzler, 1971]. The initial MR investigations began more than 30 years ago with the work of Shepard and Metzler (1971) who tachistoscopically presented pairs of two-dimensional perspective drawings to subjects and required them to make judgments as to whether the 3D objects they portrayed, were the same or differen. A near perfect linear relationship was found between the amount of angle rotation difference between the pairs of objects, and the reaction time to decide whether or not the objects were the same or different. Since precise mathematical relationships between hypothesized mental representations and behavioral performance are relatively rare, MR has been the focus of considerable research. Everyday life situations rely on this ability to use imagery to turn over or manipulate objects mentally. These include automobile driving judgments, organizing items in limited storage space, using a map, sports activities, and many other situations where one needs to visualize the movement and ultimate location of physical objects in 3D space. High-level mathematics performance has also been linked, in large part, to MR ability [Casey et al, 1995]. Indeed, it was noted that world-renowned physicist, Stephen Hawking, “…translates mathematics into geometry, and turns around geometrical shapes in his head.” [Cole, 1998]. The MRT, a paper and pencil test, uses two-dimensional image stimuli that portray three-dimensional objects and requires mental processing of the stimuli without any motoric involvement. This test is also noteworthy in that it produces the most consistent and sizeable sex differences, in favor of males, in the cognitive literature [Voyer et al, 1995].

This project has implemented the paper and pencil MRT in a 3D virtual environment and shown that “acting” out the rotation using 3D translation/rotation (instead of mentally rotation and choosing from multiple answers) also exhibited the same discriminating effect of the original test. There is a growing effort in using the 3D virtual environment as a platform for evaluating human spatial abilities in many different ways [Kauffman][Vorderer], and in this sense the UI evaluation is deeply tied to the user’s spatial ability.

One of the early observable signs of Dementia of the Alzheimer’s Type (DAT) can be seen in difficulty finding and navigating around familiar and unfamiliar environments (wayfinding). Visuospatial (VS) perception, the ability to process and interpret visual information about where one is in space, is a cognitive process often assessed and monitored in clients with DAT since VS impairments are thought to contribute to wayfinding difficulties, increased disorientation and a loss of functional independence. Traditional visuospatial paper and pencil tests typically employ 2D representations of 3D objects that must be visualized and mentally processed, rotated or transformed. In rodents, wayfinding and place learning performance is commonly tested using the Morris Water Task, which requires the animal to find their way to a submerged platform in a round pool of “milky” water. However, such wayfinding tests are not practical for use with humans. To address these traditional limitations, two VR environments will be used to assess visuospatial (VS) perception (3D VRVS task) and wayfinding/navigation (VRMWM task) in healthy adults (HA) and adults with mild Dementia of the Alzheimer’s Type (DAT). The VRVS task delivers 3D stimuli, viewed on a PC using stereoscopic shutter glasses and allows manipulation of virtual objects using a tracking/input device. One advantage over existing 2D paper-and-pencil tests is the ability to view and manipulate objects in 3D using a hand held input device that allows for more precise measurement of performance. Pilot tests with this VR system on healthy young and aged adults produced no adverse side effects and results indicated later improvements on 2D paper and pencil mental rotation following brief practice with the 3D VR tests (Rizzo et al 2001). A VR version of the Morris Water Maze (VRMWM), designed by researchers at the Olin Neuropsychiatry Research Center, allows the user to navigate through a virtual water pool on a PC screen to find a hidden platform using a joystick. Precise measurements of time and distance traveled can be recorded and previous research has demonstrated that older adults take longer time and cover more distance when engaged in finding the platform in the VE than younger adults (Astur et al 1998). The aim of the current research is to compare the 3DVRVS and VRMWM tasks with traditional 2D paper-and-pencil tests as cognitive indicators of the early stages of DAT. From January 2007, a total of 30 healthy adults and 30 adults with mild DAT (CDR=1) will be recruited. Each subject will complete the VRVS and VRMWM tasks, nine 2D paper-and-pencil tests and two ‘real-world’ wayfinding/navigation tasks. The VRVS tasks will be presented in monoscopic and stereoscopic format and in passive and interactive formats. The ‘real-world’ wayfinding tasks involve finding a hidden platform in an empty room and finding an unmarked chair in an auditorium from different starting points. Usability issues and preliminary data analyses (regression) comparing results of VRVS, VRMWM, paper-and-pencil and real-world navigation tasks between HA and DAT groups will be presented for the sample collected from January 2007 to May 2007.

Funder: Alzheimer’s Association ETAC Grant (ETAC = Every Day Technologies for Alzheimer’s Care)

Hemispatial neglect, a failure to attend to and/or act on objects on the side of space opposite a brain lesion, is a common and persistent consequence of stroke, who have suffered a right hemisphere cerebral vascular accident (RCVA). One of the major aim of this project is to perform clinical trials assessing the short-term and long-term efficacy of two VR-based treatments for neglect that induce different types of mis-match between proprioceptive and visual information. One potential outcome of this work includes information about the efficacy, stability, and generalizability of these two different VR-based treatments for neglect.

We present a virtual environment system having a 360 degree panoramic display space. The system is approximately 3.5 m in diameter and is composed of five plasma screens arranged as a pentagon mounted on a supporting framework. Several human viewers may enter the environment and perceive a panoramic view while standing in the the interior.

The system can display live video content captured by a panoramic camera system, or virtual environment content from 3D graphics models that are rendered using Unreal Editor. The display on each screen of the system is driven by a single PC running the Unreal game engine from a specific perspective. Thus, five PCs deliver five different perspectives with an appropriate viewer position and field of view to form the panoramic virtual scene.

User navigation in the system is by means of an optical 3D tracking device composed of dual-webcams and dual-LEDs, and an sixth PC to process the optical tracking data. The 3D spatial position of each LED is located and tracked via a color tracking algorithm and stereo rectification. Position, translation and rotation vectors are computed from this information. The main features of this optical device are that it is illumination independent and wireless.

To interact with the displayed virtual environment, we place the virtual camera so that it moves along the path taken by the optical 3D tracking device. The tracking result is sent to five PCs simultaneously to update the location of the virtual camera and drive the five displays. Data exchange among the PCs is via a TCP/IP communication network. Thus, the virtual camera moves where the user is pointing in a natural and intuitive manner, and the user can interact with or navigate in the 3D world using the optical device.

We have developed a real-time wireless passive device that generates buzz-like passive haptic sensations via an RF transceiver when activated by code-driven VR tasks. The system includes two subsystem modules: module A and module B. The PC communicates/sends commands to the module A through a serial-port (RS232) communication interface, and the module A real-time verifies, converts and passes commands to the module B (user-end module) through a wireless network. After the module B receives commands, it checks the commands again, and actuates the array of vibrating devices with different patterns as user expected. Further system design and system features with associate adaptors and applications are described below.

The development of a virtual “classroom” is specially aimed at the assessment of ADHD. VE technology appears to provide special assets for addressing impairments seen in ADHD that are not available using existing methods. The scenario consists of a standard rectangular classroom environment containing desks, a male or female teacher, a blackboard across the front wall, a side wall with a large window looking out onto a playground and street with vehicles and people, and, on each end of the opposite wall, a pair of doorways through which activity occurs. Within this scenario, children’s attention performance is assessed while a series of typical classroom distracters (that is, immersive audio-supported ambient classroom noise, movement of other pupils, activity occurring outside the window, and so on) are systematically controlled and manipulated within the virtual environment. The child sits at a real desk while seeing a virtual desk in the HMD within the virtual classroom. On-task attention can be measured in terms of performance (reaction time) on a variety of attention challenges that can be adjusted based on the child’s expected age/grade level of performance. For example, on the simpler end of the continuum, the child can be required to press a response button upon the direct instruction of the virtual teacher or whenever the child hears the name of a special target color mentioned by the teacher (focused or selective attention task). Sustained attention can be assessed by manipulating the time demands of the testing. More-complex demands requiring alternating or divided attention can be developed whereby the student needs to respond only when the teacher states the target color in relation to an animal (the brown dog, as opposed to the statement, “I like the color brown”) and only when the word dog is written, or a picture of a dog appears on the blackboard. In addition to attention-driven reaction time performance, behavioral measures that are correlated with distractibility and/or hyperactivity components (head turning, gross motor movement), and impulsive nontask behaviors (time playing with “distracter” items on the desk) can be measured via strategically located trackers. Our frst study is presently comparing ADHD diagnosed children (aged eight to twelve) with a nondiagnosed control group using more-basic attention challenges that are commonly seen on currently available continuous performance tasks and in common classroom tasks (listen-look-respond).

Albert, "Skip", Rizzo (Advisor), Institute of Creative Technologies, University of Southern California, USA

Alexander Sawchuk (Advisor), Department of Electrical Engineering, University of Southern California, USA

Carolee J. Winstein, Department of Biokinesiology & Physical Therapy, University of Southern California, USA

Margaret McLaughlin, Annenberg School for Communication, University of Southern California, USA

Alexandre François (Advisor), Department of Computer Science, University of Southern California, USA

Bonnie L. Kennedy, Department of Occupational Science and Occupational Therapy, University of Southern California, USA

Ann Page, Roski School of Fine Arts, University of Southern California, USA

Richard Palluel-Germain, Laboratory of Psycholgy and NeuroCognition, Pierre-Mendes France University, FR

Belinda Lange, Institute of Creative Technologies, University of Southern California, USA

Thomas Parsons, Institute of Creative Technologies, University of Southern California, USA

Patrice L. (Tamar) Weiss, Department of Occupational Therapy, University of Haifa, Israel

Uri Feintuch, School of Occupational Therapy, Hadassah-Hebrew University, Israel

The University of Southern California does not screen or control the content on this website and thus does not guarantee the accuracy, integrity, or quality of such content. All content on this website is provided by and is the sole responsibility of the person from which such content originated, and such content does not necessarily reflect the opinions of the University administration or the Board of Trustees

Motion Pattern	Pointing	Pinch	Pronation/Supination	Reaching (static)	Reaching (dynamic)
Interface
Graphics Environment
Demo Video	Download	Download	Download	Download	Download

Motion Pattern	Grasping	Water Pouring	Pronation/Supination
*Virtual Environment*
Demo Video	Download	Download	Download

*Degree of Freedom*	3 DOF	4 DOF	5 DOF	Tux Racer
*Virtual Environment*
Demo Video	Download	Download	Download	Download