Bourns College of Engineering

Welcome to the TechHorizons 2009 Test Bed

It is a staple of science fiction speculation.

Computer networks that can monitor large areas, recognize what is ‘seen,' learn from it, and act on the information.

Hal 9000 could do it eight years ago.

But with all of the staggering advances in machine intelligence, pattern recognition and computer networks, it has never really been done.

Electrical Engineering Professors Amit Roy-Chowdhury and Bir Bhanu have assembled a video network of 80 cameras along second floor corridors in Engineering Building Unit II where they plan to teach the system to recognize, remember and respond to dynamic activity in real time.

Professors Roy-Chowdhury and Bhanu have received a series of grants from the National Science Foundation, the Army Research Office, the Office of Naval Research, CISCO, and UC Micro, funding a multi-year effort to bring significant advances to video sensor network technology.

The agencies' interest is justified by the importance of the applications the technology could provide. Intelligent video networks could be a huge benefit for homeland security. Ports, public water supplies, and a host of other potential targets could be monitored with an untiring thoroughness not otherwise possible.

Border security and monitoring of nuclear plans and other sensitive installations would also be significantly enhanced.

Assisted living facilities and neighborhood watch programs would also benefit with new levels of safety and security from such technology.

It is important to note that there will be no involuntary surveillance of people in the building. The researchers will use "actors" and classrooms of students who have volunteered to participate in developing the theoretical and practical basis for "network-centric surveillance" as Roy-Chowdhury calls it.

The engineering challenges facing this ambitious project are vast, according to Roy-Chowdhury. Monitoring activity in large scale camera networks is analogous to weather forecasting. "To monitor the weather you have sensors all over the place and you try to come up with one forecast out of a dynamic picture - one result that will explain the situation," he said.

Researchers have already demonstrated facial recognition by a camera and primitive object tracking between a few cameras. To pull off its goals across 80 cameras, the project will be covering a lot of uncharted territory where video analysis and machine learning overlap, where networking and signal processing intersect, or more accurately where all of the information sciences are pressed into service as one to begin to approximate artificial intelligence.

"We start by basic signal processing," Roy-Chowdhury explained, "data collection at the camera." The project has already developed algorithms which can track subjects as they disappear from one camera's view and reappear in another.

To make the system work, researchers must embed intelligence in each camera with a processing power that has not been achieved before.

Current cameras are capable of detecting a face, but they will need enough processing power to handle more sophisticated algorithms to control the camera in collaboration with other cameras in the network. "It will need the intelligence to look at a subject, calculate how far away it is, how much it has to zoom to get a good picture," Roy-Chowdhury said. "And then it has to tell the other cameras that it is now zooming in, so other cameras in the network better take care of the other areas it is no longer covering."

Network intelligence will also need to be developed. "It might be that there is one particular camera I want to be active to get a best view of a particular subject," Roy-Chowdhury continued. "There has to be an allocation of proper resources to that camera so it can route priority information through nodes to a base station. A clear path to the base station has to be created and it will change according to the application requirements."

"This actually involves quite a bit of processing power," he said.

Game theory and social networking will also be central areas of study throughout the project. "The way game theory came about is people started looking at how animals behave and react, especially when they are hunting in groups. So there is a target and they all want to make sure they are all tracking on it. But the real target is the whole area because they don't want to leave anything out, so all the other cameras need to adjust their parameters."

Intelligent surveillance will be able to learn what is normal in different locations. The goal is to create context-based analysis. For instance, if a camera network can distinguish between someone walking or running, running may be of little notice in a shopping mall. Running in a bank, however, may be unusual enough to warrant additional scrutiny.

Taking that capability several running jumps further, the researchers will also experiment with social networks - can a machine understand the complexity of human interaction and human dynamics?

The researchers will experiment by monitoring movements of students from different labs in controlled conditions two hours a day. They will test whether the camera system can be designed to distinguish students who work together in the same lab or attend the same classes.

Currently, the work involves collaboration with faculty from three college departments. They are: Walid Najjar and Frank Vahid, computer science and engineering professors; Jay Farrell and Ertem Tuncel, professors of electrical engineering; and Sundararajan Venkatadriagaram, professor of mechanical engineering.

Saswati Sarkar, associate professor electrical engineering at the University of Pennsylvania will participate as a network expert as the team works to integrate visual processing and networking.