Tiago H. Falk

If now you're a machine and your smartphone is aware that you have now left your office and you're in the cafeteria and it's noisy, it can adapt itself so that the enhancement part kicks in or it boosts the volume up so that you can hear better. If it detects that you have gone from an indoor to an outdoor setting, it will detect the light changes. We don't just throw away the noise. We try to mine as much information from the noise as possible, as well.

Thorstensen: When you're talking about signals that you're receiving, it sounds like you're talking about signals from the person's environment but it could also be some bio signals from the person's body. Is that correct?

Can you use these devices? Can you use cameras to try to infer what the person is trying to say? Maybe from a mouth opening, if they open their mouth, that becomes their control signal so if you can detect from a camera them opening their mouths. If you can have them hum, if they're able to make sounds, they're not understandable sounds but they can make some grunts or they could hum, could you transfer that into a control signal? We've used speech coding technologies to actually measure hums and different properties of the hums.

In the video now we can now drive a wheelchair based on humming. If the pitch of your hum goes up, the wheelchair goes forward. If it goes down, the wheelchair goes backward. If you keep a low pitch for a persistent amount of time, the wheelchair turns left. If you keep it high, the wheelchair goes right. Now you're using multimedia technologies together with a bio signal sensor that's measuring vibrations in the vocal chords and you put it in an application, which is driving the wheelchair. Now it allows a person that was in bed completely dependent on their caregivers, now it gives them some sort of freedom to move about. We've tried this technology to turn on and off lights, put the volume on the TV up and down, so it gives them an independence. That's one application.

Another that we're working on now is can you bring video games now into the equation and virtual reality and develop video games that are specifically targeting parts of the brain that we know are affected by Alzheimer's disease. We're looking at developing virtual reality games combined with brain monitoring and seeing if we can use that to help diagnose Alzheimer's disease. Today a lot of the tools that are used are very expensive, nueroimaging technology such as MRI, PET scans and things like that. We're seeing can we bring that down to a device that costs maybe a couple of hundred of dollars together with a virtual reality headset and maybe bring the cost own to something less than a thousand dollars that could be use in low and medium income countries and things like that. Those are two that are more health care applications.

Falk: It was very rewarding and invigorating. My postdoc was at the University of Toronto at the Holland Bloorview Kids Rehabilitation Hospital. It was in the department which was aimed for developing technologies for kids that had disabilities with which nothing that was in the markets would work for them. Of all the assisted devices that were available in the market, they could not use for whatever number of reasons. These had to be customized to a particular child or to a particular youth. A lot of the kids that had come through, there was no notice that they had this type of ... They had very limited control of their arms or their limbs, a lot of spastic movement in the head, but they still had somewhat intact their vocalizations. It wasn't something that they could actually say words but they could make sounds. What was unique about this is that the sounds and the hums cause a periodic vibration of the vocal chords. This is the premises of speech coding. My PhD and my early years had been on speech coding.

Once I arrived, people were struggling with this. Just by bringing this multi-disciplinary view, this view of speech that nobody had in this biomedical department, quickly we jumped into that idea. Today, it's one to the technologies that ... I just spoke to Professor Tom Chau yesterday, who is the director of the lab. He said there's roughly about three dozen people now that are actively using this device. The first child that we tried it on apparently is so proficient today that he has programmed a device to play video games so he can actually play mortal combat video games just using hums. I haven't seen it but I was told he plays just like any other kid with two hands a controller plays. He does it with hums on the wheelchair. I'm looking forward to maybe seeing a video of that soon.

Thorstensen: That sounds really interesting. That would be interesting to see.

Falk: Bringing that capability of giving somebody the independence and a lot of the kids have gone on to write their autobiographies. If you attach that to what's called a non-screen flashing keyboard ... Basically, it flashes the rows of the keyboard versus the columns and once they perform a hum, it stops it at that particular letter. That gives them also the ability to write so they can write; they can have pre-programmed messages stored on their device. If they want to go to the bathroom, they can just play that message. They can pick what voice, so you can have a synthetic voice that tells the messages out loud so they can select what voices they want. They can have character voices. It was a very fun. It was a very interesting period in my career. I miss it a lot so hopefully, we'll get back to working with kids soon.

Thorstensen: I also saw that you have used this research for some mental health applications, too. I saw the one about the remotely monitoring someone's state to see if they are seriously depressed. Could you explain how that one works?

Falk: That started as part of a challenge that's run every year by a multimedia organization. What they do is they release audio visual content from individuals that have been rated by psychologists as having different depression levels. The challenge is for individuals to come and develop technologies that will bring together the audio aspect and the vision aspect and try to predict what that person's particular mental state is or depression level is.

We participated a couple of years ago in this challenge and we got some very interesting results. We've been trying to take this technology forward. We've been finding that the audio aspect contributed more than the vision part. We've been looking at what can we infer from the speech signal. What is the unique pattern that discriminates a depressed versus a non-depressed person? What is so unique about their vocal patterns and can we extract that information? Can we analyze their speech signal and extract that information?

That's particularly useful if you're, for example, on a suicidal hotline where you have somebody that's calling in. You don't want somebody that's at high risk of committing suicide to be received with a "Please hold. We are receiving a high number of calls now." If you're able, just from that initial call, that initial five seconds of the person saying something, to say, "Oh, this person is very high risk. Let's give him or her extreme priority" versus "Oh, this person is okay. He can hang on for a few seconds," it can allow for that kind of adaptation also within these hotlines.

Thorstensen: Also, could you talk more about your Alzheimer's disease diagnosis and how that works?

Falk: We started with looking at, again, different ways of analyzing signals. Our expertise here is in signal processing and by signal processing, I mean can you clean up the signals? Can you extract information from the signals? What are new ways of looking at the signals? Can you take it into a different dimension where, in that dimension, you can easily see what is signal and what is noise? Can you infer specific details about the signals? That's where a lot of our research comes in as to how do you play with the signals; how do you extract this information from the signals.

Thorstensen: You've been noted by the nominators who put your name up for this award as successfully leading interdisciplinary research teams. Do you have any advice about what it takes to successfully lead an interdisciplinary research team?

Falk: You have to be very open-minded, I think, is the main thing. Different people will come with very different ideas and if you're not open to those new ideas, I think you'll never get anywhere. I think I learned a lot of that from my Postdoc. Most of my life had been on multimedia, speech coding, speech processing and I arrived at my Postdoc to do brain computer interfaces, having never seen a brain signal in my life. The first thing I realized when I arrived was that biomedical signal processing eight years ago was using tools that, in speech processing, were being used fifty years ago. I was able to look at the fifty years of speech processing and how it developed over those fifty years. The new tools that were being used, I started playing around and seeing if they could be applied into biomedical signals. In many, many cases they gave results much better than what people were using back then.

Thorstensen: What type of backgrounds do people in your lab have?

Falk: We have physicists; we have computer scientists; we have engineers, biomedical engineer, electrical engineers. Every now and then we get somebody in the arts. They will come in to help with user design and to develop some of the interfaces for the games. Again, from computer science to engineering to physics. My lab technician is in astrophysics. He's doing his PhD in astrophysics.

Thorstensen: Out of all the areas that you work with human and machine interfaces, do you have a favorite application that you've worked on?

Falk: That's a good question. I don't want to put any favorites so students will get upset.

Thorstensen: Okay.

Falk: We try to work very closely with industry as well. I find that it's very disappointing when you have a technology that's developed that has great potential but it just sits on a shelf somewhere and doesn't get used. A lot of our tools are developed in combination with industry. It fills a gap that's there. A lot of the stuff we build has a practical application. It starts off already with that mentality that something will come out of it. Individuals with disabilities will benefit; individuals with mental health will benefit. There's always that end goal that we have. There's always that challenge that we have to meet or that expectations or those outcomes that have to be achieved.

I would say personally, for me, for example, the Hummer would be something that would be very high up there because I see so many kids that were touched by that. It brought new dynamics to the family. Now parents can talk to the kids in many cases. There was a case of an individual who was eighteen. He had never spoken. The first time we tried them out and asked him to type something, the first thing he typed was, "Mom, I love you." That was the first time in eighteen years that that child's mom could hear him say, "Mom, I love you." Something like that is priceless. I think that would be a great example of something that really touches.

Thorstensen: That is another thing that one of your nominators brought up, that you are able to find solutions to problems rather than coming up with solutions and then looking for a problem to apply it to. Do you think it is because of the way that you work with industry or is that just the way that you've always thought about your research?

Falk: I think the way that I work with industry is more of a consequence. I think it's the way I approach research more in being open and thinking outside the box. Why throw this information away if we can use it for something? Instead of just filtering out the noise, let's gather as much information from that as we can so let's explore, let's analyze, let's see what's there, let's play around.