ņů÷¶ ”∆Ķ≥…»ňapp T researchers develop video camera that captures 'huge range of timescales'
Computational imaging researchers at the ņů÷¶ ”∆Ķ≥…»ňapp have built a camera that can capture everything from light bouncing off a mirror to a ball bouncing on a basketball court ‚Äď all in a single take.
Dubbed by one researcher as a ‚Äúmicroscope for time,‚ÄĚ could lead to improvements in everything from medical imaging to the LIDAR (Light Detection and Ranging) technologies used in mobile phones and self-driving cars.
‚ÄúOur work introduces a unique camera capable of capturing videos that can be replayed at speeds ranging from the standard 30 frames per second to hundreds of billions of frames per second,‚ÄĚ saysŐżSotiris Nousias, a post-doctoral researcher who is working with Kyros Kutulakos, a professor of computer science in the Faculty of Arts & Science.Őż
‚ÄúWith this technology, you no longer need to predetermine the speed at which you want to capture the world.‚ÄĚ
The research by members of the Toronto Computational Imaging GroupŐż‚Äď including computer science PhD studentŐżMian Wei, electrical and computer engineering alumnus Rahul Gulve ≤Ļ≤‘ĽŚŐżDavid Lindell, an assistant professor of computer scienceŐż‚ÄďŐżwas recently presented at theŐż2023 International Conference on Computer Vision, where it received one of twoŐżbest paper awards.
‚ÄúOur camera is fast enough to even let us see light moving through a scene,‚ÄĚ Wei says. ‚ÄúThis type of slow and fast imaging where we can capture video across such a huge range of timescales has never been done before.‚ÄĚ
Wei compares the approach to combining the various video modes on a smartphone: slow motion, normal video and time lapse.
‚ÄúIn our case, our camera has just one recording mode that records all timescales simultaneously and then, afterwards, we can decide [what we want to view],‚ÄĚ he says. ‚ÄúWe can see every single timescale because if something‚Äôs moving too fast, we can zoom in to that timescale, if something‚Äôs moving too slow, we can zoom out and see that, too.‚ÄĚ
While conventional high-speed cameras can record video up to around one million frames per second without a dedicated light source ‚Äď fast enough to capture videos of a speeding bullet ‚Äď they are too slow to capture the movement of light.
The researchers say capturing anŐżimage much faster than a speeding bullet without a synchronized light source such as strobe light or a laser creates a challenge because very little light is collected during such a short exposure periodŐż‚Äď and a significant amount of light is needed to form an image.
To overcome these issues, the research team used a special type of ultra-sensitive sensor called a free-running single-photon avalanche diode (SPAD). The sensor operates by time-stamping the arrival of individual photons (particles of light) with precision down to trillionths of a second. To recover a video, they use a computational algorithm that analyzes when the photons arrive and estimates how much light is incident on the sensor for any given instant in time, regardless of whether that light came from room lights, sunlight or from lasers operating nearby.
Reconstructing and playing back a video is a matter of retrieving the light levels corresponding to each video frame.
The researchers refer to the novel approach as ‚Äúpassive ultra-wideband imaging‚ÄĚ that enables post-capture refocusing in time ‚Äď from transient to everyday timescales.
‚ÄúYou don‚Äôt need to know what happens in the scene, or what light sources are there. You can record information and you can refocus on whatever phenomena or whatever timescale you want,‚ÄĚ Nousias explains.
Using an experimental setup that employed multiple external light sources and a spinning fan, the team demonstrated their method‚Äôs ability to allow for post-capture timescale selection. In their demonstration, they used photon timestamp data captured by a free-running SPAD camera to play back video of a rapidly spinning fan at both 1,000 frames per second and 250 billion frames per second.
The technology could have myriad applications.
‚ÄúIn biomedical imaging, you might want to be able to image across a huge range of timescales at which biological phenomena occur. For example, protein folding and binding happen across timescales from nanoseconds to milliseconds,‚ÄĚ says Lindell. ‚ÄúIn other applications, like mechanical inspection, maybe you‚Äôd like to image an engine or a turbine for many minutes or hours and then after collecting the data, zoom in to a timescale where an unexpected anomaly or failure occurs.‚ÄĚ
In the case of self-driving cars, each vehicle may use an active imaging system like LIDAR to emit light pulses that can create potential interference with other systems on the road. However, the researchers say their technology could ‚Äúturn this problem on its head‚ÄĚ by capturing and using ambient photons. For example, they say it might be possible to create universal light sources that any car, robot or smartphone can use without requiring the explicit synchronization that is needed by today‚Äôs LIDAR systems.
Astronomy is another area that could see potential imaging advancementsŐż‚Äď including when it comes to studying phenomena such as fast radio bursts.
‚ÄúCurrently, there is a strong focus on pinpointing the optical counterparts of these fast radio bursts more precisely in their host galaxies. This is where the techniques developed by this group, particularly their innovative use of SPAD cameras, can be valuable,‚ÄĚ saysŐżSuresh Sivanandam, interim director of the Dunlap Institute for Astronomy & Astrophysics and associate professor at the David A. Dunlap department of astronomy and astrophysics.
The researchers say that while the sensors that have the capability to timestamp photons already exist ‚Äď it‚Äôs an emerging technology that‚Äôs been deployed on iPhones in their LIDAR and their proximity sensorŐż‚Äď no one has used the photon timestamps in this way to enable this type of ultra-wideband, single-photon imaging.
‚ÄúWhat we provide is a microscope for time,‚ÄĚŐżKutulakos says. ‚ÄúSo, with the camera you record everything that happened and then you can go in and observe the world at imperceptibly fast timescales.
‚ÄúSuch capability can open up a new understanding of nature and the world around us."