Using Video Microscopy to Characterize Micromechanics of Biological and Man-Made Micromachines (invited)

Dennis M. Freeman and C. Quentin Davis

Presented at the Solid-State Sensor and Actuator Workshop

Hilton Head Island, SC, June 1996.

Part II: Computer Microvision:
Video Microscopy + Computer Vision

I'll describe a computer microvision system for measuring sound-induced motions of inner ear structures.
still picture (18K gif)
We mount the inner ear of a lizard in an experimental chamber, so that we can stimulate it with sound while observing its motion with a microscope. We are interested in motions at audio frequencies. Therefore we use stroboscopic illumination and we record the resulting images using a CCD camera.
still picture (72K gif)
This slide shows an image taken with our system. The drawing represents a cross sectional view of the cochlea. The cluster of hair cells rests on the basilar membrane, and are surmounted by a gelatinous tectorial membrane. We view the cochlea in the direction indicated by the arrow, and the focal plane of the microscope is indicated by the dashed line. The left panel shows the image seen at this focal plane, which is through the top of the tectorial membrane.
still picture (85K gif)
By changing the focal plane of the microscope, different parts of the cochlea become visible, a property called optical sectioning. At this plane of section, one can see the tips of several hair bundles. The edge of the tectorial membrane can also be seen, especially at the right margin of the image.
still picture (97K gif)
At this plane of section, the bases of several hair bundles can be seen, as well as the right boundary of the tectorial membrane. In general, we record sufficiently many images to characterize all of the structures in the inner ear, as shown in the attached video.
still picture (104K gif)
We are interested in the motions that result in response audio frequency stimulation. Therefore we use stroboscopic illumination to obtain a sequence of images, each locked to a different stimulus phase. The vertical arrow in the inset indicates the phase at which the corresponding image was acquired. The attached video shows a slow motion video sequence of the motions of the bases of several hair cells. The sequence consists of images acquired at 8 phases of the stimulus period.
still picture (91K gif)
We can repeat these measurements at other planes of focus. This image and the associated video show the tips of several hair bundles. Thus, by combining stroboscopic illumination with optical sectioning, we obtain information about the motions of all structures in the field of view.
still picture (9K gif)
Even after magnification by a light microscope, the sound-induced motions of inner ear structures are small -- much smaller than the pixel spacing of a modern CCD camera. Nevertheless, images obtained during subpixel displacements still contain information about motion, which is illustrated in this figure and in the associated video. The squares represent the pixels of our camera. The circle represents a target moving horizontally. The bar graphs at the bottom of the figure indicate the brightnesses of the row of pixels indicated by the arrow. Even though the circle translates less than a pixel, its motion modulates pixel brightnesses. We use algorithms from computer vision to make quantitative estimates of the displacement of the circle from the changes in pixel brightness.
still picture (27K gif)
The computer vision algorithm is easy to understand by analyzing motions of a 1D wave of brightness, as illustrated here and in the associated video.
still picture (5K gif)
Horizontal motions of the wave modulates the brightness of each pixel in the image. By watching the brightness changes at a particular pixel, we can estimate the time-derivative of brightness at that point, as illustrated here and in the associated video.
still picture (5K gif)
But we are interested in motion. And any particular temporal rate of change could correspond to either a large or a small motion depending on the rate at which brightness changes spatially across the image. We therefore estimate the spatial derivatives of brightness by comparing the brightnesses of neighboring pixels.
still picture (5K gif)
The ratio of the temporal and spatial derivatives then provides an estimate of the velocity of the target. We repeat this estimation procedure for each pixel in the region of interest and combine the results using a least squares technique.
still picture (10K gif)
To summarize, our computer vision system consists of a video microscope, a strobe lamp, a CCD camera, and motion estimation algorithms from computer vision. To test the system, we developed a microscopic target whose motions could be measured by an independent method. We then used the computer microvision system to determine the motion and compared the results from the video images with the independent method.

We found systematic differences between the two measurements. To understand the origin of these differences, we performed simulations. We simulated the target, and we simulated motions of the target by software displacements the simulated target. We simulated the predominant sources of measurement noise and we simulated camera defects. The results of the simulations matched the measurements very well: both the simulations and measurements showed the same systematic errors.

It is easy to understand how noise or camera defects could introduce errors. However, the systematic motion measurement errors remained even when noise and camera defects were removed from the simulations. It turns out that the systematic errors are an intrinsic property of the motion detection algorithms.

still picture (27K gif)
The source of the errors is easy to understand by analyzing results for a 1D sinusoidal brightness pattern shown here and in the associated video.
still picture (19K gif)
The motion detection problem for the sine wave images can be stated as follows: use the measured brightness patterns at two times (t1 and t2) to estimate the shift (d) between the images. The sinusoidal case is simple enough to determine an analytic expression for the estimate (d hat) from the computer vision algorithm. Ideally, the estimate (d hat) should equal the displacement (d), as shown by the straight line. But this analysis reveals systematic differences, which we refer to as bias, as indicated by the dashed lines. Furthermore, the bias depends on the image. Here we've indicated the biases the result for waves with three different spatial frequencies.

These systematic errors limit our ability to measure subpixel motions. We have therefore developed new computer vision algorithms to reduce the magnitudes of these errors.

still picture (17K gif)
These lines show results for the new algorithm. In each case, the bias of the estimates from the new algorithm is much smaller than that in the original algorithm.
still picture (8K gif)
To test the new algorithms, we used the same technique as before: we imaged a target whose motion could be determined with an independent method. The results demonstrated the improvements suggested by our sine wave analysis. Measurement errors are on the order of 14 nm: less than 3 percent of the wavelength of the light used to obtain the image.

To understand the significance of this small measurement error, it is important to remember that the system is not a scanning near-field system, nor does it use an interference method. The underlying images are obtained using brightfield microscopy. And, as we will see in the upcoming sections, brightfield microscopy has important advantages. Most importantly, brightfield microscopy allows simultaneous 3D motion estimates from all structures in the field of view.

It's also important to remember that motion resolution depends critically on the optical contrast of the target. In these tests, the target was chosen to approximate our biological targets, which have very low optical contrast. As we will see later in the talk, considerably more precise estimates result for silicon structures.

Let me now move on to show how the computer microvision system can be used to obtain insights into inner ear micromechanics.

Application to Hearing