Image-Based Synthetic Aperture Rendering MIT9904-14 PIs: Prof. Leonard

Image-Based Synthetic Aperture Rendering MIT9904-14 PIs: Prof. Leonard

Image-Based Synthetic Aperture Rendering
MIT9904-14 PIs: Prof. Leonard McMillan (MIT LCS), Prof. Julie Dorsey (MIT LCS), and Dr. Hiroshi Murase (NTT)

Project Goals
The primary focus of our
research effort is to develop
technology to create virtual
experiences that will approach
the fidelity of the real world. In
the future, such technologies
will have a dramatic impact on
the way we work and play.
They will enable new forms of
commerce, bring together
individuals separated by large
distances, and provide us with
new forms of entertainment.

Real-time Acquisition
Ultimately we intend to
create a device for capturing
and processing dynamically
reparameterized light fields
in real-time. We call this
device a synthetic aperture
camera array. It is composed
of a two-dimensional array of
randomly accessible image
sensors that memory-mapped
in the address space of a host
processor. Such a system will
allow images to be synthesized
from a wide range of virtual camera positions in real-time. We plan to support
multiple simultaneous video streams to support stereoscopic display as well as
multiple viewers.

Approach

closest ray

(PCI)
(PCI)

FIFO

Interface
Interface
Solution
Solution

closest camera

Address

FIFO

Light Field Parameterization
Data

desired
ray
camera
surface

reference
images

Low Cost Acquisition

Sensor
Pod - A

B

A

B

C

D

C

D

A

B

A

B

C

D

C

D

focal
surface

We take a non-traditional approach to
computer graphics modeling and rendering, in
which a scene is represented by a collection of
images rather than the geometry and surface
properties used in typical computer graphics.
Essentially, we treat a collection of images as a
database of rays. New views can be constructed
from this database on a ray-by-ray basis by
selecting the closest ray to each desired ray.

Motherboard

A high-level block diagram of our proposed system is shown above. The cameras
host interface will be an industry standard personal computer bus. The camera array
will be constructed from modular sensor units mounted on a common motherboard.

Address

Mux/
Logic

Sensor
Pod - A

Sensor
Pod -A

Sensor
Pod -A

Sensor
Pod -A

Sensor
Pod - B

Sensor
Pod - B

Sensor
Pod - B

Sensor
Pod - B
Mux/
Logic

32 bit

Motherboard Interleaving

Our dynamically reparameterized light field representation allows us to synthesize
images with photographic effects such as variable focus and depth-of-field. Depthof-field effects are created by varying the extent of the reconstruction filters used
on the camera surface.

A variable focal length can be simulated by varying the focal plane used in the
reconstruction process. In a synthetic aperture camera both the aperture and focallength settings can be varied from pixel to pixel. The allows effects that are
impossible with a traditional camera.

We have also prototyped two lowcost devices for acquiring light fields.
We have developed two acquisition
systems for acquiring light fields of
static scenes. The first uses a robotic
XY-platform to move a digital camera.
This system allows us to explore the
trade-offs between camera spacing and
resolution in order to estimate the performance of our camera array. This
system uses a precision image sensor,
precision optics, and a motion platform with a travel distance of approximately one
meter squares. It can acquire a 16 by 16 image light field in under 20 minutes, and
it cost approximately $10,000 US to construct. Our second system is based on an
off-the-shelf flat bed scanner, and an array of plastic lenses. We have modified the
scanner to operate off of battery power so that this system can be taken out into the
field to acquire images. Additional processing is required to correct for shortcoming
in the image sensor and low cost optics. None the less, the system can acquire an 8
by 12 image light field in under 3 minutes, and cost under $100.

Data

The addressing of sensor modules will be interleaved in order to maximize
the communication bandwidth between the image sensors and the host
computer. Each sensor
pod contains a CMOS
Sensor Pod
Data
image sensor, buffer Address
memory, and glue
logic. The multiframe buffer
Control
CMOS Sensor
memory is used for
(on board A/D)
two functions. It
is used to store
information for
Data
noise cancellation,
and it allows the
Address
host to access
Address
FPGA
SDRAM
image rays asynchData
Logic
ronous to the image
scanning process.
This modular
design approach
will allow us to upgrade to higher resolution sensors as they become available.

Display Technology
We have also developed techniques for direct autostereoscopic viewing of our light fields. These methods
are similar to various lenticular techniques for viewing
stereo images. Our synthetic aperture generation approach provides much greater flexibility than tradition optical approaches. In particular
it can overcome many limitations such a focus control and skewed frustums. We have
demonstrated viewers with true parallax (both horizontal and vertical), and variable
controlled focus. Our displays have nearly all of the desired properties of holograms,
yet they are true color and viewable
under normal lights. Furthermore,
the technology is easily adaptable
to the display of dynamic 3-D
images. Currently we are only
limited by the resolution of flat
panel displays.
The image on the left,
when viewed through a
hexagonal lens array,
can be seen as a threedimensional image of a
flower. It can be simultaneously seen by
multiple viewers. It was
computed from a dynamically reparameterized light
field, which allows us to
precisely control the
focus at all viewing
angles. The inset
provides a magnified
view of the image.

Recently Viewed Presentations

  • SEG 2100 Software Design II - University of Western Ontario

    SEG 2100 Software Design II - University of Western Ontario

    Welcome To CS2212 http://www.youtube.com/watch?v=tLa3Wac4O2A "Programming today is a race between software engineers striving to build bigger and better idiot-proof ...
  • Elements of Art

    Elements of Art

    Elements of Art LINE What defines a LINE? A line connects two points and can go on forever. It is made by a pointed tool such as a pen, crayon, marker, paint brush, etc. It can create shapes, texture, and...
  • Interviewing Techniques - communitysolutionsva.org

    Interviewing Techniques - communitysolutionsva.org

    Avoid using Police Terminology. Clarify any slang terms that the victim uses to ensure that you understand what they mean. Mimic terms used by the victim without acting shocked or embarrassed by them.
  • sequences - Mrs. Murphy's Website (Sweeney)

    sequences - Mrs. Murphy's Website (Sweeney)

    Arithmetic Sequences. An arithmetic sequence is a sequence in which each term differs from the pervious one by the same fixed number. Example. 2,5,8,11,14
  • Fyrirsgn Frutiger Bold Meginml Frutiger Roman

    Fyrirsgn Frutiger Bold Meginml Frutiger Roman

    Arial MS PGothic Calibri ヒラギノ角ゴ Pro W3 Frutiger-Bold Frutiger-Roman Office Theme Fyrirsögn Frutiger Bold ...
  • Teaching and Learning - downemanor.ealing.sch.uk

    Teaching and Learning - downemanor.ealing.sch.uk

    Mill Rythe. This modern centre offers high standards of accommodation with extensive grounds and superb facilities. The centre offers spectacular views across ChichesterHarbour, an area of outstanding natural beauty. Schools allowed to introduce own approach to formative assessment
  • Use of Venn Diagrams to find the HCF and LCM

    Use of Venn Diagrams to find the HCF and LCM

    Venn Diagrams to find the HCF and LCM What's It All About? You are going to learn: How to find the HCF or the LCM of two or more values using a Venn Diagram. What skills should you have already?...
  • Bond Dissociations associated with GTP hydrolysis

    Bond Dissociations associated with GTP hydrolysis

    Many-body perturbation theory energies and gradients. Effective core potential energies, gradients, and second derivatives. Coupled cluster [CCSD and CCSD(T)] and configuration interaction energies. Segmented and generally contracted basis sets including the correlation-consistent basis sets.