This commit makes it so CameraIntrinsics is no longer hardcoded
to use the traditional polynomial radial distortion model. Currently
the distortion code has generic logic which is shared between
different distortion models, but had no other models until now.
This moves everything specific to the polynomial radial distortion
to a subclass PolynomialDistortionCameraIntrinsics(), and adds a
new division distortion model suitable for cameras such as the
GoPro which have much stronger distortion due to their fisheye lens.
This also cleans up the internal API of CameraIntrinsics to make
it easier to understand and reduces old C-style code.
New distortion model is available in the Lens panel of MCE.
- Polynomial is the old well-known model
- Division is the new one which s intended to deal better with huge
distortion.
Coefficients of this model works independent from each other
and for division model one probably want to have positive values
to have a barrel distortion.
Based on the patch from Sebastian Koenig, discussed with Jonathan Williamson
https://developer.blender.org/T38172
Also removed redundant modes from clip editor.
Reviewers: brecht, carter2422
Reviewed By: carter2422
CC: sebastian_k, carter2422
Differential Revision: https://developer.blender.org/D293
Summary:
Now it's possible to assign an image to plane tracks
in clip editor. This image is only used for display
in clip editor and this image is being warped into
the plane track rectangle.
Main purpose of this is to get early feedback about
how good image warping matches the footage, before
clip goes to the compositor.
Pretty much straightforward change: just compute
homography from undeformed normalized frame corner
coordinates (unity square) to plane marker corners
and apply this matrix to opengl stack.
Still could improve behavior when perspective
plane transform is degenerate, but that's not so
much critical for now i'd say.
Reviewers: brecht, campbellbarton
Reviewed By: brecht
CC: sebastian_k
Differential Revision: http://developer.blender.org/D57
Added a weight slider to track which defines
how much particular track affects in a final
reconstruction. This weight is for sure
animateable.
Currently it affects on BA step only which in
most cases will work just fine.
The usecase of this slider is to have it set
to 1.0 most of the time where the track is
good, but blend it's weight down to 0 when
tracker looses the track. This will prevent
camera from jump.
Tutorial is to be done by Sebastian.
It was rather confusing from the user usage point
of view and didn't get so much improvement after
new bundle adjuster was added.
In the future we might want to switch resection
to PPnP algorithm, which could also might be a
nice alternative to fallback option.
- Do plane re-evaluation only when transform is actually done.
Before this re-evaluation happened on every mouse move.
- Added a flag "Auto Keyframe" for the plane track, which does:
* If Auto Keyframe is enabled, then every manual edit of the
plane will create a new keyframe at current frame and update
plane motion between current frame and previous/next keyframe.
This now also implies blending detected motion with neighbor
keyframes, so there's no jump happening.
No automatic update on manual point tracks edit will happen.
* If auto Keyframe is disabled, then no keyframes are adding
to the plane and every plane tweak will re-evaluate in on
the whole frame range.
In this case manual tweaks to point tracks and re-tracking
them implies plane re-evaluation.
This commit includes all the changes made for plane tracker
in tomato branch.
Movie clip editor changes:
- Artist might create a plane track out of multiple point
tracks which belongs to the same track (minimum amount of
point tracks is 4, maximum is not actually limited).
When new plane track is added, it's getting "tracked"
across all point tracks, which makes it stick to the same
plane point tracks belong to.
- After plane track was added, it need to be manually adjusted
in a way it covers feature one might to mask/replace.
General transform tools (G, R, S) or sliding corners with
a mouse could be sued for this. Plane corner which
corresponds to left bottom image corner has got X/Y axis
on it (red is for X axis, green for Y).
- Re-adjusting plane corners makes plane to be "re-tracked"
for the frames sequence between current frame and next
and previous keyframes.
- Kayframes might be removed from the plane, using Shit-X
(Marker Delete) operator. However, currently manual
re-adjustment or "re-track" trigger is needed.
Compositor changes:
- Added new node called Plane Track Deform.
- User selects which plane track to use (for this he need
to select movie clip datablock, object and track names).
- Node gets an image input, which need to be warped into
the plane.
- Node outputs:
* Input image warped into the plane.
* Plane, rasterized to a mask.
Masking changes:
- Mask points might be parented to a plane track, which
makes this point deforming in a way as if it belongs
to the tracked plane.
Some video tutorials are available:
- Coder video: http://www.youtube.com/watch?v=vISEwqNHqe4
- Artist video: https://vimeo.com/71727578
This is mine and Keir's holiday code project :)
Implements an automatic keyframe selection algorithm which uses
couple of approaches to find out best keyframes candidates:
- First, slightly modifier Pollefeys's criteria is used, which
limits correspondence ration from 80% to 100%. This allows to
reject keyframe candidate early without doing heavy math in
cases there're not much common features with first keyframe.
- Second step is based on Geometric Robust Information Criteria
(aka GRIC), which checks whether features motion between
candidate keyframes is better defined by homography or
fundamental matrices.
To be a good keyframe candidate, fundamental matrix need to
define motion better than homography (in this case F-GRIC will
be smaller than H-GRIC).
This two criteria are well described in this paper:
http://www.cs.ait.ac.th/~mdailey/papers/Tahir-KeyFrame.pdf
- Final step is based on estimating reconstruction error of
a full-scene solution using candidate keyframes. This part
is based on the following paper:
ftp://ftp.tnt.uni-hannover.de/pub/papers/2004/ECCV2004-TTHBAW.pdf
This step requires reconstruction using candidate keyframes
and obtaining covariance matrix of 3D points positions.
Reconstruction was done pretty much straightforward using
other simple pipeline routines, and for covariance estimation
pseudo-inverse of Hessian is used, which is in this case
(J^T * J)+, where + denotes pseudo-inverse.
Jacobian matrix is estimating using Ceres evaluate API.
This is also crucial to get rid of possible gauge ambiguity,
which is in our case made by zero-ing 7 (by gauge freedoms
number) eigen values in pseudo-inverse.
There're still room for improving and optimizing the code,
but we need some point to start with anyway :)
Thanks to Keir Mierle and Sameer Agarwal who assisted a lot
to make this feature working.
Also removed unneeded image buffer scaling, it was only needed
for "early output" if there was no rotation. That is no longer
supported since it used to pixelate result a lot and interpolation
is always used now.
Saves quite a few of memory and CPU cycles.
Highlight background depending on number of tracks existing on frame.
This is not so much mathematically accurate displaying where things
shall be improved, but it's nice feedback about which frames better
be reviewed.
Bad frames are tracks < 8, highlighted with red.
OK-ish frame are 8 <= tracks < 16, highlighted with yellow.
Could be some artifacts with color region start/end, this is a bit
unclear what exactly expected to be highlighted -- frames are
displayed as dots, but in fact they're quite noticeable segments.
---
svn merge -r54572:54573 ^/branches/soc-2011-tomato
This fixes some "regressions" introduced in rev50781 which lead to much
worse solution in some cases. Now it's possible to bring old behavior back.
Perhaps it's more like temporal solution for time being smarter solution is
found. But finding such a solution isn't so fast, so let's bring manual
control over reprojection usage.
But anyway, imo it's now nice to have a structure which could be used to
pass different settings to the solver.
- Display track's reprojection error in dopesheet
- Make sure track is selected when clicking on dopesheet channel
- Attempt to make headers a bit cleaner without long labels which
doesn't actually make sense.
It was a bit confusing to synchronize settings used in pre-calculated
dopesheet channels which was storing in tracking data with settings
used for display which is in space data.
This was initially done by converting one flags to other and checking
if space's settings matches pre-calculated one, but that had several
issues if two different dopesheet are using different settings:
- Channels would be re-calculated on every redraw for each of spaces
- Dopesheet operators could fail due to the could be using channels
calculated for other space.
That was also quite nasty code checking if requested settings matches
pre-calculated one.
Added option to use Grease Pencil datablock as a mask for pattern
when doing motion tracking. Option could be found in Tracking Settings
panel.
All strokes would be rasterized separately from each other and every
stroke is treating as a closed spline.
Also added option to apply a mask on track preview which is situated
just after B/B/W channel button under track preview.
===========================================
Major list of changes done in tomato branch:
- Add a planar tracking implementation to libmv
This adds a new planar tracking implementation to libmv. The
tracker is based on Ceres[1], the new nonlinear minimizer that
myself and Sameer released from Google as open source. Since
the motion model is more involved, the interface is
different than the RegionTracker interface used previously
in Blender.
The start of a C API in libmv-capi.{cpp,h} is also included.
- Migrate from pat_{min,max} for markers to 4 corners representation
Convert markers in the movie clip editor / 2D tracker from using
pat_min and pat_max notation to using the a more general, 4-corner
representation.
There is still considerable porting work to do; in particular
sliding from preview widget does not work correct for rotated
markers.
All other areas should be ported to new representation:
* Added support of sliding individual corners. LMB slide + Ctrl
would scale the whole pattern
* S would scale the whole marker, S-S would scale pattern only
* Added support of marker's rotation which is currently rotates
only patterns around their centers or all markers around median,
Rotation or other non-translation/scaling transformation of search
area doesn't make sense.
* Track Preview widget would display transformed pattern which
libmv actually operates with.
- "Efficient Second-order Minimization" for the planar tracker
This implements the "Efficient Second-order Minimization"
scheme, as supported by the existing translation tracker.
This increases the amount of per-iteration work, but
decreases the number of iterations required to converge and
also increases the size of the basin of attraction for the
optimization.
- Remove the use of the legacy RegionTracker API from Blender,
and replaces it with the new TrackRegion API. This also
adds several features to the planar tracker in libmv:
* Do a brute-force initialization of tracking similar to "Hybrid"
mode in the stable release, but using all floats. This is slower
but more accurate. It is still necessary to evaluate if the
performance loss is worth it. In particular, this change is
necessary to support high bit depth imagery.
* Add support for masks over the search window. This is a step
towards supporting user-defined tracker masks. The tracker masks
will make it easy for users to make a mask for e.g. a ball.
Not exposed into interface yet/
* Add Pearson product moment correlation coefficient checking (aka
"Correlation" in the UI. This causes tracking failure if the
tracked patch is not linearly related to the template.
* Add support for warping a few points in addition to the supplied
points. This is useful because the tracking code deliberately
does not expose the underlying warp representation. Instead,
warps are specified in an aparametric way via the correspondences.
- Replace the old style tracker configuration panel with the
new planar tracking panel. From a users perspective, this means:
* The old "tracking algorithm" picker is gone. There is only 1
algorithm now. We may revisit this later, but I would much
prefer to have only 1 algorithm. So far no optimization work
has been done so the speed is not there yet.
* There is now a dropdown to select the motion model. Choices:
* Translation
* Translation, rotation
* Translation, scale
* Translation, rotation, scale
* Affine
* Perspective
* The old "Hybrid" mode is gone; instead there is a toggle to
enable or disable translation-only tracker initialization. This
is the equivalent of the hyrbid mode before, but rewritten to work
with the new planar tracking modes.
* The pyramid levels setting is gone. At a future date, the planar
tracker will decide to use pyramids or not automatically. The
pyramid setting was ultimately a mistake; with the brute force
initialization it is unnecessary.
- Add light-normalized tracking
Added the ability to normalize patterns by their average value while
tracking, to make them invariant to global illumination changes.
Additional details could be found at wiki page [2]
[1] http://code.google.com/p/ceres-solver
[2] http://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.64/Motion_Tracker
Originally was needed to reach easy way of defining masks used for tracking
(do eliminate textures which doesn't belong to feature when tracking.
Implemented as alternative to GP datablock for clip and added switch between
per-clip and per-track GP datablocks -- internal limitations of GP doesn't
allow to display all GP datablocks easily. So either you see.edit GP associated
with clip or with track.
GP strokes associated with track are relative to track's position, following
tracks during tracking and could be shared between several tracks.
Masking code presents in libmv and there's rasterizer of GP datablocks for
masks in blender's tracking module, but they still need to be glued together.
Some documentation cound be found at this page:
http://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.64/Motion_Tracker#Grease_Pencil
All operators which changes tracking data now just tags dopsheet as outdated,
actual re-calculaiton of happens only when this information is actually needed
(like on dopesheet draw).
This makes things a bit faster when there's no dopesheet visible in current
screen and also makes it much easier to update dopesheet using dependency
graph.
Also renamed dopesheet_sort_order to dopesheet_sort_method in rna and internal
stuff which makes much more sense and also correlated with naming in
file browser.
Supported sorting by name, longest tracked segment and total tracked frames.
Internally tracks are stored in Tracking datablock, but sort order is
a clip editor space property and sorting happens on clip editor draw.
If there's no dopesheet opened with different sort orders it's not
a problem due to re-sorting wouldn't happen.
Also fixed draw issue of tracked segments introduced in previous commit.
- Displays dopesheet information for selected tracks, and currently does not
support any kind of editing.
- Changed regions to use the whole main region for such views as curves and dopesheet.
This allows to have own panels with tools/properties in this area.
- Active clip is getting synchronized between different clip editor editors in the
same screen, so updating of curve/dopesheet views happens automatically when one
changes current clip in one of this editors.
- Panels in toolbox and properties panels are now separated to rely on current view
mode, but some operators and poll functions still need to be updated.
- Added new screen called "Movie Tracking" where layout is configured to
display timeline, main clip window, curves and dopesheet.
Expose option into interface to use modal solver which currently
supports only tripod motion.
This solver requires two tracks at least to reconstruct motion.
Using more tracks aren't improving solution in general, just adds
instability into solution and slows down things a lot.
Refirement of camera intrinsics is supported by this solver.
To use this solver just activate "Tripod Motion" checkbox in
solver panel.
Curve points of tracks curves now can be selected for X and Y channels
separately and can be moved along Y axis of curve viewer, points currently
can't change frame they belong to. This allows to smooth spikes caused by
unwanted marker jump.
Also fixed some mistakes in cancel transform in cases when locked tracks
were being trying to be moved; fixed incorrect calculation of marker speed
for curve drawing.
This commit extends limit of ID and objects to 64 (it means 63 meaning
characters and 1 for zero-terminator). CustomData layers names are also
extended.
Changed DNA structures and all places where length constants were hardcoded.
All names which are "generating" from ID block should be limited by MAX_ID_NAME-2,
all non-id names now has got own define called MAX_NAME which should be used all
over for non-id names to make further name migration stuff easier.
All name fields in DNA now have comment with constant which corresponds to
hardcoded numeric value which should make it easier to further update this
limits or even switch to non-hardcoded values in DNA.
Special thanks to Campbell who helped figuring out some issues and helped a lot
in finding all cases where hardcoded valued were still used in code.
Both of forwards and backwards compatibility is stored with blender versions newer
than January 5, 2011. Older versions had issue with placing null-terminator to
DNA strings on file load which will lead to some unpredictable behavior or even
crashes.
This commits merges object tracking implementation from tomato branch.
Summarized changes from branch:
- Added list of objects to be tracked. Default there's only one object called
"Camera" which is used for solving camera motion. Other objects can be added
and each of them will have it;s own list of tracks. Only one object can be used
for camera solving at this moment.
- Added new constraint called "Object Tracking" which makes oriented object be
moving in the save way as solved object motion.
- Scene orientation tools can be used for orienting object to bundles.
- Object has got scale to define "depth" in camera space.
- All tools which works with list of tracks or reconstruction data now
gets that lists from active editing object.
- All objects and their tracking data are available via python api.
- Improvements in witness cameras workflow,
- Enable bicybic filtering fir image displayed in track preview
- Option to show grayscale content of track preview
- When some channels are disabled, display exactly the same
content of preview image which is sending to tracker library.
Merged from tomato branch using command:
svn merge -r42382:42383 -r42384:42385 -r42394:42395 \
-r42397:42398 -r42398:42399 -r42406:42407 \
-r42410:42411 -r42417:42418 -r42471:42472 \
^/branches/soc-2011-tomato
Added slider to define scale of object solution which is used to define
"depth" of object relative to camera position. This slider effects on all
"users" of object solution such as bundles display, constrained objects and so.
Added new operator called "Set Solution Scale" to set real scale for object
solution based on real distance between two bundles reconstructed for this object.
New slider and operator can be found on "Object" panel in toolbox when in
reconstruction mode and active tracking object isn't a camera.
This commit implements basis stuff needed for object tracking,
use case isn't perfect now, interface also should be cleaned a bit.
- Added list of objects to be tracked. Default there's only one object called
"Camera" which is used for solving camera motion. Other objects can be added
and each of them will have it;s own list of tracks. Only one object can be used
for camera solving at this moment.
- Added new constraint called "Object Tracking" which makes oriented object be
moving in the save way as solved object motion.
- Scene orientation tools can be used for orienting object to bundles.
- All tools which works with list of tracks or reconstruction data now
gets that lists from active editing object.
- All objects and their tracking data are available via python api.
Comment from Keir's commit:
Add a new hybrid region tracker for motion tracking to libmv, and
add it as an option (under "Hybrid") in the tracking settings. The
region tracker is a combination of brute force tracking for coarse
alignment, then refinement with the ESM/KLT algorithm already in
libmv that gives excellent subpixel precision (typically 1/50'th
of a pixel)
This also adds a new "brute force" region tracker which does a
brute force search through every pixel position in the destination
for the pattern in the first frame. It leverages SSE if available,
similar to the SAD tracker, to do this quickly. Currently it does
some unnecessary conversions to/from floating point that will get
fixed later.
The hybrid tracker glues the two trackers (brute & ESM) together
to get an overall better tracker. The algorithm is simple:
1. Track from frame 1 to frame 2 with the brute force tracker.
This tries every possible pixel position for the pattern from
frame 1 in frame 2. The position with the smallest
sum-of-absolute-differences is chosen. By definition, this
position is only accurate up to 1 pixel or so.
2. Using the result from 1, initialize a track with ESM. This does
a least-squares fit with subpixel precision.
3. If the ESM shift was more than 2 pixels, report failure.
4. If the ESM track shifted less than 2 pixels, then the track is
good and we're done. The rationale here is that if the
refinement stage shifts more than 1 pixel, then the brute force
result likely found some random position that's not a good fit.
svn command used: svn merge -r 42375:42376 -r 42377:42379 ^/branches/soc-2011-tomato
add it as an option (under "Hybrid") in the tracking settings. The
region tracker is a combination of brute force tracking for coarse
alignment, then refinement with the ESM/KLT algorithm already in
libmv that gives excellent subpixel precision (typically 1/50'th
of a pixel)
This also adds a new "brute force" region tracker which does a
brute force search through every pixel position in the destination
for the pattern in the first frame. It leverages SSE if available,
similar to the SAD tracker, to do this quickly. Currently it does
some unnecessary conversions to/from floating point that will get
fixed later.
The hybrid tracker glues the two trackers (brute & ESM) together
to get an overall better tracker. The algorithm is simple:
1. Track from frame 1 to frame 2 with the brute force tracker.
This tries every possible pixel position for the pattern from
frame 1 in frame 2. The position with the smallest
sum-of-absolute-differences is chosen. By definition, this
position is only accurate up to 1 pixel or so.
2. Using the result from 1, initialize a track with ESM. This does
a least-squares fit with subpixel precision.
3. If the ESM shift was more than 2 pixels, report failure.
4. If the ESM track shifted less than 2 pixels, then the track is
good and we're done. The rationale here is that if the
refinement stage shifts more than 1 pixel, then the brute force
result likely found some random position that's not a good fit.
In some cases solving can take a while (especially when refining is used)
and keeping interface locked is a bit annoying. Now camera solver is moved
to job system and interface isn't locking.
Reporting progress isn't really accurate, but trying to make it more linear
can lead to spending more effort on it than having benefit. Also, changing
status in the information line helps to understand that blender isn't hang
up and solving is till working nicely.
Main changes in code:
- libmv_solveReconstruction now accepts additional parameters:
* progress_update_callback - a function which is getting called
from solver algorithm to report progress back to Blender.
* callback_customdata - a user-defined context which is passing
to progress_update_callback so progress can be updated in needed
blender-side data structures.
This parameters are optional.
- Added structure MovieTrackingStats which is placed in MovieTracking
structure. It's supposed to be used for displaying information about
different operations (currently it's only camera solver, but can be
easily used for something else in the future) in clip editor.
This statistics structure is getting allocated for time operator is
working and not saving into .blend file.
- Clip Editor now displays statistics stored in MovieTrackingStats structure
like it's done for rendering.
This commit implements:
- Configurable settings for newly creating tracks
Now it's possible to set tracking algorithm and it's settings for
all newly creating tracks including manual tracks creation and
tracks creation by "Detect Features" operator.
- Moves margin, frames limit and adjust frame inside per-track
settings.
Was request from Francois for this.
- Adjust Frames replaced with menu called Pattern Match where it's
possible to choose between matching pattern from keyframe frame
or from previously tracked frame.
Didn't see somebody used adjust frames values differ from 0 and 1,
and this menu should make things more clear here/
Also replace adjust frames integer slider with menu to choose between matching
pattern from keyframe position of from previous frame. think this settings
wouldn't be so confusing now.
Sergey Sharybin
Campbell Barton
Keir Mierle