This commits merges object tracking implementation from tomato branch.
Summarized changes from branch:
- Added list of objects to be tracked. Default there's only one object called
"Camera" which is used for solving camera motion. Other objects can be added
and each of them will have it;s own list of tracks. Only one object can be used
for camera solving at this moment.
- Added new constraint called "Object Tracking" which makes oriented object be
moving in the save way as solved object motion.
- Scene orientation tools can be used for orienting object to bundles.
- Object has got scale to define "depth" in camera space.
- All tools which works with list of tracks or reconstruction data now
gets that lists from active editing object.
- All objects and their tracking data are available via python api.
- Improvements in witness cameras workflow,
- Bundles selection is now available for object's bundles
- If bundles selection wasn't changed in 3D viewport, fallback to regular
object selection, so objects behind bundles can be selected
- Snap cursor to selection now respects object's bundle selection
- Object and rack name now can be selected from list in constraint settings
- Added preset for tracks used for object tracking
Malloc is getting to be safe on sequence tracking by the jobs system,
so no additional set up are necessary.
This hopefully fixes crash on OSX with OpenMP enabled when compiling by older gcc
This commit implements basis stuff needed for object tracking,
use case isn't perfect now, interface also should be cleaned a bit.
- Added list of objects to be tracked. Default there's only one object called
"Camera" which is used for solving camera motion. Other objects can be added
and each of them will have it;s own list of tracks. Only one object can be used
for camera solving at this moment.
- Added new constraint called "Object Tracking" which makes oriented object be
moving in the save way as solved object motion.
- Scene orientation tools can be used for orienting object to bundles.
- All tools which works with list of tracks or reconstruction data now
gets that lists from active editing object.
- All objects and their tracking data are available via python api.
In some cases solving can take a while (especially when refining is used)
and keeping interface locked is a bit annoying. Now camera solver is moved
to job system and interface isn't locking.
Reporting progress isn't really accurate, but trying to make it more linear
can lead to spending more effort on it than having benefit. Also, changing
status in the information line helps to understand that blender isn't hang
up and solving is till working nicely.
Main changes in code:
- libmv_solveReconstruction now accepts additional parameters:
* progress_update_callback - a function which is getting called
from solver algorithm to report progress back to Blender.
* callback_customdata - a user-defined context which is passing
to progress_update_callback so progress can be updated in needed
blender-side data structures.
This parameters are optional.
- Added structure MovieTrackingStats which is placed in MovieTracking
structure. It's supposed to be used for displaying information about
different operations (currently it's only camera solver, but can be
easily used for something else in the future) in clip editor.
This statistics structure is getting allocated for time operator is
working and not saving into .blend file.
- Clip Editor now displays statistics stored in MovieTrackingStats structure
like it's done for rendering.
This commit implements:
- Configurable settings for newly creating tracks
Now it's possible to set tracking algorithm and it's settings for
all newly creating tracks including manual tracks creation and
tracks creation by "Detect Features" operator.
- Moves margin, frames limit and adjust frame inside per-track
settings.
Was request from Francois for this.
- Adjust Frames replaced with menu called Pattern Match where it's
possible to choose between matching pattern from keyframe frame
or from previously tracked frame.
Didn't see somebody used adjust frames values differ from 0 and 1,
and this menu should make things more clear here/
Currently this stuff is used for 2d tracking job only, but the same
thing would necessary for camera solver job, so moved it into more
general utility stuff.
Reporting progress isn't really accurate, but trying to make it more linear
can lead to spending more effort on it than having benefit. Also, changing
status in the information line helps to understand that blender isn't hang
up and solving is till working nicely.
Main changes in code:
- libmv_solveReconstruction now accepts additional parameters:
* progress_update_callback - a function which is getting called
from solver algorithm to report progress back to Blender.
* callback_customdata - a user-defined context which is passing
to progress_update_callback so progress can be updated in needed
blender-side data structures.
This parameters are optional.
- Added structure MovieTrackingStats which is placed in MovieTracking
structure. It's supposed to be used for displaying information about
different operations (currently it's only camera solver, but can be
easily used for something else in the future) in clip editor.
This statistics structure is getting allocated for time operator is
working and not saving into .blend file.
- Clip Editor now displays statistics stored in MovieTrackingStats structure
like it's done for rendering.
In some cases solving can take a while (especially when refining is used)
and keeping interface locked is a bit annoying. Now camera solver is moved
to job system and interface isn't locking.
Currently, no progress is displaying in the interface and solving can't
be canceled. It'll be implemented separately (need some changes in libmv).
Currently this stuff is used for 2d tracking job only, but the same
thing would necessary for camera solver job, so moved it into more
general utility stuff.
- Add support for refining the camera's intrinsic parameters
during a solve. Currently, refining supports only the following
combinations of intrinsic parameters:
f
f, cx, cy
f, cx, cy, k1, k2
f, k1
f, k1, k2
This is not the same as autocalibration, since the user must
still make a reasonable initial guess about the focal length and
other parameters, whereas true autocalibration would eliminate
the need for the user specify intrinsic parameters at all.
However, the solver works well with only rough guesses for the
focal length, so perhaps full autocalibation is not that
important.
Adding support for the last two combinations, (f, k1) and (f,
k1, k2) required changes to the library libmv depends on for
bundle adjustment, SSBA. These changes should get ported
upstream not just to libmv but to SSBA as well.
- Improved the region of convergence for bundle adjustment by
increasing the number of Levenberg-Marquardt iterations from 50
to 500. This way, the solver is able to crawl out of the bad
local minima it gets stuck in when changing from, for example,
bundling k1 and k2 to just k1 and resetting k2 to 0.
- Add several new region tracker implementations. A region tracker
is a libmv concept, which refers to tracking a template image
pattern through frames. The impact to end users is that tracking
should "just work better". I am reserving a more detailed
writeup, and maybe a paper, for later.
- Other libmv tweaks, such as detecting that a tracker is headed
outside of the image bounds.
This includes several changes made directly to the libmv extern
code rather expecting to get those changes through normal libmv
channels, because I, the libmv BDFL, decided it was faster to work
on libmv directly in Blender, then later reverse-port the libmv
changes from Blender back into libmv trunk. The interesting part
is that I added a full Levenberg-Marquardt loop to the region
tracking code, which should lead to a more stable solutions. I
also added a hacky implementation of "Efficient Second-Order
Minimization" for tracking, which works nicely. A more detailed
quantitative evaluation will follow.
Original patch by Keir, cleaned a bit by myself.
- Replace set of booleans with menu, so now you'll simply be unable to choose
unsupported refine combination
- Some internal code cleanup and minor refactor
===========================
Commiting camera tracking integration gsoc project into trunk.
This commit includes:
- Bundled version of libmv library (with some changes against official repo,
re-sync with libmv repo a bit later)
- New datatype ID called MovieClip which is optimized to work with movie
clips (both of movie files and image sequences) and doing camera/motion
tracking operations.
- New editor called Clip Editor which is currently used for motion/tracking
stuff only, but which can be easily extended to work with masks too.
This editor supports:
* Loading movie files/image sequences
* Build proxies with different size for loaded movie clip, also supports
building undistorted proxies to increase speed of playback in
undistorted mode.
* Manual lens distortion mode calibration using grid and grease pencil
* Supervised 2D tracking using two different algorithms KLT and SAD.
* Basic algorithm for feature detection
* Camera motion solving. scene orientation
- New constraints to "link" scene objects with solved motions from clip:
* Follow Track (make object follow 2D motion of track with given name
or parent object to reconstructed 3D position of track)
* Camera Solver to make camera moving in the same way as reconstructed camera
This commit NOT includes changes from tomato branch:
- New nodes (they'll be commited as separated patch)
- Automatic image offset guessing for image input node and image editor
(need to do more tests and gather more feedback)
- Code cleanup in libmv-capi. It's not so critical cleanup, just increasing
readability and understanadability of code. Better to make this chaneg when
Keir will finish his current patch.
More details about this project can be found on this page:
http://wiki.blender.org/index.php/User:Nazg-gul/GSoC-2011
Further development of small features would be done in trunk, bigger/experimental
features would first be implemented in tomato branch.
===========================
Code cleanup and minor refactor. Moved some functions from movieclip.c
to tracking.c due to they're operating on tracking data only.
===========================
- Set as Background button will enable displaying background images in 3d space.
- Move options in Display panel to column. Saves some space.
- Camera solving now should deal fine with non-centered principal.
===========================
- Changed 3D viewport reconstruction settings to more usable defaults
- Some code clean-up
- Option to place markers for detected features across the whole frame,
inside grease pencil areas or outside of grease pencil areas.
- Remove the overly-conservative use of libmv's re-track tracker. The re-track
tracker would take a normal tracker such as TRKLT or KLT or pyramid KLT, and
track from frame 1 to 2, then back from the position found in 2 back to 1.
Then, when the reverse-track doesn't match the original track with high
precision, the track is considered "failed". This is a good approach for
fully automatic reconstruction, but is too conservative for supervised
tracking.
The retrack-tracker will return when fully automatic tracking is added.
- Always solve for (dx, dy) in the TRKLT loop even if the linear system is
ill-conditioned. The client (Blender in this case) can still use the solved
position, even though it is less reliable.
- Expose the pyramid level setting to the tracking UI when in KLT tracking
mode. While it was tempting to hide this detail from the user, in reality it
does more harm than good, since the way tracking fails depends on the pyramid
level. For now, exposing the pyramid details is a good compromise.
- Move the settings for the tracking algorithm, including search window,
tracking algorithm, pyramid levels, etc, into the track object instead of a
global movie clip object.
===========================
Naming refactor: avoid using acquire from movie clip related functions.
This stuff doesn't need releasing, all locks happens in getters,
so using get instead of acquire makes more sense here.
===========================
Some improvements for feature detectors:
- Sort features by score when filtering features by
distance using FAST library.
- Added option to place markers only in areas outlined
by grease pencil.
===========================
Added option to choose which method to use for feature detector: FAST or Moravec
Moravec was supposed to be more "stable", but it collected worse feedback.
before returning back to FAST detector added option to easier toggle between
different detector to compare it's result and find out if we can leave only FAST.
===========================
Support for (un)distortion of images with overscan. Needed for
easier keeping this up-to-date until proper overscan support is
implemented in renderer. Not exposed into UI yet at all.
===========================
Fixed threading issues used to happen when compositor tree
and rendering happened in the same time. Problem was caused
by cached camera intrinsics stored in MovieTracking structure
and access to which wasn't thread-safe.
Now each movie distoriton mode stores it's own camera intrinsics
in it's storage which makes compositor (un)distoriton be totally
thread-safe and this also makes compositor tree faster when
(un)distortion happens on images with different resolution.
===========================
- Merge Movie Distort and Movie Undistort nodes into one node
called Movie Distortion where action (distort/undistort is specified).
- Implemented more proper distortion/undistortion for scaled images.
NOTE: Please, delete distortion nodes with current blender first
before opening files with blender versions from this commit/
===========================
- Bundles and camera path should be displayed correct
when there are several solved cameras in the same scene.
- Added option to use clip from active camera as background
in 3d viewport.
===========================
- Do not disable track when tracking frame-by-frame and tracking
threshold became bad.
- Show anchored image in track preview widget.
- Do not show search area for non-selected and disabled markers.
- Keep anchor constant position when sliding offset parameters in panel.
- Re-enabled occasionally disabled openmp for tracking.
- Renamed clearing operator so now buttons are more clear.
- Updated eigen to the very recent version.
NOTE: I had to enable static aligning again due to it gave crashes
when tracking on my new laptop. I'm not sure it'll work fine
on another systems.
===========================
- Added python method MovieClip.tracking.tracs.add() to add new track.
Marker for this track is getting created on position (0, 0) and it's
keyframed to specified frame number.
- Implemented rotation 2D stabilization. It's supposed to be used in the
following workflow:
* Set tracks which defines location stabilization.
* Choose one track which defines orientation.
* Vector which connects median point of location and track used for
rotation would have the same orientation along all frames.
2D stabilization with rotation can be a bit slow.
- Some internal refactor and code clean-up.
===========================
- Bundling new libmv, issues with undefined uint and C99 functions should gone.
- Changes to Detect operator to match new API.
===========================
- Distortion/undistortion were accidentally swapped in both of grease
pencil calibration and grid.
This was fixed but distortion coefficients should be re-adjusted.
There are some issues with high distortion coefficients but it
came from libmv and until it'll be fixed there nothing really
can be done here.
- Fixed displayed aspect ratio for muted footage when pixel/display
aspect ratio is differ from 1.0 or proxy size is not Full Render,
- Added c-api for image distortion/undistortion, Not used yet.
===========================
Changes into sensor size use case.
There was problem with vertical sensor size which can't actually
affect on things due to we've got fixed image resolution. So
as soon horizontal size gets defined, both of sensor height
and vertical FOV is known and exposing it into UI makes things
difficult to understand.
That's why vertical sensor size was dropped.
Also added pixel aspect to camera settings fo clip editor.
Solver will sue this value rather than value from display
aspect ratio.
This pixel aspect would be copied to render settings after solving
so things should match now between footage, solving anf footage.
===========================
Interpolate camera location and orientation for frames
where's no reconstruction data.
This fixes motion blur compositor node (first and last camera's
position was at origin, so motion vector was totally incorrect).
===========================
- Fixed bug with resetting marker's flag for non-transforming
tracks when canceling transformation.
- Fixed bug with keyframing non-selected tracks when initializing
tracking tools.
- Changed selection policy: now invisible parts of marker
can be selected.
- Added operator redo panel to Clip Editor.
- Set Scale operator now uses operator property for scale.
- Added operator "Clean Tracks" which currently checks if tracks
are tracked long enough and can select bad tracks, delete them
or delete tracking segments which are too short.
===========================
- Made naming in libmv-api more uniform.
- Move aspect ration detection of clip to movieclip.c.
- Remove unused width and height from camera intrinsics function.
- Reconstruction should now deal correct with aspect ratio.
===========================
- Fixed some silly things ni DNA design. Now all
reconstruction data is stored in Tracking->Reconstruction.
Please, re-solve your cameras -- reconstruction data
wouldn't be read from files saved in blender below this
commit.
- RNA accessors for reconstruction data.
- Store average reconstruction error in new reconstruction
structure and show it in clip editor header after
reconstruction.
- Highlight failed to reconstruct frames with red in cache line.
- Added "group" "Failed Tracks" in Select Grouped operator,
Meant to be used for selecting tracks bundles from which
failed to to be solved.
- Hotkey to delete marker: Shift-X.
- Jump to next/prev failed frame operator. Hotkeys are
Ctrl-Shift-Left/Right Arrow.
===========================
- Do not show frame boundary border when stabilization isn't enabled.
- Separate stabilization settings from display in clip editor.
Now clip can contain stabilization data but still be displayed
un-stabilized in clip editor.
- Internal changes in stabilization:
* Use separated location/scale parameters rather than 4x4 matrix.
In some ares "decomposed" data is needed (text draw functions, i.e.).
Also such decomposed data could be used in compositor.
* MovieClip now uses own structure for cache where additional data
can be stored. MovieCache structure now one of properties in
this new structure.
* Get rid of stable image buffer stored in MovieClipStabilization
structure. Pre-created buffer for scaling still stored there.
This helps to keep playback realtime -- re-creating this buffer
introduces ~15% slowdown.
- Added sliders to 2D stabilization panel which controls intensity
of translation/scale which applies on shot.
- Added filter type to Stabilize2D compositor node. Supports nearest,
bilinear and bicubic interpolation.
- After discussion with Sebastian and Francois added new node called
Transformation. It can apply translation, rotation and scale. It's
not the same thing as applying this components separately -- all
transformation is happening inside "canvas". And it should be more
accurate on interpolation and sub-pixel translation.
Need to check order of applying translation/scale/rotation btw.
- Added output sockets to movie clip compositor node. They holds
stabilization data which can be used by Translate or Transform
nodes.
- Minor fix of UI issues in Display panel.
===========================
- Fixed occasional crash caused by conflicting access
to non-threadsafe moviecache. Added lock on movie clip
ibuf acquire level.
- A bit nicer feedback from libmv when doing reconstruction.
Now if all tracks and frames were reconstructed average
error would be reported in "notification area".
If there's something failed to be reconstructed warning
message would appear and failed items would be printed to
the console.
===========================
Changed behavior of 2D stabilization:
- Fixed epic spelling error.
- Neither 3d viewport nor MovieClip compositor input node
are using stable footage now.
- Now one thread lock can be avoided -- stable shot
isn't acquiring from several threads.
- Added compositor noe Distort->Stabilize 2D which is
supposed to stabilize "incoming" image using movie clip
block as reference.
Probably it could be useful to define MAT4 socket type and
use it in MovieClip node as output for stabilization and
as input socket for Stabilization node so relation could be
easily visible on "graph".
===========================
Implemented 2D stabilization:
- Based on average point of several tracks.
- Tracks for stabilization are controllable from
"2D stabilization" panel.
- Implemented autoscaling. Could fail if average point is
too close to frame boundary.
- Background clip in 3D viewport and MovieClip input
compositor nodes are using stable shot.
Some internal weak things:
- Stable shots aren't cached in moviecache. There is
only one "slot" for stable shot inside stabilization
structure.
- This could fail if several threads are requested different
frames (for example when starting playback, pause to make
Nodes Editor trigger refresh, continue playback).
- Due to it shouldn't crash i decided to implement better
caching for stable shots a bit later.
===========================
- Operator "Join Selected Tracks".
This operator joins all selected tracks to active track.
Hotkey is Ctrl-J.
- Operator "Select Grouped".
Now it's easy to select all locked/keyframes/etc markers.
Hotkey is Shit-G.
- Operator to jump to beginning/end of active path.
Hotkeys are Shift+Ctrl+Arrow Left/Right.
- Operator to copy color from active track to all tracks.
===========================
- Fixed crash when holding down S and R keys.
- Fixed deleting of first marker.
- Tracking should now work fine with float buffers.
- Added option to show distortion grid. It's most probably
temporary solution until [auto]calibration tools aren't
merged.
This grid defines straight lines on the image.
===========================
- Internal change: do not use maroses for checking the only
one bit of track/marker flags.
- Do not display track name for disabled markers when
"Show Disabled" is turned off.
- Option to lock tracks. Locked tracks can't be be moved,
scaled, re-tracked and so.
===========================
- Selected bundles are now highlighted with outline.
It looks more Blender-way of highlighting selected items.
- Added option to show bundles names in 3D viewport.
- Bundles now coudl be selected from 3D viewport.
===========================
- Bundle re-newed libmv library.
- Update API in libmv-capi to match new library.
- Initial implementaition of feature detection.
- Fixed bug with incorrect user sync when tracking
for first frame.
Sergey Sharybin
Keir Mierle