This change makes it so build system and update utilities for Blender builds
are using pre-compiled libraries and other resources attached as Git modules
instead of using checkout of SVN repositories in the parent folder.
The directory layout:
```
* release/datafiles/
* assets/ -> blender-assets.git
* publish/
* ...
* README.txt
* lib/
* darwin_x64/ -> lib-darwin_x64.git
* darwin_arm64/ -> lib-darwin_arm64.git
* linux_x64/ -> lib-linux_x64.git
* windows_x64/ -> lib-windows_x64.git
* tests/
* data/ -> blender-test-data.git
```
The changes about configuring the actual Git sub-modules are not included
into this patch, as those require repository to actually exist before it
can be used.
The assets submodule is enabled by default, and the rest of them are
disabled. This means that if someone runs `git submodule update --init`
they will not get heavy libraries. The platform-specific and tests
related submodules are enabled when using `make update` or `make test`.
All the submodules are tracked: this means that when new commits are
done to the submodule, the blender.git repository is to be updated to
point them to the new hash. This causes some extra manual work, but it
allows to more easily update Blender and its dependencies to known good
state when performing operations like bisect.
Ref #108978
Pull Request: https://projects.blender.org/blender/blender/pulls/117946
the path for clang_format_paths.py changed when the submodules
moved but format.cmd was never updated for that.
the work previously done by check_submodules.cmd is now done by
make_update.py so this file can be removed.
The issue we ran into a lot is we have a python script that updates
git+SVN. Which works fine most of the time, except when we have a
python update in SVN, or worse a python version change. Python really
doesn't enjoy having its files being deleted or changed while it is
running and users generally end up with a corrupted lib folder.
This change updates the library folder using svn.exe first before
letting the python script run sidestepping the issue in most cases.
The python script will still run and do the more elaborate work
like updating git and switching SVN branches which could still
run into issues cause python still doesn't like being changed
while running but there's not a whole lot we can about that,
for *most* people however things will just work now.
Several sub commands tried on their own
to locate python, given I wanted to look
in several locations for a broader libdir
compatibility this is best done in a
central location.
Python 3.9 is still preferred, but if
3.10-3.12 are available that be accepted
as well.
note: this is about the python version
make.bat uses to run various python helper
scripts, this change has no influence on
the python version blender itself uses.
SVN seems to die randomly *a lot* during
large updates for some users, and I'm no
closer to finding out why that keeps happening.
"The internet" seems to imply some AV vendors
may be at fault here but nothing conclusive.
The solution however is repeatedly running
`svn cleanup`and `svn update` in the library
folder to repair the corruption and finish the
update.
This change adds a small convenience helper
to automate the repair.
This is done inside the make.bat code rather
than the shared python based update code, since
python lives in the library folder and may
or may not exist when this corruption occurs.
This adds support to compile the html python api docs from the command line by running `make doc_py` matching support between windows and unix.
This patch also makes it so the compiler is not needed if you set the `blender_bin` variable, this affects icon generation as well.
In the future, I want to move away from generating the build output in the build directory but that can come in a later change.
Reviewed By: LazyDodo
Differential Revision: https://developer.blender.org/D12144
This adds support for building the icons from make.bat
unlike bash there is no passing environment variables
on the command line.
The scripts go out of their way to locate both blender
and inkscape however if they are not found, the user is
given a helpful error message telling them how to set
the variables.
Although some extra help can be given there, if your
normal build is a 2019 full build running
`make 2019 full icons`
will help it find the blender executable as well.
finally if you know the name of your build folder
running
`make builddir build_windows_Lite_x64_vc16_Release icons`
will also work, if all fails you can point directly to
the blender executable by running
`set BLENDER_BIN=c:\where\blender\lives\blender.exe`
before running `make icons` or `make icons_geom`
The python scripts needed some small modifications since
without the PATHEXT, SystemRoot and SystemDrive
environment variables python will not initialize properly
on windows. (Not blender related, even mainline python
won't start without those)
This change removes 32 bit support from the helper make.bat scripts
as we are dropping official 32 bit support, you can still build for
32 bit by configuring your build yourself using cmake and pointing
the LIBDIR cmake variable to your own 32 bit library folder.
There are couple of caviats currently:
- The script requires system-wide Python 3 available in the current
search PATH as python.exe.
This will get addressed soon by distributing unpacked Python binary
in our libraries.
- Since the libraries folder is to be known, this requires to have
MSVC detected. Not too bad, since formatting is still way slower
than detection, but still doesn't feel ideal.
ninja is an alternative to msbuild designed for fast rebuilds. However there is no IDE support, builds only from the command line.
Comparison between msbuild and ninja for a full build, build time in seconds.
Full Clean Build
msbuild 867.5
Ninja 801.2
Difference -66.3 (-7.6%)
Minor Change
msbuild 43.0
Ninja 14.9
Difference -28.1 (-64.4%)
No Changes
msbuild 23.0
Ninja 6.1
Difference -16.9 (-73.5%)
make.bat was starting to become hard to maintain, this refactors it into separate batch files for each stage of the process.
-Improved detection of msvc2013/2015
-Improved failure handling.
-Added check for working msbuild and C++ compiler
-Added verbose switch to ease trouble shooting.
-Added Check if svn/cmake/git are in the path before using them
-Display the build configuration before asking to download the libraries
-Offer an option to recover an interrupted checkout of the libraries.
-Automatically check out sub-modules in-case they are missing.
The registry hack we were using wasn't very reliable, the recommended way to locating visual studio is using vswhere (15.2 and up), using it also allows to switch between the regular and pre-release versions.
The thing that most often still goes wrong for new users building blender on windows is checking out the libraries, some skip over the wiki, some check out to the wrong folder, in an effort to reduce the time i spend on this, I added detection of svn and misisng libs to make.bat .
When the user has svn installed, and the libdir is missing he'll be asked if he wants to download them
if svn is not installed, or the user chooses 'no' the current error message is shown.
Reviewers: Blendify, sergey, juicyfruit
Reviewed By: sergey
Differential Revision: https://developer.blender.org/D2782
This was set to maxcpu which in an 8 core box would be 8, each project would then spawn
8 instances of cl.exe, making a possible of 64 simultaneously running compiler instances
slowing the compile down instead of speeding it up.
Currently the tests don't run on windows for the following reasons
1) render_graph_finalize has an linking issue due missing a bunch of libraries (not sure why this is not an issue for linux)
2) This one is more interesting, in test/python/cmakelists.txt ${TEST_BLENDER_EXE_BARE} and ${TEST_BLENDER_EXE} are flat out wrong, but for some reason this doesn't matter for most tests, cause ctest will actually go out and look for the executable and fix the path for you *BUT* only for the command, if you use them in any of the parameters it'll happily pass on the wrong path.
3) on linux you can just run a .py file, windows is not as awesome and needs to be told to run it with pyton.
4) had to use the NAME/COMMAND long form of add_test otherwise $<TARGET_FILE:blender> doesn't get expanded, why? beats me.
5) missing idiff.exe for msvc2015/x64 in the libs folder.
This patch addresses 1-4 , but given I have no working Linux build environment, I'm unsure if it'll break anything there
5 has been fixed in rBL61751
Reviewers: juicyfruit, brecht, sergey
Reviewed By: sergey
Subscribers: Blendify
Tags: #cycles, #automated_testing
Differential Revision: https://developer.blender.org/D2367
- Explicitly specify the platform for msbuild, to facilitate builds with just the Visual C++ Build Tools installed.
- When vs2013 is not found, try looking for 2015 as a fallback
- Clear up any batch variables that might have been set from previous runs
- The build folder name used to be depended on the order of the parameters, this is now normalized to
"build_windows_[Release/Full/Lite/Headless/Cycles/Bpy]_[x86/x64]_vc[12/14]_[Release/Debug]" regardless of the order of the parameters.
-Use CUDA8 for all kernels when building the release convenience target with visual studio 2015
New features:
1) Release target that checks for both cuda 7.5 and 8 with WITH_CYCLES_CUDA_BINARIES=ON and CYCLES_CUDA_BINARIES_ARCH=sm_20;sm_21;sm_30;sm_35;sm_37;sm_50;sm_52;sm_60;sm_61 options set.
2) Option to switch between x86 and x64 builds, the default remains (auto detect the architecture) but can be overridden.
3) Option to switch between vs12(2013) and vs14(2015) default is 2013.
Reviewers: juicyfruit, sergey
Reviewed By: sergey
Tags: #platform:_windows
Differential Revision: https://developer.blender.org/D2180
Sergey Sharybin
guishe
Ray Molenkamp
Campbell Barton
Campbell Barton
Aaron Carlisle
Brecht Van Lommel
Sergey Sharybin