@blenderyard can you shade a light on this maze?
It is suggested is to start with the implementation of an operator that users can execute from the UI.
The render operator (F12 key) is bpy.ops.render.render(). You can know it from the tool tip of the Render button. In the Python console of Blender, the bpy.ops.render.render.idname() method gives you the operator’s internal identifier ‘RENDER_OT_render’.
The ‘RENDER_OT_render’ operator is implemented in source/blender/editors/render/render_internal.c. To find it, you need a good string search tool that works efficiently on many files in the Blender code base. I’ve been using VS 2013 Express.
A bit of code reading and guesses will figure out the following series of function calls: RENDER_OT_render() >> screen_render_invoke() >> render_startjob(). The last function then calls either RE_BlenderAnim() or RE_BlenderFrame(), i.e., animation rendering or still image rendering API function (FYI, the RE_ prefix indicates the function is part of the render API. Similarly, BLI for blenlib, BKE for blenkernel, and ED for editors).
These render API functions are defined in source/blender/render/intern/source/pipeline.c. The control flow from one of the render API functions are rather straightforward, although a series of function calls can be quite deep. Basically individual functions in the source file address a set of render options. Just start following the control flow from RE_BlenderFrame(). Initially the control flow from there would look like a maze indeed, but it is my impression is that the render code is well organized and relatively easy to understand.
It is important to rely on a good code reading tool. Jumping to the definition/declaration of a function from a call of the function, as well as finding all occurrences of an identifier (function name, macro name, etc.) are essential operations that the code reading tool should provide you (VS 2013 Express is a good option in this sense).
Thanks for the hints!
That helps a ton.
@blenderyard, can you show screenshots of how you traverse thru the code. That would help a lot for non coder like me to start to see the big picture of BI’s structure. In laymen’s terms that is. XD
Alright, here are a few screen captures to demonstrate code reading in Visual Studio Express 2013 for Windows Desktop (a free edition). Four commands useful for code navigation are explained.
The first is the Find All References command that lists up all occurrences of a symbol. For example, all references of the RE_BlenderFrame() function can be identified as follows:
The second is the Go To Declaration command that jump from a reference of a symbol to the declaration of the symbol, as shown in the two screen captures below:
The third one is the Go To Definition command that brings you the definition of a symbol as follows:
The last one is the string search command (Ctrl-F) with the search range set to the Current Project or Entire Solution:
You can move forward and backward in the traversal history by using the arrow buttons in blue filled circles available in the upper left part of the VS main window.
These screenshots are certainly useful for everyone trying to dig into Blender’s coding. Thanks @blenderyard.
I made a worksheet in open office which contains the following:
The workflow for the BI renderengine as described by Brecht.
A link to all the files in the source/render structure, so the actual files can be accessed directly.
Per directory the update per file what has been fixed/patched in previous years. It is a link to the patch as well.
Use the following link:
I will be updating this file with diagrams to show the relationships between blocks of code. Anyone can alter the file or add comments.
Thanks, but I believe its a bit the same as the fossies link we are currently using. http://fossies.org/dox/blender-v2.71/bake_8c.html
I will try to use your explanation for VC as I installed the express version yesterday (2012).
Sure, any code reading assistance tool should be able to find all references of symbols, jump to the definition/declaration of symbols, and find strings from the entire code base.
VS 2012 should work fine as well with these code navigation commands (:
Can we have Google Spreadsheet so that we can collaborate on updating that part? Leveraging each other’s knowledge, it will be quicker that way. Way quicker than doing it alone.
Better explain the reason why I choose for not making it a google docs and chose for openoffice. Google docs do not have a good way of incorporating graphics. The reason why I would use symbols like arrows is to make a tab that has a graph -just like the ones in the Brecht notes which are incorporated in the document. In this way we could easily make the structure readible and referring to the code blocks, loops , calls etc without going in a direct detail. The Brecht notes are on a higher abstract level and we need to go one or two levels deeper. Well thats how I envision to document stuff. Using powerpoint is not very handy.
Besides, the document is readable/writable for everyone with the link. And it only would need open office to install which is open software. I there are ways for using google docs better then would use them because no one has to install anything.
I delved into the BI code for a bit and discovered that it pretty much works the same as described in the Brecht notes (http://wiki.blender.org/index.php/Dev:Source/Render/Pipeline).
I still have some questions regarding the pipeline. As I noticed there is a selection for which renderengine is used (external yes or no). But what I was wondering is if this BI pipeline is the same as the cycles pipeline. The reason why I ask this is because there are special codes in there for halos and blur which makes this pipeline a bit more complex to use or to alter. Besides if Cycles has it own pipeline/flow of things I expect it would be cleaner and easier to use and alter. In fact an own pipeline is the best but freestyle has already been coded into both pipelines now (if there two) so making a copy of one of them would be the smartest way and then alter it to our needs.
Hope you can shine some light on this.
The BI pipeline is not the same with Cycles’. The divergence of the two is quite quick when the control flow comes to the first call of a rendering engine API function, i.e.,
RE_engine_render(). After entering this last function, external render engines will go through their own pipelines and no longer rely on the BI pipeline. External rendering engines have a few callback functions with which they interact with Blender. The rest of the rendering engines can be whatever you can imagine.
If BEER is going to be an external rendering engine for Blender, then we need to check if the expected interactions between BEER and Blender can be done through the render API (i.e., by means of the callback functions).
Freestyle has been integrated into Blender as a post-processing option, just in line with the compositor and video sequencer. There was a discussion to integrate Freestyle as an external rendering engine using the render API, but that did not happen. Freestyle is embedded in the BI pipeline as well as in the render API. Recent integration of Freestyle into Cycles did not alter the Cycles internal at all, and modifications are only made in the render API. That means that not only Cycles but also any external rendering engines can benefit from Freestyle line rendering capabilities.
I believe BEER needs its own rendering pipeline. I don’t think copying one of the existing rendering pipelines is the way to go. BEER is very likely to have a specific set of implementation requirements, which define BEER’s own pipeline. By pipeline I merely mean a series of procedural steps that start with a bunch of 3D triangular faces and result in a rendered 2D raster image. For sure the existing code base of the Blender Internal and Cycles will give us insights on rendering engine development (having the source of a working program in hand is far better than having nothing).
One option at this stage of BEER development could be to prototype BEER as an external rendering engine for Blender by writing it purely in Python. The prototype may have a proper GUI in Blender with a set of options whose values can be save in and loaded from a .blend file. The rendering output of the prototype would be a blank image or something very primitive. Still this case study will help us check if using the render API is feasible for the real BEER implementation in C++.
Thanks for your extensive answer. This gave me a lot to think about but many points where already discussed. Still we are looking for focus on the project.
Understood about the BI Pipeline and the external render engines. Thanks. My question was more about the fact that if cycles is the renderengine what happens after the RE_engine_render. Does cycles follow the BI pipeline or does it have an own pipeline?
If it does it would be easier to implement a Beer pipeline because we could copy the way it activates the cycles pipeline. I can read code and flows but I am pretty new at C/C++ and Python.
True Beer needs it own rendering pipeline. But because our goal is to get an internal renderer most things in Blender could be reused. Beer wants to make use of the opengl pipeline so that would mean that a lot of code regarding how pixels are converted to 2D is not necessary in the BEER pipeline. So you could let out parts of the code. Things I expect you do need is the way scenes are setup in the database and materials are applied to them. This is information that is translated to the opengl api’s. The fun thing is that there is already an opengl renderer implemented in Blender but that is only using the viewport as a camera and outputs to a jpg file including dotted lines from lamps etc. But that will be pretty close to what we want because even in viewport data is sent from a database in memory to the opengl pipeline. In fact we need to study all three options to understand how the render process works.
The prototyping suggestion is a good one in my opinion and there was a point where we discussed this. But it has some pro’s and con’s.:
Pro’s (in favour):
- Prototyping in Phyton sounds very good, will be slower then c/c++ but Python is a higher level language and easier to code.
- We could study the way Luxrender does things in Blender as they have nicely documented their code. There are others too and maybe we could even get in contact with some of the builders.
- Having something to show that works will attract more people to our cause then having only powerpoints and papers. I.e. better for funding and people who are willing to contribute.
- Will take up much more time, we have to make a separate program and we have to study for how to do this as an external render. All of this is not our current goal.
We have to discuss this within the project.
Freestyle is an absolute must to have in the BEER pipeline and currently the BI pipeline has the best implementation for Freestyle because of the latest functionality for using textures.
We have to make some projectplanning on who does what, and which things to look into.
Please feel free to respond to the above mentioned points and if I misunderstood things please explain this in some more detail so I will better understand what you mean.
Cycles has its own rendering pipeline. There is almost no dependence on the Blender Internal pipeline. Cycles provides us with a good reference implementation of a rendering engine in Blender. The existing code base of Cycles (after stripping the majority of Cycles-specific portions) will be a template for developing a new rendering engine. In that sense copying the existing code fully make sense.
Freestyle will be available in Cycles starting from the upcoming Blender 2.72 release. This new feature has been implemented through the rendering engine API. This means that any other external rendering engines (including BEER) integrated by means of the API will benefit from Freestyle line rendering capabilities, without or with minor integration efforts on the Freestyle side. It is likely that BEER-Freestyle integration is almost automatic.
I was thinking the same for both 1) and 2), based on a very quick reading of the Cycles’ classes.
An added advantage is that it’s C++ instead of C, something that allows for better code structure (I’d say code modeling but here it’s ambiguous… :D)