The Machinery — March 2022 (version 2022.3)

Mar 31, 2022

Thanks to everyone who ventured outside of the Moscone area for our GDC meetup! It was great to see you, and also great for parts of our team to be able to meet physically for the first time since the pandemic started. We understand that everybody makes different risk assessments and that, for many, going to something like GDC doesn’t make a lot of sense, so we’ll try to do more virtual events too. For example, on the 29th of April, Niklas will be giving a prerecorded talk at the Hytradboi conference.

As usual, we have a bunch of fixes and new features for you this month.

If you are already running The Machinery, this new release should pop up in your Downloads tab. Otherwise, get it from our download page and have a look at this quick introduction video. We are also working on a book about The Machinery. Note that the book is still a work in progress.

If you find bugs in this release or have requests for specific features, post them to our issue tracker. For more general discussions, hit us up on GitHub discussions or Discord.

Key highlights in this release are:

And as usual, we have a number of smaller updates.

Buffer Eviction

The engine now has a low-level API for evicting unused buffer data from The Truth to reduce the memory use. In addition, lock(), unlock(), and frame_lock() calls have been added to allow buffers to be marked as in-use or not used and free to evict.

Since not all parts of the engine support the lock() and unlock() protocol, the buffer eviction system is currently opt-in. I.e., only buffers that have explicitly said that they can be evicted will be up for eviction. When the locking system has been fully implemented, eviction support will be mandatory.

You can inspect the current state of the buffers in the new Buffers view in the Memory Use debug tab:

This shows the memory used by all buffers. Buffers that have opted in to the eviction system have an Evict button next to them. If the buffer is currently in-use and can’t be evicted, the evict button is grayed out. Otherwise, you can press the evict button to force-evict the buffer.

In an upcoming release, we’ll add a high-level system built on top of this low-level foundation that allows you to stream in and out sets of buffers in the background — for example, all the buffers that are used by a particular level.

Another change is that buffers are no longer auto-loaded when you load a project. Instead, buffers will be loaded on-demand as they are needed.

DCC Asset Reimport Overhaul

Reimport of DCC assets now works better and is less destructive. Before this, we tried to in-place update the whole DCC asset by comparing the names of the objects within the DCC asset. This has been replaced with a more robust process where we replace the whole DCC asset and then just patch it up by carrying over data from the old asset that the user may have affected.

The reimport will look for any entity imported from the DCC asset and go through all entities imported from it. It will add, update, and remove mesh entities to reflect what exists in the new DCC asset.

There is also an option to do re-extraction of materials and images on reimport, this will bring in any new materials and images that didn’t exist before.

We removed the tm_the_truth_replacer_api that was used to do the old in-place rebuild of the DCC asset as it is now unused and a bit broken.

Clouds Component (Preview)

We are releasing an early preview of our physically-based clouds component, this way we can hear your feedback in the early stages of development.

With the default settings we do raymarching in 1/4th of the viewport resolution and do a temporal reconstruction to achieve the final output. If quality is more important than performance we offer an option to render the clouds in full resolution. Besides the basics settings, you’ll need to create a weather texture, where the Red channel gives the clouds coverage and Green channel the cloud type, with 0.0 stratus, 0.5 for stratus-cumulus, and 1.0 for cumulus.

Community Contributions

Thanks to Tisten for multiple fixes:
- Fix for a crash when changing transforms. #1218
- Fix for bad placements of rigid body shapes. #922
- Fix for bad transforms for child objects to physics bodies. #919
- Added a Save As Copy menu entry. #910
- Fix for the reimport dialogue. #1143 #902
- Update to the core project. #899
- Parallelized the compression of collaboration buffers. #894
Thanks to jamesmintram for multiple fixes:
- Fix for a bug causing proliferation of animation curves. #1195 #920
- Fix for an issue causing physics entities to move in editor mode. #918
- Better error message on type mismatch. #917
- Add a NULL check to prevent a crash in the animation preview. #916
- Remove the log warning when the physics shape is TM_PHYSICS_SHAPE_NONE. #915
- Removed an unused reference in the physics mover. #912
- Notify the task manager API about animation compression tasks. #911
- Add cancellation support to the analyze types task in the Truth Inspector. #908
Thanks to Selmar for performance fixes and other improvements to the stack trace printing on Windows. #898
Thanks to ElementalAndy for a fix in the checking of the kinematic flag when mirroring rigid body transforms. #897

Collaboration

The Machinery now supports synchronization of avatars that represent the other collaborators in a collaboration session. In the future, we will use this to visualize the other collaborators in the scene to give a greater feeling of presence.
Fixed a deadlock when modal dialogs are shown during a collaboration session. #1059 #1091
Fix for a crash when a collaboration server is saving. #1055
Better behavior of Reimport during collaboration sessions. #1093
Collaboration Cache files are now saved using a write/rename protocol to prevent corruption if the engine crashes in the middle of the write.
Compression of buffers no longer uses a temp allocator, as this could consume too much memory when the project is large.
Added a modal dialogue warning when a client loses connection unexpectedly.

UI

Fix for a threading issue that could cause corruption in the IMGUI active data. #1123
The Numpad Enter key can now be used in all places that accept the regular Enter key.
The UI now requires an Up event between two Down events to register a double-click. Just having two Down events is not enough. This prevents double-clicks from being registered in some situations where multiple Down events were being sent. #1090
Fix for an infinite loop when drawing Bezier paths with NaN values in them. #1100
Better clipping of workspace names. #1210

Entity Tree

You can now double-click on an Animation State Machine component to open the corresponding state machine. #1164
Custom Entity Menu Items can now be implemented via the tm_entity_tree_custom_context_menu_item_i interface.
It’s now possible to remove an instantiated child in a single operation. #1159
Fix for Replace With Asset menu being shown for overridden children. #1158
Locally added components and children will now be maintained when the prototype of an entity changes. #953
Fix for not showing the color of the root entity correctly.
Added Make Unique context menu option for instantiated entities. #1157
Fix for error when creating entity asset after right-clicking in the tab background. #1207

Foundation

The engine now uses a round-robin scheme to look for dirty plugins to hot-reload. This cuts down the check_hot_reload() time from 4.4ms to 0.3ms.

The Truth

Truth types can now be excluded from the changelog using the tm_tt_aspect__skip_changelog aspect. This can be used to mark “transient” data that shouldn’t be synced over the network or restored in the case of a crash.
Fix for import de-duplication in the case of recursive references. #1015
Fix for potentially corrupting project when saving as directory project. #1119
Fix for propagate_property_subobject() not taking into account locally removed subobjects.
Fix for the propagation of changes to the prototype being broken when there were references within the propagated object to newly added The Truth objects. This makes graphs where lots of additions have been made propagate properly.
Added an aspect called tm_tt_aspect__before_propagate that can be used to inject a callback just before the propagation of an object happens. This is useful if the object needs some additional logic in order to properly do the propagation. This is used in the graph to propagate connections that are implicitly disabled by the graph code, but not in The Truth.
Added clear_object_recursively. It does what clear_object does but also destroys subobjects recursively, instead of just clearing the object passed. This is the behavior expected in many situations and this function is now used in many places.
Deprecated use of ensure_all_loaded() — we never want to force the engine to load all buffers since that could potentially take a long time. #1233

Preview Tab

The Default Lighting Environment used by the Preview Tab now contains a sky entity, giving it a nicer default light source.

Asset Pipeline

Core now contains a default material. This material is automatically assigned when asset import fails to find a material to use.

Graph Editor

Only re-wire connections when CTRL is held. Before this you needed to hold CTRL to make new connections, i.e. the logic is reversed now.
Focus mode editing of graphs works again. It was broken due to indirection index buffers not being merged while in focus mode.

Entity Graphs

Fix for graph wires not being properly initialized. #1090
Added the Set Parent node. #1141
Added the Clear Entity Variable node. #1153
Added Get Asset Reference node.
Added Float Negate.
Added Float Sign (Without Zero).
Added Float Ternary.
Added Vec2 Sign, Vec3 Sign, Vec4 Sign.
Added Vec2 Sign (Without Zero), Vec3 Sign (Without Zero), Vec4 Sign (Without Zero).

Creation Graphs

Fixed a bug in Construct Transform node math where scaling wasn’t respected properly.
New Write Instance Input node for better interop between multiple creation graph instances.
Get Instance node now has a Required bool that will retry to lookup another instance if fails due to dependency ordering issues.
Better state management for handling shared resources.
Circular dependency management — prevention from getting stuck in infinite loops if two instances reads output from each other.
Fix for crash sometimes happening when setting up Loops using the Loop Begin and Loop End nodes.
GPUSim - Spawn node now supports spawning elements into a GPUSim system based on the elements of another GPUSim system. This allows for driving spawn operations completely on the GPU.

Properties

The Add Component button now works better when multiple entities are selected. #1081
Fix for wrong Paste behavior when copying overridden components. #1110
The entity prototype picker is now shown if the root entity is selected in the Entity Tree. #1155

Scene

Fix for not correctly accounting for the last selection when applying duplication offset. #1125

Core

Fix for core update sometimes failing to remove assets that have been removed from the core project.

Entity

Removed get_component() and added read_component() and write_component() to facilitate the transition toward component change detection.
The local position of the transform component in The Truth is now stored in doubles.
Added a boolean flag to the add() and remove() component callbacks that indicates whether the component is being added/created as part of the entity creation/destruction or not.
Fix for set_parent() not updating the world transform if setting the parent to NULL.
Fix for asset propagation not working properly when an entity has the physics_body component. #1211
Fix for an error message in the game state. #1232
Added has_component to check whether a component is available without accessing its data.
Fix for notify not being called after some load_asset calls.
Persistent and Replication settings can now be changed at runtime.

Physics

For Box shapes, there is now a button to calculate the shape from the bounding box of the render component.

PhysX

The PhysX-specific components cannot be accessed from external code anymore: this makes it easier to support multiple physics backends.
Physics components are now created without any delay. #98
PhysX shape creation now takes the scale into consideration, this makes the offset of the shape correct when the parent entity has a scale set.

Project Management

The Network Settings are now part of the Project Settings.

Entity Spawner

Faster physics management.
The physics creation now respects the stride of the buffer data that comes from the Creation Graph. This makes it possible to have data that contains more than just transforms, even if you just use the transforms.
Made it possible to delete entity spawner channels.
Added initialization creation graph that can be used for setting up shared resources accessible from each channel. This makes it possible to more cleanly setup things like occupancy maps to prevent intersecting entities placed by different channels.
Make sure to invalidate cell spawner entites if any channel becomes dirty.

Paint Nodes

Instantiate graph node auxiliary data when it is inherited.
Don’t move the brush while the camera moves.
Introduced Paint Settings Object, moved frequency from toolbar to settings object.
Support for overriding picked paint normal with an explicit normal to better support painting on stuff like terrain.
Stop triggering Brush Render event while the camera moves.

Thumbnails

Fix for material thumbnails always getting dirty.
Fix for crash while moving icon size slider due to not waiting for GPU readbacks to finish in the thumbnails code.

Publish

Fix for The Truth errors while publishing.

Simulation Entry

Fix for crash when modifying scene entity while simulating.

Viewer

Removed viewer_render_args from the VT of tabs and instead put in an interface called tm_viewer_render_args_provider_i that tabs etc can implement. With this, the thumbnail generation can properly generate cubemaps, as it can get hold of the render args and entity context needed to do the generation.

Task Manager

Fix for displaying the wrong number of tasks in progress.

Animation

Fixed drawing issues with the blend state map. #1208

Depth of Field

Fixed bordering artifacts. #1149

Renderer

Added a new flag to create_image that allows it to be cleared on the device before the first use.
Removed TM_RENDERER_BUFFER_USAGE_VERTEX.
NaN/INF debug visualize now also shows negative values in blue.
Constant/resource buffers created by the shader system now use the shader name as part of their debug tag, making it easier to debug them.
Alpha masked materials are now rendered into its own gbuffer layer after fully opaque materials have completed.
Vulkan backend: Better validation and error handling if dangling tm_renderer_handle_t are passed to the backend by accident.
Render Component now supports visualizing its bounding volume used for culling.
GPU Scene Submission system now does bounding sphere culling culling with its own origin.
GPU Scene Submission system now has GPU performance markers around all indirect draw calls and compute dispatches
Structs in all rendering systems now have unique names to avoid confusing the Visual Studio Debugger.

API Documentation

Added missing documentation about shader interpolation modifiers.

Math

Added tm_aabb_t to extra path plugin.
Added various utility functions for AABB’s to extra math plugin.
Fixed a bad conversion case for tm_quaternion_to_euler(). Also clarified that the Euler angles use ZYX order. #1206