Oh man. On Army of Two we were running a fully deterministic simulation and we had to patch many part of PhysX to maintain determinism. In the end, we had to get a waiver from Sony to ship a binary-patched version PhysX libraries. Good times.
All that hassle just for them to opensource it in the end. Yay bureaucracy! By the way good work on that game. I thoroughly enjoyed it when it came out.
We could detect if the simulation diverged. Then we had to replay deterministicly up to the frame of divergence and then break on various things that cause divergence. Floating point underflow/overflow, NaN, uninitialized memory access, divide by zero, etc. If my memory serves correct, the physx code referenced uninitialized memory. We had to patch their binary to initialize it.
This is the reason why most game physics engine run with a fixed timestamp. You normally set the hz update of your physics engine to either 120hz, 60hz, 30hz. Now if your rendering engine is rendering at 120hz and your physics engine is working at 60hz, you can do some very basic interpolation just simply using the objects mass, velocity.
Otherwise you run into situation where large deltaV, or very small deltaV completely break the physics calculations. The next is that rounding issue and precision with double still creep in and having a physicist engine that is deterministic for a given hz and float/double speeds up development time.
Without knowing the game: Some kind of replay/recap feature maybe or physics in cut scenes.
When you replay a series of inputs, you want the accompanying physics to come out the same. Solvers that use randomness in your engine will either need to have their RNG seeded with the same value or have the rng removed.
Another thing causing problems might be how your engine handles time or when it iterates over objects in a specific order relative to how they were added (might not have preserved that order in your snapshot).
Sometimes network code is easier with deterministic engines. Some engines simply replay the remote/network inputs as if they were local (with various strategies for handling the time sync/time delays), and rely on the property of determinism of their underlying simulations to avoid divergence in player world states.
Knowing only how much Army of Two stressed its cooperative network play nature (including in its title), that's my best lay guess of the strong reason the game wanted deterministic simulations.
The impression I got from dev team discussions/features on The Halo Channel/Master Chief Collection was that Halo was built that way: its network engine wanted rock solid deterministic physics, so all the replay/recap features added to later games were a "free" bonus they were able to build on top from that earlier netcode requirement.
This has it advantages and disadvantages. One its great and requires less bandwidth, but also means that any floating calculation rounding issue can throw different machines out of sync from each other.
It was definitely a real pain to ensure the whole stack was deterministic. Since we only targeted ps3 and xbox360 it was easier to ensure determinism. All in all, I wouldn’t recommend it.
Yeah, unless you have a entity count that exceeds your available bandwidth you're usually better off with a dead-reckoning system.
That also has the advantage of being able to "fake" events with effects until the server can resolve them meaning a latent connection can 'feel' faster.
Kudos to you for shipping a lockstep solution, we could never flush out the determinism bugs on our titles so we always just used dead-reckoning unless it was some dumb turn based game.
Depends on the mode the FPU is put into, fast or precise. If you want to have some fun, start a multiplayer game of AOE and switch the FPU mode from fast to precise. Doesn't take long to get out of sync, even for a basic game.
More recently we've taken the approach of "close enough is good enough". See Red Dead Redemptions "hilarious" cut scenes where the NPCs get involved, or the horse grts attached to the trains.
Another thing to consider about a replay system is that you don't have to be perfect. What you as a player see isn't what the server(if you have one) sees, and it's not what the other player sees. Your view is only a best guess at that, and for most replay systems in games, they will also be best guesses.
You need determinism in all multiplayer games that do not have an authoritative server. Although I would argue that you would want determinism in all games.
In terms of this game in particular, it's a multiplayer coop action shooter so you want everything the behave the same way on both players machines. The player models should be in consistent places in the world, bullets should fly in the same direction on both, grenades and other physics objects should bounce the same way.
If these things do not happen then you will have diverging realities on each of the machines.
