UPDATE:
Well, I finally managed to fix the problem described in this thread — at least on my affected systems.
After all the driver changes, Windows reinstalls, hardware replacements and settings experiments, the practical solution turned out to be changing how completed GPU frames are handed to OBS. I made an experimental fork that adds a small ready-frame queue for Game Capture and DXGI Desktop Duplication.
The long periods where the preview and recording suddenly looked like 30–40 FPS are now gone in my tests. Rogue Trader source improved significantly. Very short isolated hitches may still happen, so I am not claiming this will fix every type of stutter on every PC.
Source code, build instructions and a Windows installer are here:
https://github.com/veelasama/obs-studio-capture-stutter-fix
The installer is unsigned, so Windows SmartScreen will show an unknown-publisher warning. The release includes a SHA-256 checksum, and
everything can also be built directly from the public source.
OpenAI Codex assisted with the code investigation and implementation, and I manually reproduced and tested the changes.
If anyone here has the same specific symptoms, I would be interested to hear simply whether it helped or not.
A quick note to the moderators: if the repository link, project name, or any part of this post conflicts with the forum rules, please let me know and I will edit it or remove the link immediately. I would really prefer to preserve the thread because it documents a real long-running capture problem, extensive testing, and a possible solution that may help other affected users. I am not trying to present this build as official or use the forum for promotion.