Ashes of the Singularity DirectX 12 vs DirectX 11 Gaming

Today we'll be looking at one of the very first DirectX 12 game benchmarks by using Stardock's real-time strategy game, Ashes of the Singularity. Ashes of the Singularity was developed with Oxide’s Nitrous game engine and tells the story of an existential war waged on an unprecedented scale across the galaxy. The Ashes of the Singularity Benchmark was never designed as a synthetic stress test, but a real world test that was used internally to measure overall system performance. That internal developer tool was released to the public today as a DX12 benchmark! Ashes of the Singularity Here is what the developers had to say about the Ashes of the Singularity benchmark:
The benchmark is representative of real world gameplay. That means that while the strategic AI is scripted, everything else is being simulated just like it would be in game, such as tactical AI, physics, pathfinding, targeting, visibility, etc. To that extent, there will be more variability in the results than a synthetic benchmark would have. What we have noted however, is that performance has been consistent and several runs can be used to generate an accurate representation of performance. Even though we have been working with Microsoft on DirectX 12 for a while now, DirectX 12 and Windows 10 have only recently released to public. As such, we expect that the DirectX 12 results will only improve from here on out. We along with AMD, Microsoft, and Nvidia will continue to make improvements. With that in mind, this version should be treated as more of a performance preview to celebrate our excitement for Windows 10 and DirectX 12, as opposed to a hardcore synthetic test Ashes of the Singularity Benchmark Technical Features:
  1. Multi-Threaded Performance: Compared to DirectX 11, which used single-threaded and multithreaded performance, Ashes of the Singularity will see huge gains in complex workloads. This will allow for significantly improved balance of workload between CPU cores.
  2. (Coming Soon) Explicit Multi-GPU Support: With DirectX 12 comes the ability to use multiple GPUs. This allows for better control of multi-GPU rendering by developer, thus controlling the frame. Without this ability, the integrated GPU would remain idle and unused as an untapped resource to increase rendering and speed up the frame.
  3. Asynchronous Shaders: This allows the schedule of work for the GPU that will be performed. Traditionally, the GPU’s command queue would have had stalls, and DirectX 12 essentially provides more work done for free.
  4. Explicit Frame Management: DirectX 12 will see a reduction in latency, which will provide a more responsive game experience. This will also allow tracking of specific information, such as whether something is GPU or CPU bound.
  5. Updated Memory Management Design: A radical change in the memory management design allows developers to remove traditional performance issues such as micro-stutters that have plagued D3D11.
  6. DirectX 12 has taken great strides to prevent "screen tearing" by defaulting to having images rendered at the screen's refresh rate (which would result in frame rate limiting).  However, for benchmark purposes, we have taken steps to bypass this to give a more accurate representation of potential performance.
  7. MSAA is implemented differently on DirectX 12 than DirectX 11.  Because it is so new, it has not been optimized yet by us or by the graphics vendors.  During benchmarking, we recommend disabling MSAA until we (Oxide/Nidia/AMD/Microsoft) have had more time to assess best use cases.
Ashes of Singularity Benchmark When you launch the Ashes of the Singularity Benchmark you are greeted with this screen that clearly notes that this is a pre-Beta build that "will probably kill your pets, wreck your computer." Ashes of the Singularity Game Settings We disabled MSAA since there are known issues with having that enabled as well as VSync. We left the Temporal AA Quality and Terrain Shading Samples on 'Mid' and the other image quality settings at their default value of 'High'. Since we are using an EVGA GeForce GTX 960 SSC 4GB video for testing and set the screen resolution at 1920 x 1080. This card was selected due to it being $229.99 shipped as it is what we would consider a mainstream video card that is popular right now. Let's check out the results on the next page!

DX12 versus DX11 Game Testing

DirectX 11 CPU Usage - Ashes of the Singularity:  DirectX 11 CPU Load - Ashes of the Singularity When running our Intel Core i7-4960X processor with just four active cores and no Intel Hyper-Threading enabled we found that we were at about 75% CPU load when running the Ashes of the Singularity benchmark in DirectX 11 mode. DirectX 12 CPU Usage - Ashes of the Singularity:  DirectX 12 CPU Load - Ashes of the Singularity When we ran the DirectX 12 version of the benchmark we saw that the CPU load dropped to 50% and that the test systems power consumption at the wall was down by about 30 Watts . DX12 is supposed to better leverage all your processors available CPU threads to maximize performance instead putting the user-mode driver, DXGKernel, and Kernal-mode driver all on a single thread. By distributing the load across multiple threads and removing the kernel-mod driver from the workflow, it should help in-game performance by getting what the GPU needs to render the frames faster. GPU limited game titles won't see huge performance improvements, but CPU limited game titles should see solid improvements. DirectX 11 Driver Overhead - Ashes of the Singularity:  DirectX11 Driver Overhead - Ashes of the Singularity After the built-in game benchmark finishes running it gives you a benchmark graph that shows the driver overhead during the course of the benchmark run in orange. As you can see by clicking the image above the driver overhead is present and it differs in the various scenes over the three minute long benchmark when run in DirectX 11 mode. DirectX 12 Driver Overhead - Ashes of the Singularity:  DirectX 12 Driver Overhead - Ashes of the Singularity When running the DirectX 12 version of the benchmark we see that the driver overhead drops to basically nothing and stays like that over the course of the benchmark run, which is exactly what we expected to see with DX12 enabled. We also see that we are basically 100% GPU limited even with the Intel Core i7-4960X running at 3.6 GHz with just four cores active and Intel Hyper-Threading disabled. This is what the developer has to say about the benchmark results:
Once the test has finished you will see a graph, which plots the values over the 180 second run. The driver overhead score is the percent of the total frame rate spent in driver, otherwise it is in FPS. In the breakdown screen, the top 3 scores are the meta marks of the shots, those are the most important numbers. If we want to look at Raw GPU performance, the overall score or Normal Score is good, if we want to look at D3D12 performance, we recommend using the FPS score on the Heavy mark, which is also what we recommend for using as a CPU score. In DX11, it is not possible to track some information like driver throughput or CPU score, for that reason, that data will not be provided during the benchmark DATA POINTS EXPLAINED Because we have more precise data with D3D12, we can provide some new data not provided before. The first data point, FPS is well known to everyone. Here is a list of what they mean:
  1. FPS. The average frames per second, averaged by the total runtime of the benchmark divided by the number of frames rendered
  2. Weighted FPS. Slow frames are weighted more than fast frames, with the equation being the square root of the sum of the square of all individual frame times.
  3. Percent GPU bound. The percent frames which were blocked by the GPU. If this number is near 100%, then a faster CPU would make no difference in the score
  4. CPU FPS. An estimated FPS based on how fast the system would have been if the system wasn’t blocked by the GPU. That is, what would happen if we put a much faster GPU in the system. This number may be the true delta between DX11 and DX12. That is, if DX11 is CPU bound, then this number will indicate how much faster DX12 would be if we had GPU fast enough to keep up. This field is actually very accurate measure of CPU performance, especially on the “Heavy Batches” sub-benchmark.
  5. Batches per ms. Also known as draw calls, this number includes compute dispatch as well. This tells us roughly how fast the driver is processing commands
Let's chart our results for you, so you can better see the differences. Ashes of the Singularity Benchmarks The test results with the EVGA GeForce GTX 960 SSC 4GB video card running at 1920 x 1080p with MSAA disabled was that DX11 actually had a higher average frame rate that DX12. It didn't matter if we had 4 threads or 12 threads on the Intel Core i7-4960X processor active. This is a bit shocking as when we looked at DirectX 12 performance improvements with Futuremark's API Overhead Feature Test we found that the higher the thread-count you had meant you had higher DirectX 12 performance. We checked with the PR firm that Stardock is using and they confirmed that they are seeing similar results. Using one Alpha benchmark as an indicator of how DirectX 12 gaming performance looks compared to DirectX 11 didn't feel too smart, so we stopped our testing here as we saw no point into spending more than a day benchmarking and writing this article up. Down the road Stardock plans on releasing new benchmark versions as the game matures and we can't wait to see how those are look! The day before the embargo lifted on these DX12 benchmark results we were told that Dan Baker, Co-Founder, Oxide Games put up a blog post addressing some of our DX12 concerns with his benchmark. He basically dismissed NVIDIA's claims that the Ashes of the Singularity Benchmark has application side MSAA issues and said that it's a video card driver issue and that AMD, Intel and NVIDIA have had access to this source code for over a year. Obviously, there is an issue somewhere, so we are comfortable with our decision to halt testing and wait for this all to blow over. You can read some the comments by Mr. Baker below.
 There are incorrect statements regarding issues with MSAA. Specifically, that the application has a bug in it which precludes the validity of the test. We assure everyone that is absolutely not the case. Our code has been reviewed by Nvidia, Microsoft, AMD and Intel. It has passed the very thorough D3D12 validation system provided by Microsoft specifically designed to validate against incorrect usages. All IHVs have had access to our source code for over year, and we can confirm that both Nvidia and AMD compile our very latest changes on a daily basis and have been running our application in their labs for months. Fundamentally,  the MSAA path is essentially unchanged in DX11 and DX12. Any statement which says there is a bug in the application should be disregarded as inaccurate information. So what is going on then? Our analysis indicates that any D3D12 problems are quite mundane. New API, new drivers. Some optimizations that that the drivers are doing in DX11 just aren’t working in DX12 yet. Oxide believes it has identified some of the issues with MSAA and is working to implement work arounds on our code. This in no way affects the validity of a DX12 to DX12 test, as the same exact work load gets sent to everyone’s GPUs. This type of optimizations is just the nature of brand new APIs with immature drivers. Immature drivers are nothing to concerned about. This is the simple fact that DirectX 12 is brand-new and it will take time for developers and graphics vendors to optimize their use of it. We remember the first days of DX11. Nothing worked, it was slower then DX9, buggy and so forth. It took years for it to be solidly better then previous technology. DirectX12, by contrast, is in far better shape then DX11 was at launch.  Regardless of the hardware, DirectX 12 is a big win for PC gamers. It allows games to make full use of their graphics and CPU by eliminating the serialization of graphics commands between the processor and the graphics card. I don’t think anyone will be surprised when I say that DirectX 12 performance, on your hardware, will get better and better as drivers mature.
NVIDIA DX12 Test Results NVIDIA did some testing with a GeForce GTX Titan X with an Intel Core i7 5930K processor and saw that performance drop across the board when moving from DX11 to DX12 with high image quality settings. When they dropped the image quality down (even though they were averaging nearly 40 FPS on the 1080p Heavy test) they found they could get performance improvement by moving to DX12 on the 1080p Heavy test. Instead of reducing the number of cores on the processor with this card the team over at NVIDIA dropped to clock speed of the Intel Core i7-5930K from 3.5 GHz to 2.0 GHz and saw huge performance gains by going from DX11 to DX12. This is all fine and dandy and shows that DX12 is working, but it's not realistic scenario for real world gamers. How many desktop gamers do you know that are running an Intel quad-core processor with clock speeds that low? It is really disappointing to see our results and then NVIDIA basically confirm them and then the test scenario that shows any benefit was to take a $1000 and put it on a 6-core processor and downclock the frequency to 2GHz in order to show that DX11 to DX12 performance gains can be had. nvidia-test-results2 The only other test NVIDIA showed was dual-core performance and they found that some nice performance gains could be seen for on a dual-core platform, but again, how many dual-core gamers are left out there? The Intel Pentium Processor G3258 was a somewhat popular recently released dual-core processor that came out over a year ago and is clocked at 3.2 GHz. You can read our G3258 CPU review & gaming performance articles if you are interested in how that processor performed. ashes DX12 gameplay We talked with NVIDIA and AMD about the benchmark and both noted that this is alpha software and we took that as they felt it might not be an accurate measurement of DX12 at this point in time. The benchmark basically tells you how your system hardware will run a series of scenes from the alpha version of Ashes of the Singularity and that is about it. After talking with NVIDIA, they made it clear that they feel that there will be better examples of DirectX 12 performance coming out shortly that don't have as many issues as we experienced in this benchmark. It's cool to see some DirectX game benchmarks coming out, but the results here are far from exciting and clearly there are performance issues that are related to either the video card drivers, application or a combination of both. These DX12 results are interesting to look at, but it's obvious that DX12 is new and everyone doesn't have it dialed in yet! In closing, we did find out that Stardock/Oxide will be releasing the standalone Ashes of the Singularity benchmark that the public will be able to download and use for free in a few weeks. By that time we have a feeling many of the 'issues' we ran into will be solved and we'll be able to look at performance on both AMD and NVIDIA graphics cards with the latest drivers and game build.