When Doom showed up back in May, I ran a bunch of benchmarks to see how it performed. At the time, we were promised that a patch with support for the Vulkan API would show up "soon after launch," which apparently meant around two months. The good news is that after all this waiting, the public Vulkan patch went live last week; there's also a FAQ on Doom's Vulkan support that has additional information.
So what is Vulkan and why should anyone care? The short summary is that Vulkan is the cross-platform low-level API put out by the Khronos Group, the same group that handles the cross-platform OpenGL API. Alternatively, Vulkan is to OpenGL as DirectX 12 is to DirectX 11. The key element we want to discuss is what it means to be 'low-level' and how that changes the game engine. This gets technical, but it will help set the stage for what we see in the benchmarks.
So, what is an API?
Software developers typically use programming libraries to help make their jobs easier. Imagine you want to create a game for Windows; there's a lot of work involved in doing so, but many common tasks can be handled by a programming library an API, or Application Programming Interface. Rather than reinventing the wheel for each new program, tasks like graphics, audio, window resizing and positioning, reading and writing to storage, and more can simply use an existing library to make life easier.
Focusing specifically on the realm of graphics, the API helps handle things like texturing, lighting, and creating all the amazing visuals we see in modern games. But using a library also involves some level of abstraction, and in the world of graphics it means you have a driver that supports the set of functions in the library, and it maps those to the actual hardware. While AMD and Nvidia GPUs as an example might be similar in many areas, dig deep enough and there are plenty of differences.
In the past decade or two, most graphics APIs have been 'high-level,' meaning there's a larger amount of abstraction. This generally makes the job of the programmers easier, at the cost of some performance optimizations. Some developers have wanted ways to extract more performance from the hardware, however, and they've basically asked for 'low-level' access to the hardware. That means more work in some cases, but it can also improve performance if you know what you're doing. And that brings us to Vulkan and DirectX 12.
There are plenty of differences, just as there are differences between OpenGL and DirectX 11. Microsoft is in charge of the DirectX world, and it only works on Windows platforms; Khronos Group handles OpenGL/Vulkan, and they support multiple platforms including most smartphones and tablets via OpenGL ES. Where Microsoft had pressure from game developers and hardware companies to create DirectX 12, Khronos Group took a different route and leveraged much of AMD's work on their own low-level Mantle API to create Vulkan. Ultimately, the end goal is the same: allow developers to extract more performance from the hardware (if they want to put in the effort).
There's some politics involved with the low-level API discussion as well. The biggest item is that AMD supports a feature called asynchronous compute, which is basically the ability to mix and match graphics and compute instructions in the execution units. AMD has had their Asynchronous Compute Engine (ACE) as part of their core graphics hardware since the very first GCN GPUs (the HD 7970 and 7950 launched in January 2012), but until Mantle, DirectX 12, and Vulkan came around it didn't actually do a lot. That's because DirectX 11 and OpenGL didn't really have a good way to leverage the ACE, but low-level access changes things.
Spec sheets don't tell the whole story
Fundamentally, AMD and Nvidia architectures aren't the same, and this is why we run benchmarks. Otherwise we could just look at the specifications and say, "Oh, it looks like AMD's Fury X does 8601 GFLOPS and has 512GB/s of bandwidth; the GTX 980 Ti has 6054 GFLOPS and 336GB/s bandwidth. That makes the Fury X 40-50 percent faster." The reality is that all processors have a theoretical performance, but actually getting close to that figure can be difficult, and the specifics of the architecture help determine how close the real world is to the theoretical world. And this is where ACE can help AMD quite a bit.
I picked the Fury X and GTX 980 Ti for a good reason: on paper the Fury X should be substantially faster, but in practice the 980 Ti ends up with a small lead of around five percent (the Fury X does lead by five percent at 4K, however). That's about a 40 percent difference from the theoretical performance, due to the ways in which Nvidia's Maxwell and AMD's 3rd generation GCN differ. With a low-level API, AMD's ACE has the potential to help better utilize certain resources, particularly if the developers spend some effort to better optimize their code. Instead of utilizing 60-70 percent of the available execution units, they might be able to get to 80-90 percent utilization, and that could make a big difference when it comes to the end user's experience.
Now combine the above discussion with AMD's presence in all the current generation of consoles, and AMD has a vested interest in finding ways to improve the performance of their hardware. This is arguably why they created Mantle, and why they're so interested in DirectX 12. But couldn't AMD accomplish something similar by simply spending a lot of resources to optimize their DirectX 11 drivers? Probably not to the same degree, simply because the API isn't designed for things like their ACE.
What about Nvidia don't they have features and hardware elements that are handicapped by high-level APIs? Probably, though Nvidia for their part has been more interested in creating other gaming libraries rather than specifically focusing on low-level APIs. Certainly developers can do stuff with the PhysX API that they can't do using plain DirectX 11 though much of PhysX could almost certainly be done in other ways. They support Vulkan and DirectX too, and helped id Software with the Vulkan port of Doom.
With all of the above out of the way, let's talk expectations. In my view, the goal of any developer using a low-level API should always be to beat the performance they can get via a high-level API; if performance is worse (than OpenGL or DirectX 11), then it represents a lot of wasted effort.
Imagine someone coming to you with a customized sports car; they brag about replacing the engine, tweaking the transition, and doing all sorts of other work. Then you ask them how much better it performs compared to the stock car. If all their changes resulted in less horsepower, worse handling, a lower top speed, and reduced acceleration, you'd probably think they had lost their mind. On the other hand, they might tune for one or two specific areas at the cost of others so a higher top speed, or better acceleration which we also see in software development.
On the next page: Doom Vulkan performance numbers and charts galore.
Doom Vulkan performance comparisons
PC Gamer's Graphics Card Test System
CPU: Intel Core i7-5930K @ 4.2GHz
Mobo: Gigabyte GA-X99-UD4
RAM: G.Skill Ripjaws 16GB DDR4-2666
Storage: Samsung 850 EVO 2TB
PSU: EVGA SuperNOVA 1300 G2
CPU cooler: Cooler Master Nepton 280L
Case: Cooler Master CM Storm Trooper
OS: Windows 10 Pro 64-bit
Drivers: AMD Crimson 16.7.2, Nvidia 368.69/.64
For the benchmarks, I'm using my standard testbed, which you can see to the right. I didn't do a full set of additional CPU scaling tests this round, due to time constraints, but I did a few quick spot checks that I'll discuss below. I've also retested every single graphics card in the charts, with OpenGL 4.5 and Vulkan, using the AMD Crimson 16.7.2 drivers and the Nvidia 368.69 drivers the exception being the recently launched GTX 1060, where I used the 368.64 drivers. Some of the OpenGL numbers have improved a bit since the initial testing in May, so I wanted to keep things consistent.
For the charts, I've separated each resolution into AMD and Nvidia hardware, and colored Vulkan results red and OpenGL results blue. (Trying to put everything into one chart just resulted in a massive pile of results that ends up not being particularly readable, though if you're interested, here are the full 1080p, 1440p, and 4K results.) I'll discuss each chart in turn, starting at 1080p. I could have also tested at 1080p High or Medium quality, but the gains in performance aren't all that great and at some point I had to draw the line and simply finish the testing.
One thing to note is that my Doom benchmark sequence uses FRAPS for OpenGL and PresentMon for Vulkan, and there is a bit more variance between benchmark runs than some other games. Basically, I play a 150 second sequence (give or take) where I run though a section of the game and fight a bunch of hellspawn. I do my best to run the same path each time, but there are minor variations depending on where the demons show up. Testing the same card at the same settings multiple times gives a variance of around four percent, so anything less than that can be considered equivalent performance not that you'd really notice anything below ten percent anyway.
In case you're new to my testing, I also report minimum frame rates, but these aren't strict minimums. Instead, I calculate the average fps for the bottom three percent of frames. This is done by finding the 97 percentile (the value where 97 percent of frames render faster, using the frame times) and then selecting the remaining three percent of frames and finding their average fps (total number of frames divided by seconds). I do this because sometimes there's a single frame out of thousands of frames that 'glitched' or whatever, and in practice it's far more useful to know what typical minimum fps is like rather than focusing on that one frame.
AMD starts off with an impressive showing at 1080p Ultra, with nearly all of the cards showing around a 30 percent performance improvement compared to OpenGL. That's a massive jump, and what's more it happens even on lower-end hardware like the R9 380 not that any of these are 'slow' GPUs, but it's good to see improvements on more than just the high-end parts. The one exception here is the R9 385 2GB, which is the only AMD card with less than 4GB VRAM that we tested; it improves, but by roughly 10 percent instead of 30 percent. Minimum fps is also up significantly, though the nature of testing creates a lot more variation between runs when talking about minimum fps; the 285 shows almost no change, while the other cards are all 30-40 percent higher on 97 percentile scores.
In contrast to AMD's hardware, Nvidia's showing with Vulkan is a lot more mixed. The GTX 1080 performance goes up by 20 percent, the 1070 improves by 10 percent, and most of the remaining cards fall within the margin of error. We do see a few performance regressions, and the GTX 950 shows a nearly 10 percent drop with Vulkan. The 2GB 960 also shows a small drop, and interestingly the 1060 does as well perhaps the 368.64 drivers aren't fully baked, or it might just be variance between runs. Vulkan does generally improve minimum frame rates, and only the 950 shows a clear drop in minimum fps. The 1060 has a small drop, while other cards show a 10-20 percent improvement the 1080 even shows a 50 percent improvement, and I should note that the 1080 is frequently hitting the 200 fps frame rate cap that Doom imposes, so potentially it could run faster.
Pulling back to talk about the entire spectrum, Nvidia's 1080 and 1070 still claim the top spots, with the Fury X effectively tying the 1070. The 980 Ti, Fury, and Nano all cluster together, followed by another grouping of the 980, 480, and 390 in that order. Vulkan basically elevates AMD's GPUs from seriously lackluster positioning in Doom to being right in the mix.
Again, AMD delivers some massive improvements at 1440p, showing that Vulkan isn't just a case of reducing CPU bottlenecks but it's also allowing the software to extract additional performance from the hardware. Average frame rates are up by 20-30 percent on everything, this time including the R9 285 2GB. Minimum frame rates show similar and even slightly higher improvements, ranging from 15-35 percent.
Running a game like Doom at 1440p Ultra usually makes the GPU the bottleneck, and that's basically what we're seeing. The GTX 1080 does get a decent 10 percent boost, but all the other cards are pretty much within the margin of error. The 2GB VRAM cards also tend to be a bit more prone to hiccups in testing, or simply have higher variance; here the 960 2GB drops by nearly 10 percent, but the 950 (also 2GB) shows a barely perceptible dip. Minimum frame rates for all of the cards are within the wider margin of error, so basically at 1440p none of Nvidia's cards beyond the GTX 1080 really benefit.
The lack of improvement on the Nvidia side ends up changing overall positioning a bit. The 1080 remains the fastest GPU, but the Fury X now edges past the 1070 this isn't really that unusual, however, as the Fury X has gobs of memory bandwidth and frequently improves relative to the competition at higher resolutions. The Fury edges past the 980 Ti, which is just ahead of the Nano. Similarly, the 390 and 480 edge past the 980, 1060, and 970. At the bottom of the overall standings, things are pretty much in line with what we usually see: 380X beats 380, and both beat the 960 and 950.
Finally, with settings at maximum (more or less we're still not running the Nightmare quality options), AMD continues to show moderate improvements thanks to Vulkan. This time the range is more like 15-25 percent, for both average and minimum fps, but the fact that Vulkan is giving the GPUs any help at all at 4K ultra is frankly astonishing. I'm not sure what bottleneck is being alleviated, and all of the cards still fall below the 'magical' 60 fps mark, but if you have a 4K FreeSync display Doom is definitely playable at these settings on the RX 480 and above.
Nvidia on the other hand ends up with slight drops in performance on nearly all the cards we tested. I'd call it bad luck or margin of error, except it's so consistent that more likely there is a slight performance hit at 4K ultra with Vulkan. Either Nvidia's OpenGL drivers are simply so tuned at this point that it's hard for game developers to match them, or some other factor is at play. Of course, the GTX 1080 is still the only GPU to break 60 fps at 4K ultra, even if Vulkan doesn't help out.
The overall picture once more shifts farther into AMD's court, particularly on the Fiji cards with their HBM VRAM. Under OpenGL, Nvidia was the undisputed champion at Doom, but Vulkan tilts the scales to generally favor AMD hardware at least in terms of a bang for the buck.
On the next page: What this tells us about the current state of low-level APIs.
Does your CPU matter?
One of the things I've been interested in testing is whether low-level APIs will reduce the CPU requirements. That's what I thought they would do back when DX12 was first being discussed after all, Xbox One and PlayStation 4 both get by with CPUs that are downright pathetic compared to a modern Intel Core processor.
I did some limited testing and found that while the GTX 1080 still showed good 1080p scaling, Doom simply doesn't hit the CPU that hard. Performance with a simulated Core i3-4350 (dual-core with Hyper-Threading at 3.6GHz) in most cases is within spitting distance of our standard i7-5930K OC (six-core with Hyper-Threading at 4.2GHz). Even slower CPUs might change things a bit, but an i3-4350 is reaching pretty far down the totem pole.
Welcome to the low-level API future
Doom + Vulkan doesn't really change the overall perspective on the market; it's just one game, and there are plenty of games that continue to favor Nvidia's hardware. But if every game started to support DX12 or Vulkan, we're now reaching the point where the pattern is becoming pretty clear.
Doom is about as close to a vendor agnostic game as I expect to see, given both AMD and Nvidia have previously touted the performance improvements Vulkan brings to the table. Hell, Nvidia even had Doom come to their GTX 1080 preview party to show off Vulkan support. It's interesting that 1080p ended up being the only place where Nvidia showed major gains, and even then it was mostly on the 1080.
I talked earlier about theoretical GFLOPS (billions of floating point operations per second). GFLOPS is a good indication of theoretical performance, and many compute-oriented GPU calculations tend to scale well with these figures. Gaming performance on the other hand often ends up being quite a bit different. You can't usually sustain the peak GFLOPS of an architecture (it's the number of cores times the clock speed times two for both AMD and Nvidia GPUs right now, if you're wondering), but high-level APIs often don't extract maximum performance from the hardware, and certain architectural design decisions can exacerbate that problem. Vulkan in the case of Doom, as well as some other DX12 titles like Ashes of the Singularity and Hitman, seem to be moving AMD's performance closer to their theoretical level.
There's still the possibility that AMD helped id Software optimize for Vulkan more/better than Nvidia did, but I think it's equally likely that Nvidia's DX11 and OpenGL drivers are simply really good at utilizing most of the hardware resources and are perhaps hitting memory bandwidth bottlenecks at higher resolutions. Until we see a DX12 or Vulkan game where Nvidia hardware shows across the board performance improvements, that's my conclusion. That doesn't mean I think the Fury X is going to surpass the GTX 1080 in gaming performance, but if future games that utilize low-level APIs continue down this path, the positions of some other GPUs may change.
Of course by then, we might all be running Nvidia 'Volta' and AMD 'Navi' GPUs. And we still need developers to work on eliminating the 2-3 month lag time between launch and good DX12/Vulkan patches.