Comparing Apples To Apples (The Myth Of Benchmarks)

I so love benchmarks. There supposed to give us an idea, albeit rough sometimes, about the differences in choosing one thing over another. By looking at the comparative numbers, we should be able to make an intelligent decision regarding raw performance. However, sometimes the tests are a bit skewed. Case in point, I came across a benchmark. Not just any old benchmark, but rather one that a company selling a C#->VB converter points to and uses it as the #1 reason as to why you should convert your code from VB to C#. Well, as luck would have it, you can actually download the code. First problem is that the two benchmarks for C# and VB are not as close as the author originally thought. In some ways he’s comparing apples to oranges.

I’m going to focus on just the C# vs VB portion of these tests. The first problem I saw was that (and as I’ve already pointed out in the past) that the VB version was not using streams for the IO testing. Utilizing the “built in” file i/o methods is like comparing a glider to a Humvee. You just can’t compare the two on any level. The built in methods support so much more than streams do, no wonder they are slower. If you want performance, you have no choice but to utilize streams. Streams have built in caching and are about as raw of an access as you can get. Pretty much a no-brainer. So I modified the IO method accordingly.

Next problem… and this one exists in both the C# and VB version. On my machine, there was about a 7+ second hit in performance due to the way the 64bit integer tests were being ran. For the most part, it made it appear that using 64bit integers was over 3 times slower than using 32bit integers. Doubling the bits triples the hit? That didn’t seem right to me. The issue is that the calculations were constantly overflowing due to the seed values. C# by default allows this to happen without any error and the overflow causes the value to loop. VB has overflow checking on by default, so this method would actually crash. You can enable/disable this at the project level in both languages and C# allows you to wrap areas in question using the checked keyword. I modified this method so that overflowing was not occurring to better match the Int32 test.

One huge mistake I see many people make with VB is to not utilize Option Strict On and Option Explicit On. The end result is a slight hit in performance. Turning these on, according to the VB team, will improve overall performance. How much? Don’t really know, but at the end of the day, turning these on helps in the overall development, debugging and maintenance of your applications… and you might get improved performance… so turn them on.

The end result was actually somewhat surprising. Compiling both of these in release mode, turning off overflow checking and enabling strict type checking… the VB version seems to perform better nearly every single time. Not by much mind you, but better nonetheless. My goal was not to show VB running faster, just to correct some of the issues I saw between the two samples to get the comparison to be on par with one another so who would have thought the results would be what they were.

Here’s a screen shot of a typical run…

So how about them apples?

Note: And if you haven’t figured this out yet… don’t trust someone else’ tests! Try them out for yourself. Validate them based on your type of development. Yes, every test I run using this code shows (and I’m having to pinch myself) that VB is faster. However, understand that this is just based on these types of tests. I’ve made every attempt to compare apples to apples, but that’s not to say I missed something either. Benchmarks, like statistics, are essentially presenting you a story using numbers that usually reflects the answers (meaning viewpoint) that the source of the data is trying to sway you towards.