Welcome to AddressOf.com Sign in | Join | Help

Comparing Apples To Apples (Revisited)

Like I stated, don't trust someone elses benchmarks. ;-)

Don't get me wrong, in my testing, VB still seems to be a hair faster; but now only by a very slim margin (less than 400ms across all tests).  Please keep in mind that I did not come up with this suite of tests; rather just doing my part to validate the results (since they original results didn't seem to make since to me in the first place).

After going to bed last night, I couldn't help to be bothered by the results I was getting.  How could something that is essentially running very similar IL... even if there are a few minor differences... could be significantly different?  This morning I realized that I wasn't comparing apples to apples. ;-)

The original C# version contained the following code in the test loop for the 32bit integer test:

intResult -= i++;
intResult += i++;
intResult *= i++;
intResult /= i++;

So to translate this to functionally equal code in VB, you'd likely have the following:

intResult -= i : i += 1
intResult += i : i += 1
intResult *= i : i += 1
intResult \= i : i += 1

There the same, right?  As it turns out, there are significant differences between them in IL.  By changing the methods using this pattern of code in the C# benchmark, there was a measurable performance increase.  Looking at the IL you'll see that using the original code the dup operator (which copies the current topmost value on the evaluation stack, and then pushes the copy onto the evaulation stack) is being utilized; whereas the modified code works directly on the value in the stack.

The other flaw in my apples to apples was that I left the C# code alone for the IO testing.  In VB I utilized a For/Next loop instead of a while loop and adding to an integer value until it reached the maximum number.  I modified the C# version to use the same as the VB version.

After modifying these, the tests are much closer together in regards to performance and look more like what I was originally expecting.

As Jason asked, I've attached the code so the skeptics can do as I've done... play with it and see what you can find.

[download]

Published Wednesday, November 15, 2006 1:31 PM by CorySmith
Filed under: ,

Comments

# re: Comparing Apples To Apples (Revisited)

Wednesday, November 15, 2006 11:25 AM by Mike Dunn

Now your comparing Candy Apples to Apples.

I'll fix you C# code for you so it is a real equivlant to your VB.Net later today.

But one example off the Bat is you are using the "using" statement in your VB code but not in your C# code for the IO stuff.

Vb.Net

Using streamReader As New StreamReader(fileName)

Should Be in C#

using (StreamWriter streamWriter = new StreamWriter(fileName))

Does make a difference.

# re: Comparing Apples To Apples (Revisited)

Wednesday, November 15, 2006 12:53 PM by Wesner P Moise

Generated code for the IntArithmetic test becomes identical for C# and VB if you replace the intMax parameter an integer not long in VB. The parameters are not consistent between the two languages. Also the accessibility should be set to both public. It probably doesn't make a difference, but accessibility does dramatically impact calls through reflection. With the changes above, C# outperforms VB in the IntArithmetic tests.

I haven't check the other tests yet.

# re: Comparing Apples To Apples (Revisited)

Wednesday, November 15, 2006 1:40 PM by Wesner P Moise

The IL in the IntArithmetic and DoubleArithmetic case are identical, except for the ordering of the locals and DateTime calculations.

In C#, unlike VB, the order that the locals are declared in IL matches the order in the source code. The order is important because earlier primitive variables are enregistered. By declaring the numeric variables in C# earlier, but initializing after the timer, differences in timings are erased.

Another issue is the use of DateTime.Now, which have different behavior in VB and C#. You should use the System.Diagnostics.Stopwatch class, which is much easier to use and also far more precise.  watch.Start(), watch.Stop() and watch.ElapsedMilliseconds.

I looked at removing differences in class declaration: C# was made into a static class to match VB's module declaration. C# uses beforeFieldInit which might have an impact, since you are setting and reading a static field; this can be turned off by adding a static constructor; but I can't really see why it would it be significant enough to impact the benchmarks.

In the trig case, C# was more aggressive in reusing variables space, so it only allocates 2 locals, rather than the 8 that VB does.

# re: Comparing Apples To Apples (Revisited)

Wednesday, November 15, 2006 2:29 PM by CorySmith

Making a few of the modifications you suggested and running the test again... the results are even more what I would have expected.

(Again, to be clear, this is code that I picked up from somewhere else, so some of the "mistakes" seen were not my initial doing.) ;-)

Start C# benchmark

Int      : 7319

Double   : 2174

Long     : 12167

Trig     : 1421

IO       : 5164

Total C# : 28245 ms

Start VB benchmark

Int      : 7313

Double   : 2177

Long     : 12151

Trig     : 1410

IO       : 5238

Total VB : 28289 ms

The numbers are so close now that the ultimate point I was attempting to make now resonates.  They are essentially equal in performance; the differences can easily be attributed to other processes occuring on the machine, garbage collection, caching, etc.  If anyone is interested, I'll be happy to upload the modified code.

Also, I received a few messages via IM asking why I'm spending my time on this subject.  I spent over an hour trying to explain to a Amazon.com evangelist that VB is not slower than C# and that C# developers are not automatically "better" than VB developers.  VB does not promote bad code; crap code can be written in any language and VB does not have a monopoly in that regard.  I even went so far as to state that there are many, many, many C# developers that are actually VB developers hiding in the land of C# since they believe that by playing in the C-type language arena they'll a) make more money, b) be more respected by their peers and c) have a brighter future.  I don't have a problem with any of this; however, I do have a problem with people propagating myth and conjecture without backing it up with evidence just so they feel better about their language choice.

Are these tests conclusive? Not by any means. Which was the original point I was trying to make.  At the end of the day it's all IL.  Sure, some compilers may write slightly different IL; but it's IL nonetheless running within the same type of environment.  If you encounter what appears to be a bottleneck, no matter what type of language you choose to develop in, you'll need to optimize accordingly and the language using is irrelevant.

# re: Comparing Apples To Apples (Revisited)

Wednesday, November 15, 2006 2:51 PM by CorySmith

For what it's worth, I made one more modification to the tests.  Since we are testing the languages interaction with I/O and both languages are utilizing the same subsystem... access the physical drive is not really necessary.  I modified the routine to utilize a memory stream with default initialization and increased the test iteration to 5 million.

# C# - So Easy Even A 10 Year Old Can Get an MCAD

Thursday, November 16, 2006 6:01 AM by DonXml's All Things Techie

I'm not going to get into the whole VB or C# is faster debate (because it really doesn't matter since

Anonymous comments are disabled