So I'm just going to dive into this issue... I've got a heavily used web application that, for the first time in 2 years, failed doing an equality check on two doubles using the equality function a colleague said he'd also been using for years.
The goal of the function I'm about to paste in here is to compare two double values to 4 digits of precision and return the comparison results. For the sake of illustration, my values are:
Dim double1 As Double = 0.14625000000000002 ' The result of a calculation Dim double2 As Double = 0.14625 ' A value that was looked up in a DB
If I pass them into this function:
Public Shared Function AreEqual(ByVal double1 As Double, ByVal double2 As Double) As Boolean Return (CType(double1 * 10000, Long) = CType(double2 * 10000, Long)) End Function
the comparison fails. After the multiplication and cast to Long, the comparison ends up being:
Return 1463 = 1462
I'm kind of answering my own question here, but I can see that
double1 is within the precision of a double (17 digits) and the cast is working correctly.
My first real question is: If I change the line above to the following, why does it work correctly (returns
Return (CType(CType(double1, Decimal) * 10000, Long) = _ CType(CType(double2, Decimal) * 10000, Long))
Decimal have even more precision, thus the cast to Long should still be
1463, and the comparison return
False? I think I'm having a brain fart on this stuff...
Secondly, if one were to change this function to make the comparison I'm looking for more accurate or less error prone, would you recommend changing it to something much simpler? For example:
Return (Math.Abs(double1 - double2) < 0.0001)
Would I be crazy to try something like:
(I would never do the above, I'm just curious about your reactions. It would be horribly inefficient in my application.)
Anyway, if someone could shed some light on the difference I'm seeing between casting
Long, that would be great.
Relying on a cast in this situation is error prone, as you have discovered - depending upon the rules used when casting, you may not get the number you expect.
I would strongly advise you to write the comparison code without a cast. Your Math.Abs line is perfectly fine.
Regarding your first question:
My first real question is: If I change the line above to the following, why does it work correctly (returns True)?
The reason is that the cast from Double to Decimal is losing precision, resulting in a comparison of 0.1425 to 0.1425.