|
|
| GeoCommunity Mailing List |
| |
| Mailing List Archives |
| Subject: | GISList: ArcView3x: math malfunction |
| Date: |
07/30/2003 06:25:01 AM |
| From: |
=?iso-8859-1?Q?S=F6ren_Alsleben?= |
|
|
Just FYI
While calculating and relating attributes of a feature table using some
Number-Requests, I encountered a strange phenomenon:
10/3 is NOT equal to (1/3) * 10
Ok, I always guessed that my math teacher was a liar. But seriously,
this bug seems to be caused by some internal AV-limitations dealing with
numbers. Any value that consists of more than 16 digits (plus radix and
leading zeros for values ranged from 1 to -1) seems to be stripped after
the 16th digit. Then AV adds a final digit that usually is not a result
of traditional rounding.
Two scripts demonstrating this misbehaviour are appended. As far as I
know this error occurs on any plattform and AV version. Assuming that a
lot of AVs functions / requests should be based on similar arithmetics
makes me shiver...
Cheers,
S=F6ren
' --- SCRIPT1: START ---
' Precision of Numbers
n =3D -10
valstr =3D "10^n / 3"+nl
for each i in 1..30
val =3D (10^n/3).SetFormatPrecision (30)
valstr =3D valstr+nl+(10^n).AsString+" / 3 =3D "+val.AsString
n =3D n + 1
end
n =3D -10
valstr =3D valstr+nl+nl+nl+"(9 * (10^n)) / 10"+nl
for each i in 1..30
val =3D ((9 * (10^n)) / 10).SetFormatPrecision (30)
valstr =3D valstr+nl+((9 * (10^n))).AsString+" / 10 =3D "+val.AsString
n =3D n + 1
end
MsgBox.Report (valstr, "Precision of Numbers")
' --- SCRIPT1: END ---
' --- SCRIPT2: START ---
' Relating Numbers
val1 =3D (10 / 3)
val2 =3D (1 / 3) * (10)
if (val1 =3D val2) then
aBool =3D true
else
aBool =3D false
end
msgbox.info (val1.AsString+" =3D "+val2.AsString+nl+"is
"+aBool.AsString.UCase, "Relating Numbers: 10/3 =3D (1/3)*10 ?!?")
' --- SCRIPT2: END ---
To unsubscribe, write to gislist-unsubscribe@geocomm.com
________________________________________________________________________
GeoCommunity GeoBids - less than $1 per day!
Get Access to the latest GIS & Geospatial Industry RFPs and bids
http://www.geobids.com
Online Archive of GISList (and numerous others) available at:
http://spatialnews.geocomm.com/community/lists/
Setup a GeoCommunity Account and have access to
the GISDataDepot DRG & DOQQ Catalog
http://www.geocomm.com/login.php
|
|
Sponsored by:
For information
regarding
advertising rates
Click Here!
|
