|
|
| GeoCommunity Mailing List |
| |
| Mailing List Archives |
| Subject: | RE: GISList: ArcView3x: math malfunction |
| Date: |
07/30/2003 07:15:01 PM |
| From: |
Irving, David |
|
|
This is a direct consequence of the limited precision of computer arithmeti=
c, and is not exclusive to ArcView. You should get similar results from pro=
grams written in C, FORTRAN, BASIC, ... on any computer (the results will d=
iffer between computer architectures, but not generally between languages).=
There is an excellent treatment of this in Knuth, or in any elementary tex=
t on numerical analysis.
To repeat, it is _not_ a malfunction.
David Irving
Technical Systems Analyst
Innovation Team
Santos Ltd
91 King William St
Adelaide SA 5000
Ph : +61 8 8224 7810
Fax: +61 8 8218 5960
email: david.irving@santos.com
-----Original Message-----
From: S=F6ren Alsleben [mailto:GIS_mail@web.de]=0D
Sent: Wednesday, 30 July 2003 20:51
To: GIS-List
Subject: GISList: ArcView3x: math malfunction
Just FYI
While calculating and relating attributes of a feature table using some
Number-Requests, I encountered a strange phenomenon:
10/3 is NOT equal to (1/3) * 10
Ok, I always guessed that my math teacher was a liar. But seriously,
this bug seems to be caused by some internal AV-limitations dealing with
numbers. Any value that consists of more than 16 digits (plus radix and
leading zeros for values ranged from 1 to -1) seems to be stripped after
the 16th digit. Then AV adds a final digit that usually is not a result
of traditional rounding.
Two scripts demonstrating this misbehaviour are appended. As far as I
know this error occurs on any plattform and AV version. Assuming that a
lot of AVs functions / requests should be based on similar arithmetics
makes me shiver...
Cheers,
S=F6ren
' --- SCRIPT1: START ---
' Precision of Numbers
n =3D -10
valstr =3D "10^n / 3"+nl
for each i in 1..30
val =3D (10^n/3).SetFormatPrecision (30)
valstr =3D valstr+nl+(10^n).AsString+" / 3 =3D "+val.AsString
n =3D n + 1
end
n =3D -10
valstr =3D valstr+nl+nl+nl+"(9 * (10^n)) / 10"+nl
for each i in 1..30
val =3D ((9 * (10^n)) / 10).SetFormatPrecision (30)
valstr =3D valstr+nl+((9 * (10^n))).AsString+" / 10 =3D "+val.AsString
n =3D n + 1
end
MsgBox.Report (valstr, "Precision of Numbers")
' --- SCRIPT1: END ---
' --- SCRIPT2: START ---
' Relating Numbers
val1 =3D (10 / 3)
val2 =3D (1 / 3) * (10)
if (val1 =3D val2) then
aBool =3D true
else
aBool =3D false
end
msgbox.info (val1.AsString+" =3D "+val2.AsString+nl+"is
"+aBool.AsString.UCase, "Relating Numbers: 10/3 =3D (1/3)*10 ?!?")
' --- SCRIPT2: END ---
To unsubscribe, write to gislist-unsubscribe@geocomm.com
________________________________________________________________________
GeoCommunity GeoBids - less than $1 per day!
Get Access to the latest GIS & Geospatial Industry RFPs and bids
http://www.geobids.com
Online Archive of GISList (and numerous others) available at:
http://spatialnews.geocomm.com/community/lists/
Setup a GeoCommunity Account and have access to=0D
the GISDataDepot DRG & DOQQ Catalog
http://www.geocomm.com/login.php
Santos Ltd A.B.N. 80 007 550 923
Disclaimer: The information contained in this email is intended only for th=
e use of the
person(s) to whom it is addressed and may be confidential or contain
privileged information. If you are not the intended recipient you are hereby
notified that any perusal, use, distribution, copying or disclosure is stri=
ctly
prohibited. If you have received this email in error please immediately
advise us by return email and delete the email without making a copy.
To unsubscribe, write to gislist-unsubscribe@geocomm.com
________________________________________________________________________
GeoCommunity GeoBids - less than $1 per day!
Get Access to the latest GIS & Geospatial Industry RFPs and bids
http://www.geobids.com
Online Archive of GISList (and numerous others) available at:
http://spatialnews.geocomm.com/community/lists/
Setup a GeoCommunity Account and have access to
the GISDataDepot DRG & DOQQ Catalog
http://www.geocomm.com/login.php
|
|
Sponsored by:
For information
regarding
advertising rates
Click Here!
|
