Bill Allombert on Tue, 4 Mar 2003 14:10:43 +0100

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: precision question

On Tue, Feb 18, 2003 at 08:22:02PM -0500, Igor Schein wrote:
> Hi,
> ? precision(511/10^3+.)
> 28
> ? precision(512/10^3+.)
> 38
> Looks liking adding 1 extra binary place pushes the precision to the
> next level.  I'm curious in principle, why rational-to-float
> conversion of a small rational results in higher-than-working
> precision. 

The "correct" way of converting rationals and intergers to reals is to
do: N*1., not N+0. 

N*1. return N with the default precision, when N+. return N with
precision sizedigit(N)+defaultprecision.

If you see reals numbers as range, this is logical:
If the error is eps=10^-prec, x. stand for [x*(1.-eps),x*(1.+eps)], and
0 stand for [-eps,+eps].

N*1. give us [x*(1.-eps),x*(1.+eps)]=N.  but N+. give us 
[N-eps,N+eps]=[N*(1-eps/N),N*(1+eps/N)] so we have now an error
of only eps/N, so a precisio of prec+log(N)/log(10).