Peter Bruin on Wed, 12 Feb 2014 13:24:01 +0100


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: precision of result when adding t_REAL with lots of cancellation


Hi Jeroen,

>> Would it be reasonable to round the precision _down_ in such cases?
>
> If you would that, then you would very quickly end up with a precision
> of 0 bits after a few such additions.

That is why I only suggest to do this if the precision is _extremely_
small.  If the precision of the result of an addition is 31 bits (in a
32-bit word) it doesn't hurt too much if you return this result with 32
bits of precision, but it's a different thing when you even do this when
the result has only 1 bit of precision.  I'm not proposing this as an
optimal solution and would be happy with any alternative suggestions.

Peter