Bill Allombert on Thu, 22 Feb 2024 11:44:02 +0100


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: Numerical instability in extracting base B digits of the fractional part of real


On Thu, Feb 22, 2024 at 09:13:31AM +0200, Georgi Guninski wrote:
> Some observations:
> 1. With precision 100 I get correct result.

Sure, this is due to base 2 to base 10 conversion.
33/100 cannot be represented exactly in base 2,
so depending of the precision it will be represented either as
33/100 + epsilon 
or 33/100 - epsilon
in the first case, you will get 0,3,3,0,0,0,0,...
and in the second you will get 0,3,2,9,9,9,9,9

But you can obtain the correct result by using 33/100 instead of .33

Cheers,
Bill.