Damien Wyart on Wed, 30 Jul 2003 22:38:22 +0200

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Truncating real numbers


I need to truncate (and round) reals to keep only n digits after the
decimal separator. This is both for display and for evaluation. I would
like not to interact with default precision (this is quite heavy to
change it all the time).

That is :

if a is 2.7646535646854

I both need to output a with (say) 4 digits after ., and to store this
truncated (or rounded) number in a variable.

b = neededfunctiontrunc(a,4)

-> b now contains 2.7646

b = neededfunctionround(a,4)

-> b now contains 2.7647

I looked at the function precision, but it gives strange results ; the
behaviour doesn't seem to follow the doc ! Could someone check this ?

With default precision at startup, I get :

? a = 2.7646535646854
? b = precision(a,4)
%2 = 2.76465356

With isn't %2 2.7646 ??

Ok, I can do (notice I have to use '5' and not '4' so if the number is >
10, I have to ask for 6 significant digits) :

? \p 5
? a
%3 = 2.7646

But then, I have to set \p back for the next step if I want to remain
precise enough. And a is not containing exactly the number 2.7646, it is
only a display effect.

I am missing something ?

Many thanks if you can help. Truncating or rounding numbers after n
qdecimal places seems quite common when working with reals, so I am
surprised not to have found references to this problem in the archive of
the list.

Best regards,
Damien Wyart