Rounding

dwskau's picture

I have been working on a display of weather data which displays number values. These values come in from a weather station as a string. From there I have a basic understanding of what happens to them but I do not know all of the actual mechanics for what happens (that is the part that I am hoping some one here can shed some light on). The strings get converted to an integer or floating point value and then rounded, converted to a string, and displayed to the screen. Sometimes the rounding is wrong though.

In the example I have attached the values go through a unit conversion and the resulting value is the one that is rounded incorrectly. I assume this is because floating point numbers just aren't always accurate. Is there anything that I am missing or some way to round these values correctly?

PreviewAttachmentSize
rounding problem.qtz7.51 KB

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

cwright's picture
truncation

this looks like a mix between truncation, and rounding.

there's no floating-point rounding control in QC, so it's rather difficult to do.

Javascript seems to round differently from the round patch as well.

try this javascript patch after the "km/100" patch:

function (__string outputNumber) main (__number inputNumber, __number len)
{
   var result = new Object();
   result.outputNumber = inputNumber.toString().substring(0,len);
   return result;
}

make len something like 4 or 5, and it'll make it that long at most. However, it's not quite necessary since js rounds it properly to 9.2..... shrugs

dwskau's picture
Thanks!

Works great and super fast as usual!