Precision of math calculations

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Precision of math calculations

    I was asked today, "You are reporting a 4-place decimal value, but what precision is the math calculation using to give you the result?".
    I believe I have read this somewhere but I have not been able to find it. I realize there are physical limitations to precision caused by the various CMM's but the algorithms in PC-Dmis should have a defined number length for its calculations.

  • #2
    The math is taken to the computer's floating-point double precision.
    Links to my utilities for PCDMIS

    Comment


    • #3
      Metric data shows up in Datapage as 6-place decimal.

      Now, we all know that is a pretty [email protected] small increment. I KNOW for a fact that the original scales on this CMM had a 0.0008" increment, I am not sure what the current scales have. How Pcdmis comes up with values 'in-between', I have no idea.
      sigpic
      Originally posted by AndersI
      I've got one from September 2006 (bug ticket) which has finally been fixed in 2013.

      Comment


      • #4
        Based on a random angle I measured I can count up to 15 decimal places if I take the result times something like a hundred billion zillion or so 10000000000000000000000000000000000000000000000.

        15 places is the Double floating point? For some reason I have 16 in my head .
        Mr. Comment
        SCIROCCO-NT 13-20-10
        B3C-LC Controller (Leitz Protocol), SP600M, TP200
        PCDMIS CAD++ v4.3 MR1(Build: 12/11/08)
        sigpic

        Comment


        • #5
          I think 32 places on a 32 bit system
          Links to my utilities for PCDMIS

          Comment


          • #6
            Thanks everyone for the enlightenment.
            My detractors are now seeking vengeance elsewhere.

            Comment


            • #7
              Originally posted by carpentj View Post
              Thanks everyone for the enlightenment.
              My detractors are now seeking vengeance elsewhere.
              It is just like Wack-a-mole, let 'em stick their head up.
              Links to my utilities for PCDMIS

              Comment


              • #8
                You might find this interesting:
                http://www2.hursley.ibm.com/decimal/dcspec.html
                Bill Jarrells
                A lie can travel half way around the world while the truth is putting on its shoes. - Mark Twain

                Comment


                • #9
                  Originally posted by Wingman View Post
                  You might find this interesting:
                  http://www2.hursley.ibm.com/decimal/dcspec.html
                  sounds unimplemented... new number representation, something like unicode for characters??
                  Links to my utilities for PCDMIS

                  Comment

                  Related Topics

                  Collapse

                  Working...
                  X