in Education by
Debugging some finance-related SQL code found a strange issue with numeric(24,8) mathematics precision. Running the following query on your MSSQL you would get A + B * C expression result to be 0.123457 SELECT A, B, C, A + B * C FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A, CAST(0 AS NUMERIC(24,8)) AS B, CAST(500 AS NUMERIC(24,8)) AS C ) T So we have lost 2 significant symbols. Trying to get this fixed in different ways i got that conversion of the intermediate multiplication result (which is Zero!) to numeric (24,8) would work fine. And finally a have a solution. But still I hace a question - why MSSQL behaves in this way and which type conversions actually occured in my sample? JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

1 Answer

0 votes
by
Just as addition of the float type is inaccurate, multiplication of the decimal types can be inaccurate (or cause inaccuracy) if you exceed the precision. See Data Type Conversion and decimal and numeric. Since you multiplied NUMERIC(24,8) and NUMERIC(24,8), and SQL Server will only check the type not the content, it probably will try to save the potential 16 non-decimal digits (24 - 8) when it can't save all 48 digits of precision (max is 38). Combine two of them, you get 32 non-decimal digits, which leaves you with only 6 decimal digits (38 - 32). Thus the original query SELECT A, B, C, A + B * C FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A, CAST(0 AS NUMERIC(24,8)) AS B, CAST(500 AS NUMERIC(24,8)) AS C ) T reduces to SELECT A, B, C, A + D FROM ( SELECT CAST(0.12345678 AS NUMERIC(24,8)) AS A, CAST(0 AS NUMERIC(24,8)) AS B, CAST(500 AS NUMERIC(24,8)) AS C, CAST(0 AS NUMERIC(38,6)) AS D ) T Again, between NUMERIC(24,8) and NUMERIC(38,6), SQL Server will try to save the potential 32 digits of non-decimals, so A + D reduces to SELECT CAST(0.12345678 AS NUMERIC(38,6)) which gives you 0.123457 after rounding.

Related questions

0 votes
    When I convert a factor to a numeric or integer, I get the underlying level codes, not the values as numbers. f...
asked Feb 1, 2022 in Education by JackTerrance
0 votes
    In order to create the proper queries I need to be able to run a query against the same datasource ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Mar 23, 2022 in Education by JackTerrance
0 votes
    This one will take some explaining. What I've done is create a specific custom message queue in SQL ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Mar 16, 2022 in Education by JackTerrance
0 votes
    I try to create database from entity framework code first follow with this tutorial http://www.asp.net/mvc ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Mar 2, 2022 in Education by JackTerrance
0 votes
    I'm trying to achieve the same amount of precision (number of digits after decimal point) for results ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 16, 2022 in Education by JackTerrance
0 votes
    The benefit of the _____ function is that it is the only way to perfectly repressed an R object in an ... Operations of R Programming Select the correct answer from above options...
asked Feb 13, 2022 in Education by JackTerrance
0 votes
    Numbers in R are generally treated as _______ precision real numbers. (a) single (b) double (c) real (d) ... and Out of R Programming Select the correct answer from above options...
asked Feb 13, 2022 in Education by JackTerrance
0 votes
    I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification ... labels? Select the correct answer from above options...
asked Jan 31, 2022 in Education by JackTerrance
0 votes
    Give an example where High Precision is not usable. Select the correct answer from above options...
asked Nov 12, 2021 in Education by JackTerrance
0 votes
    Calculate Accuracy, Precision, Recall and F1 Score for the following Confusion Matrix on Heart Attack Risk. Also suggest ... and why? Select the correct answer from above options...
asked Nov 12, 2021 in Education by JackTerrance
0 votes
    Write the equation and calculate the precision and recall rate....
asked Apr 27, 2021 in Technology by JackTerrance
0 votes
    What are precision and recall? In which cases are they used?...
asked Feb 5, 2021 in Technology by JackTerrance
0 votes
    This question already has answers here: Convert pandas DateTimeIndex to Unix Time? (7 answers) Closed 3 ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked May 26, 2022 in Education by JackTerrance
0 votes
    A ________________ in R programming language can also contain numeric and alphabets along with special characters like ... Programming Select the correct answer from above options...
asked Feb 16, 2022 in Education by JackTerrance
...