 |
Services Offered this Issue
> The Appraiser Coach
|
Editor’s Note: This story is from the latest print edition of Working RE. Don’t miss the next issue!
(Am I a Working RE Subscriber?)
Why Fannie Mae Shouldn’t Keep Data Secret
By Jim L. Sanders & Teresa Huff
Is appraising science? In many ways it is. Believe it or not, the methods for studying science and residential appraising are closely related. Here’s how:
- A good scientific study involves research, data collection, analysis and testing of results to ensure validity and reliability. Conclusions are based on careful study and consistent methods.• A good residential appraiser spends time analyzing, comparing, and calculating the countless variables that contribute to a property’s value. Just as with a scientific study, the appraiser’s final opinion of value is not a number pulled out of a hat; rather, it is based on well calculated research and analysis.
In both science and appraising, others are looking to the expert for a well-grounded, educated conclusion. The professional develops an opinion based on education, experience, firsthand observation, and objective methods with controlled variables. In order to make a credible claim, this estimation must be supported with clear data.
The reverse is true as well; when there is a flaw anywhere in the process – data, methods, analysis, or otherwise – the results will not be accurate but rather an estimate based upon data collected and the method used to analyze the data. The only way to validate the study is to repeat the process to ensure correct models and data.
Appraisal Tools and Methods
Residential appraisers must provide a transparent basis for their opinion of property value when submitting an appraisal report to a lender. These appraisals are then sent to Fannie Mae for review and the underwriting process begins. Richard Hagar, SRA who teaches classes on the topic, notes that reviewers may select an underwriting program provided by Fannie Mae or other vendors.
The latest tool of choice, Collateral Underwriter™ (CU™), is “a proprietary appraisal risk assessment application developed by Fannie Mae to support proactive management of appraisal quality.” This tool allows Fannie Mae to use a proprietary statistical model for real estate analysis. The lender compares the appraiser’s adjustments against this model’s results to see if they agree. The output of the CU analysis is read by the reviewer, pointing them to areas that should be further analyzed by a human. After further scrutiny, including reading the appraisal report and comments, Hagar shares that the reviewer has four options:
1) Accept the original report as written,
2) Ask questions of the appraiser,
3) Ask for clarification or more detailed explanations, or
4) Reject the appraisal and order another.
Appraisers and other professionals are invited to learn more about CU through Fannie Mae’s training resources. They cannot, however, access CU. The software is only available to lenders and their underwriters.
(story continues below)
(story continues)
Access Denied
Today’s lenders have access to massive amounts of data. According to Hagar, government lenders have access to every recorded sale in most every county across the U.S. via information providers like CoreLogic. He believes appraisal adjustments should be cross-checked against sales metrics of the five million home sales that occur each year.
However, appraisers typically use Multiple Listing Service (MLS) for residential appraising. Data provided by MLS may vary from information shown by providers like CoreLogic. MLS has more detailed information that county records don’t contain. Oftentimes, though, MLS contains errors or missing data. In other words, both types of systems have errors, but not necessarily the same errors.
Increasingly, appraisers indicate that underwriters are rejecting appraisal reports for vague reasons based on ambiguous results from CU analysis. Problems arise when these methods are not accessible or replicable for others to validate results. Mr. Hagar emailed us his opinion:
Many appraisers do not understand how the CU system works. When there is a lack of understanding, people tend to reject the results and fight back. When the CU system spits out, “Your adjustments are different from what the model indicates or other appraisers in your area are using,” many appraisers don’t understand how the lender or CU arrived at this conclusion. What algorithm was used? What data were analyzed? All are legitimate questions that usually go unanswered.
By design, the CU output is only provided to the lender. The person reading the output is not supposed to simply cut and paste the comments into an email and send them to the appraiser. Unfortunately, many poorly trained people working for lenders and appraisal management companies do just that – cut and paste – without telling the appraiser, “Data indicates something different than what you indicated; how did you [the appraiser] determine the adjustment for x?” The CU isn’t necessarily telling the appraiser they are wrong; it is merely stating that their adjustments are different and therefore questionable.
Many appraisers fail to understand that they do indeed have access to the same sales data that the CU uses. Sales are sales, regardless of the storage method. The CU analyzes the sales data using their secret algorithm. Appraisers can use the same data along with a different method and produce a similar or different conclusion. For this reason, appraisers cannot guess at what an adjustment should be; they must have a valid method to determine the adjustment or their conclusions will be tested and questioned.
This brings us back to where we started; data must be properly analyzed using an acceptable, repeatable methodology. Otherwise, results are merely a guess.
(story continues below)
(story continues)
Transparency & Reproducibility
Transparency and reproducibility are key ingredients of good science and, we would argue, of appraising also. It requires that data and methods, including computer code, be made available. Whatever the information vessel – in science, appraisals, or otherwise – these elements are imperative to validate methods. Denying access to the methods used to form an opinion becomes a serious issue, such as with CU, when the results are viewed as “true.” Neither science nor appraising can be replicated on its own; people need the data to recreate the studies. They need the model to verify the methods.
This is from (Shuttleworth, M. Reproducibility. 14 June, 2009): “Reproducibility is regarded as one of the foundations of the entire scientific method, a benchmark upon which the reliability of an experiment can be tested. The basic principle is that, for any research program, an independent researcher should be able to replicate the experiment, under the same conditions, and achieve the same results.”
On the surface, CU guidelines align with these principles by promising, “Appraisers that make a good faith effort to use the most similar comparables, provide accurate and consistent data, and support their adjustments with market data and analysis can generally expect a minimum of CU feedback that would cause a follow-up request from the reviewer.” However, word on the street from appraisers shows quite the opposite. As mentioned previously, lenders are sending feedback such as:
- Do you have data to support this adjustment?
• We are asking that you take a class on how to support and prove your adjustments.
• The appraiser’s net adjustments for the comparable sales are materially different from the model net adjustments.
• The appraiser-provided comparables are materially different than the model-selected comparables.
How is the appraiser to understand feedback like this and understand the models when they aren’t made available? The process doesn’t lend itself to either transparency or reproducibility.
Fannie Mae emphasizes, “An accurate description of the physical condition and quality of the subject property is a critical element in arriving at a supportable opinion of market value, as well as in the prudent underwriting of a mortgage loan.” Lenders initially rely on appraisals to give an accurate firsthand description of the subject since the appraisers are, after all, on the front lines of evaluating properties. However, lenders later rely on CU feedback to tell them whether or not the appraiser’s eyewitness assessment is correct.
Collateral Underwriting
According to CU guidelines outlined by Fannie Mae, specific factors contribute to successful CU implementation. Among these are considering the restrictions of automated analysis, awareness of possible property or neighborhood nuances, and ensuring educated appraiser opinions are a priority over computerized results.
Fannie Mae’s guidelines expect lenders to “use human due diligence in combination with the CU findings.” They go on, “Lenders should not, however, make demands or provide instructions to the appraiser based solely on automated feedback.” However, this appears to be what is happening. Appraisers are accountable for conclusions driven by unknown models and an obscure algorithm for selecting sales to analyze. Their common-sense opinions are being overridden by the all-knowing system.
CU legalities forbid providing appraisers access to the CU interface or reports containing CU findings. Yet Fannie Mae claims, “CU is model-based and performs a more comprehensive analysis of data integrity, comparable selection, adjustments, and reconciliation.” If this were the case, according to basic scientific principles, then outside parties (appraisers) would be allowed access in order to verify the data and analysis.
In short, residential appraisers may be unfairly criticized and condemned for their conclusions and have no method for rebuttal. Potential discrepancies are issued based upon this model without the appraiser being fully informed of the basis for criticism; the appraiser has no opportunity to examine the questionable sales and explain why adjustments were made. No clarification is offered regarding why the appraiser’s opinions are contested. The process caters to underwriters, often leaving appraisers frustrated and unable to defend their work.
Shine the Light
Once again, CU models may be perfectly good- or they may not be. We have no way of knowing since Fannie Mae won’t share them. This leaves ambiguity whether appraisers are being treated fairly.
We must ask ourselves:
- Is the CU tool a one-size-fits-all method?
• If so, is the implementation in keeping with its own design of, “Well-informed human judgement should take precedence over automated results”?
• More importantly, do CU methods align with the basic principles of good science?
Until the models are made available we don’t have answers. A disconnect exists between Fannie Mae guidelines, lender interpretations and feedback to the appraiser. Meanwhile, appraisers must continue responding to lender requests and defending their conclusions as thoroughly as possible with the limited tools available to them.
> Just Published: OREP/WRE’s 2017 Fee Survey Results! To view the results in your state, click here. If you have not already taken the survey, please weigh in here.
> CE Online – 7 Hours (approved in 40 states)
How To Support and Prove Your Adjustments
Presented by: Richard Hagar, SRA
Must-know business practices for all appraisers working today. Ensure proper support for your adjustments. Making defensible adjustments is the first step in becoming a “Tier One” appraiser, who earns more, enjoys the best assignments and suffers fewer snags and callbacks. Up your game, avoid time-consuming callbacks and earn approved CE today! Sign Up Now! $119 (7 Hrs)
OREP Insured’s Price: $99
About the Author
Jim L. Sanders, MBA, is the owner of Real Estate Appraisal Litigation in Tucson, Arizona (RealEstateAppraisalAndLitigation.com). An appraiser since 1975, Sanders delivers expert witness testimony in state and federal courts. Teresa Huff, MS Ed., provides copywriting, editing, and development services to small businesses and solopreneurs. Find out more at www.AdeoDevelopment.com
Send your story submission/idea to the Editor: isaac@orep.org
by Baggins
Great article. Anticipating model differences from a blind position. As the author states; much of the data used is available to appraisers. It’s the blind comparison to recent appraisals on similar or same properties, not included in sow statements, which I’ve found to be the most difficult unexpected consequence of the CU system. “Appraiser must document why the value increased from (1 yr recent appraisal by another unknown appraiser), to (current appraised value opinion by myself)” Well! I can’t provide review services on a report I have not reviewed, and you can’t just expand sow to include blind comparison review against data systems to which I do not have access! They honestly don’t understand the conflict. Then lender turns to requesting statement of increased appreciation to cover instead. Same deal! I don’t see that, and if my value accelerated beyond the previous valuation point, I can’t provide a reason without first assessing the credibility of the previous report through review services. My SOW was to look at the current market. Underwriters are expanding scope to include validation or explanation of blind comparisons, rather than use their own exception tools to validate or pass reports as valid. The most frequent response I use is to simply copy and paste the 10/2016 fnma cu faq’s for underwriters, and direct them to page 3. What gets me is I’m asked to justify my current opinions in the blind against previous opinions which are presumed very accurate, simply because the data came from the grand home base CU system. Your value is only validated, once it’s dropped into the CU system, and the previous value could not have had errors, that’s the common presumption. If you say to the underwriter, well, that’s probably because the last appraisal valuation report your company helped originate just less than a year ago, probably was a 2 hour fly by night automated result appraisal which missed the mark, well, you’re not supposed to handle it that way. What a mess, how about winding down FNMA instead and returning a true free market absent of these moral hazards created by government interventionism?
-by John
I use CU as the basis of almost every review I complete. The problem is appraisers do not understand is CU is an underwriting tool. CU cannot be used until the appraisal has been completed and delivered. It is then it goes through the system for analysis. Appraisers are placing too much emphasis on CU and not on their work. If you want to understand CU then read the Fannie Mae Selling Guide Part B4 Underwriting Property. Especially read section B4-1.1-04: Unacceptable Appraisal Practices. This will solve a majority of appraisal problems. As far as CU’s analysis, it is twofold. There is an adjustment analysis which is based on regression analysis. This is not to be taken literally as there may be factors that are not included. This area is an indicator of possible problems. It will be up to the reviewer to determine if there is a problem and determine how, if at all, it will affect the appraisal analysis and/or value conclusion. The second is a list of sales, including the comparables, that CU has selected in the subject area from CU’s data base. Most all of these sales should be found in MLS. CU reads each comparable from the appraisals of these closed sales (Fannie Mae sales only) and if any of these appraisals use the same comparables it analyzes how each appraiser evaluated each section of the appraisal grid. (IE. how one comparable, that was noted in several appraisals was rated for condition thus forming a consensus). If the appraiser rated a comparable a C4 while the consensus of other appraisals used rated it a C3 then that would be a red flag to investigate further and possibly ask the appraiser to explain. That is CU in a nutshell. CU does not read the appraisals commentary. That is up to the reviewer. The most egregious flaw of all appraisal reports happens when the appraiser completes the report but totally fails to write a clear, concise analysis. It’s like telling a story but leaving out the last chapter. A statement such as “ALL COMPARABLE SALES WERE CONSIDERED IN THE FINAL ESTIMATE OF VALUE” in the Summary of Sales Comparison Approach is the appraiser’s way of saying I haven’t a clue take my word for it. Thus all the work is completed but the appraiser has left out the last chapter. This could make the report misleading. A CU review is a tool to help the reviewer analyze possible problems in the appraisal report to aid the lender in their risk analysis. This review cannot be completed until the appraiser submits the appraisal. The questions submitted by the reviewer, which is before the loan is underwritten, is the last step in the appraisal process to resolve problems the appraiser has possibly overlooked. In most all cases a concise explanation will clear up any problem.
-by Debbie Biehl
As Data is transferred manually and replicated for various things such as implementation of new software, numerical errors occur. Many times I have to go back to old hand drawn sketches at the Township or City, in order to break the tie when looking at two to three different square footage sources. This is more common that one knows. Often not very significant, but enough to get grabbed by CU.
-Lake front site data, especially when most of the site is platted under water. I find great comps, but some sites are platted 800-1600 ft into the lake while on that same lake others are reported only to the water. They must be consistent for adjusting purposes. I have actually measured comparable sites from the street to the water manually. Then they fit in better when making adjustments. At different times in history different techniques were utilized. However, I know as time goes on, and we try to improve our statistics we will find that they are only as good as the person figuring and recording the data in the beginning. We have become more more lazy relying on data transfer when that is where all the errors are made. Those old cards in the Township offices, now mostly in their archives, is some of the most accurate data on properties found. The people who calculated this data did not have computers, they knew their mathematics, rods, chains and metes and bounds, and new how to measure a house. But the downside is that all houses must come to an end or a new beginning. Things change, but as an old appraiser you have to know in what parts of history and which technique’s were most helpful and accurate.
When I find these errors I realized that, CU has so many errors today, and they don’t even know it. Just think if 20 appraisers have used the same wrong data on a set of comps, and the 21st appraiser finds the error and explains why it is incorrect, how many times is this replicated all over the CU system. But it is consistent and “Pretty on Paper.”