 |
Services Offered this Issue
> a la mode – Titan Analytics > Adjustments CE (7 Hrs.) |
Adjustments: Sample Size, Relevance and Credible Solutions
By Scott Cullen, Certified Residential Appraiser
As appraisers, many of the assignment results we report in our appraisals are dependent on not just the accuracy of data we report, but also the credibility of the samples we use when analyzing data and drawing conclusions around it.
Credible reports require credible data. If our reports are based on data and samples that are unreliable or flawed, we can be exposed to unintended and unpleasant consequences.
So how do we collect data and extract samples that we can defend?
Sample Size
One characteristic of a credible sample is its size. This has been recognized for hundreds of years in the real estate business. In the 16th century, Jacob Kobel wrote:
“Stand at the door of a church on a Sunday and bid 16 men to stop, tall ones and small ones, as they happen to pass out when the service is finished; then make them put their left feet one behind the other, and the length thus obtained shall be a right and lawful rod to measure and survey the land with, and the 16th part of it shall be the right and lawful foot.”
Is 16 a sufficient sample size for any purpose? The Major League Baseball requirement for the batting title is 502 plate appearances.
What about the sampling we use in the development of a residential real estate appraisal? Does the median price of 16 sales chosen at random from last year, compared to 16 sales from this year provide a credible rate of appreciation? Or do we use all sales from last year and compare that median to the median of this year’s sales? My local MLS provides a convenient way to find the median price of the most recent three–year periods by seller type, construction type, property type, size, bedroom count and other characteristics. I can search by county, city, school district, zip code and sometimes neighborhood. I am able to refine the criteria to fit the subject property. The challenge is that doing so reduces the sample size.
How do we balance sample size with sample fit?
Sample Relevance (Fit)
On one of my recent appraisal assignments, I had an agent point out that the subject neighborhood had appreciated 13.2%. He was correct based on a sample of all sales in that neighborhood. However, I could also show from the same MLS data that houses less than 1,500 square feet with 3 bedrooms (like his listing) had actually declined 20% in price. This is an example of the tension between sample size and sample fit. Should all types of houses be included to maximize sample size, or should the range of house characteristics be narrowed down to the point where some minimum sample size threshold is met? What is the optimal trade-off between sample size and sample fit?
When I ask questions at CE classes and appraiser meetings, the answers I receive seem very squishy and subjective. I remember there being more certainty in my statistics classes back in my college days.
With the increased emphasis on the appraiser’s ability to support grid adjustments with data, the question of sample size and sample fit becomes more acute. I tried to use regression but without convincing results. I then hired a grad student who is studying statistics at the University of Minnesota to teach me on his winter break. He concluded that sample sizes are too small for the number of variables, and that I wouldn’t be able to produce reliable results.
A little while later, I learned that John Willems, a recent acquaintance, was a database consultant. He was interested in developing a repeatable and predictive model that would show contributory value of property characteristics such as GLA, bedrooms, baths and garage stalls. I gave him a large MLS database of sales from a first-ring Minneapolis suburb filled with 1950s ranch style houses. He came to the same conclusion: sample size is too small for the number of variables. However, he did have other ideas about how to support adjustments, which we will explore below. A fresh set of eyes does see a problem differently.
(story continues below)
(story continues)
At this point, I needed to know how large a sample was required for regression with a given number of variables. Is there some sort of geometric relationship? Like 16 for one variable, 16 squared (246) for two, 16 cubed (4,096) for three? I hunted down a credible source for the answer.
I contacted Maurentia Analytics who claims expertise in sample size/power analysis. You can read their response on my website. Part of the answer is that a minimum sample size of 66 is required for two variables. I don’t know about you, but I rarely have a comp pool of 66 sales that vary from the subject in only two ways! Perhaps it is credible to expand the search to include a wider geography, market time or variation among properties. But I quickly learned the limitations of regression, at least for appraisals, of individual residential properties and limited comp pools. Regression may be “state of the art” for mass appraisal, but while it is a valid statistical method, I believe it is being used by residential appraisers who lack a sufficient understanding of the limitations of sample size and sample fit.
Credible Solution
Willems was interested in pursuing a different database in search of support for sales grid adjustments. Sales data seemed to include too many variables; it cannot be filtered by condition or quality and there is a concerning amount of variability in the way the data is collected (agent bias). Agents list properties for buyers, not appraisers. It seems logical to look at cost data as an alternative. There is no bias, condition is not an issue, quality and physical characteristics can be tailored to the analysis at hand. Also, the sample size limitation is eliminated. The idea is much closer to Paired Sales than it is to statistical analysis of the MLS database. Think of it as paired cost.
Paired Data Analysis is a recognized method of developing sales grid adjustments. Paired data includes, but is not limited to, paired sales. When paired sales are not available, paired cost is a good alternative. Trying to calculate depreciation can complicate the analysis, but this can also be measured in the Cost Approach. We can solve for depreciation by starting with the sale price of the most similar comp, deducting land, and comparing the result (depreciated cost of improvements) to the total estimate of cost new of that structure. When we examine the URAR Cost Approach, some interesting relationships emerge. Starting with page one of the URAR, when you enter Effective Age, you have begun the Cost Approach.
(story continues below)
(story continues)
In my form software (TOTAL), Effective Age becomes the numerator in the depreciation calculation within the Cost Approach section. Effective Age Lifespan is the percentage of physical depreciation. An Effective Age of 20 years / 60 years Lifespan is 33.3 percent physical depreciation (See Figure 1 below). Note: all depreciation is assumed to be physical because functional is seldom present in a replacement structure, and external can be factored into site value.
(story continues below)
Figure 1
(story continues)
Where does the 60-year Lifespan originate? It is a setting you make in your form software, but it should come from your cost data, not the chronological age of the house you are inspecting. Your cost data should have a specification for Lifespan (it may be called Economic Life) that usually changes with quality level. A range of 55 to 70 years is typical for Q6 to Q1. If I am appraising a Q4, the Lifespan in my cost data is 60 years.
This ratio is applied to Total Estimate of Cost New to result in Depreciated Cost of Improvements. If the “Cost New” is $310,200 and depreciation is 33.33 percent, then Depreciated Cost is $206,810. Another way of thinking about Depreciated Cost is the amount the market pays for the building. Of course, the credibility of a statement such as “the market value of the building is $206,810” depends on the accuracy of the numbers used for Effective Age, Site Value and “As-is” Value of Site Improvements.
As we all know, it is difficult to estimate depreciation. This is why, for many years, my reconciliation stated that the “Cost Approach is given less weight because estimates of depreciation are not reliable.”
But if we begin with the sale price of the most similar comp, and use those building characteristics in cost approach calculations, we can dial in Effective Age until Indicated Value by Cost Approach matches the sale price of that comparable property. This process extracts market depreciation from a known sale price, unbiased third-party cost data, and our best estimate of Site Value and “As-is” Value of Site Improvements. Indicated Value by Cost Approach becomes Indicated Depreciation by Cost Approach (See figure 2).
Depreciated Replacement Cost
If we know depreciation is 33 percent, then the market is paying 67 percent of replacement cost for the building. Looking at the Cost Approach, can we assume that 67 percent of the $122 dwelling number is our GLA adjustment? A GLA adjustment of $82?
That would be incorrect. We do not use 67 percent of $122 because we want our adjustment to reflect the cost of one more foot of GLA, not the average total cost of GLA. Cost data is published as Average Total Cost (ATC). If the cost to build a 2,000 square foot structure is $244,000, the ATC is $122. But if we look in our cost book, a 2,200 sf structure is $116.00 per sf.
Now the similarity between Paired Sales and Paired Cost is revealed (See figure 3 below).
(story continues below)
(story continues)
The replacement cost of 200 square feet of GLA is $20,000. This is $100 per square foot.
In our example of 33 percent depreciation, when the market is paying 67 percent of cost new, a GLA adjustment of $67 is inferred.
Marginal Cost
In the financial accounting world, the $100 figure is known as marginal cost (sometimes variable cost). I have used both marginal cost and variable cost to describe the same concept.
To calculate depreciated cost adjustments, we must know both marginal cost and depreciation. There is work involved, but both can be extracted from unbiased third-party cost data and the URAR Cost Approach. The process has been automated by the suite of Solomon Appraisal calculators.
The job of the appraiser is to collect and analyze data. When Paired Sales are not available to develop and support adjustments in the sales grid, the appraiser can apply sample based techniques such as grouped sales or regression. However, if sample-based techniques are hindered by size and relevance and consequently are not reliable, a good alternative is to use Depreciated Replacement Cost as described here.
Internally Consistent Reporting
Remember, when you are calculating Depreciated Replacement Cost, use marginal cost rather than average cost. When you do, the entire report will harmonize. Your Effective Age number on page one and the cost data you use to complete the Cost Approach will reveal depreciation recognized by the market. The resulting Remaining Economic Life number (Economic Age – Effective Age) will show the percentage of replacement cost that the market is paying for the building. Sales grid adjustments related to the building can be extracted from unbiased, third party cost data you cite in the Cost Approach, factored by the percentage of cost new the market is paying for buildings like the subject. There is no liability for sample sizes that lack relevance or statistical power with this method. Stay safe out there!
> CE Online – 7 Hours (approved in 40 states)
How To Support and Prove Your Adjustments
Presented by: Richard Hagar, SRA
Must-know business practices for all appraisers working today. Ensure proper support for your adjustments. Making defensible adjustments is the first step in becoming a “Tier One” appraiser, who earns more, enjoys the best assignments and suffers fewer snags and callbacks. Up your game, avoid time-consuming callbacks and earn approved CE today! Sign Up Now! $119 (7 Hrs)
OREP Insured’s Price: $99
About the Author
Scott Cullen has been a Certified Residential Appraiser in the Minneapolis / St Paul market since 2001. John Willems began developing database applications in 1988. After retiring from public service, he founded CustomReportWriters.net. Scott and John formed Solomon Appraisal LLC to market depreciated replacement cost applications for residential appraisers.
Send your story submission/idea to the Editor: isaac@orep.org
by Steve Kahane
The article does a good job pointing out some of the shortcomings of regression and big data but falls short in the cost analysis.
Regression and complex algorithms are great at identifying correlations but lack the judgment to determine causation. Regression is also limited to property characteristics quantified numerically. Non-numeric aspects of value like quality, condition or view don’t fit into statistical models.
I see a couple flaws with the cost-based adjustment method proposed. As noted above by James McGauhey, depreciation isn’t typically linear. In my market, depreciation is steepest in the first year or two of a property’s life. Buyers pay a premium for an un-used home where they are able to customize it and select their own options rather than settle for those chosen by others. To compete, re-sales typically have to be discounted. When new construction is no longer available, the difference of a year or two in age no longer matters.
The other problem with the proposed cost methodology is that it combines fixed and variable costs. Additional square footage consists of variable costs: additional lumber, sheetrock, flooring, etc. Cost figures from a cost service include fixed costs and variable costs. Fixed costs include permitting and architectural costs, appliances, utility hookups, cabinets, counters and all the other components included in every house no matter the size). As such, a one square foot house is really expensive to build and each additional square foot is much cheaper. A GLA adjustment should reflect the additional premium paid for additional square footage, not the average cost per square foot. A 10 day trip to Italy for $5,000 ($2,000 airfare, $3,000 hotel and meals per day) equals $500/per day, but an additional day is $300, not $500. The same applies for the cost of additional square feet.
-by Hans G. Schaetzke II
The comment by James McGauhey below blows a big hole in this depreciated replacement cost concept – does that matter? What about AdV, he’s right too. Does any of this matter? I guess the rest of us are just making it up and every appraisal that was done prior to 1990 was pure fraud. Or, maybe, real appraisers are still the buffer against the BS in the market and these guys are just proving that appraisers still ROCK. My job is to reflect the market and the actions of market participants in my sales analysis – this is done by bracketing my comps in the direct sales comparison analysis. It may seem too easy, but it works – just like the FNMA guidelines help ensure that adjustments reflect contributory value in a simple and elegant way. The market does not perform according to statistical analysis nor does the market perform according to cost. People don’t negotiate this way, so your adjustments will never be a “reflection of the market”. You might come close after a bunch of number crunching but – it isn’t a reflection of the market. This is yet another door to unending tail chasing designed to impress bankers and review appraisers of which I am both and, I am not impressed.
-by AdV
It’s my understanding that the external obsolescence (if any) displayed in the depreciation grid in cost approach is that amount attributable to the improvements. The opinion of site value should already consider the external obsolescence. So if there is external obsolescence there should be loss in value to land and improvements.
-by Mike W
I can follow the logic in this article until he states “Now the similarity between Paired Sales and Paired Cost is revealed (See figure 3 below).” Why does figure 3 use $120 per square foot to determine replacement cost rather than $116. The inferred GLA adjustment would be significantly different if ATC of $116 were used for the 2,200 square foot property. Am I missing something?
-by Joseph Fassari
Try explaining this with real time application. Use numbers from an existing MLS. It will not work in Pa. You can say all you want the numbers are messed up. Cannot wait when the banks fall and we cannot be blamed since they are using other technology.
-by Dean Zibas
I have 25 years residential appraising experience. While I agree that making adjustments based on “my experience in the marketplace” isn’t proper, it is also very difficult to do accurate paired sales analysis based on the limited sample size and number of variables involved. This article supports my contentions.
-by James McGauhey, Certified General
Are you familiar with Marshall & Swift depreciation tables? I have a grid taped in front of me that shows their schedule. M&S content that real estate does not depreciate in a straight line. Based on what they claim to be market data, RE depreciation is skewed with most loss at the back end, such as if a property was neglected, the last 1/3 of its life would have extreme depreciation, and the first third the least. On M&S schedule, a 20 year depreciation of a 60 year economic life property (40 yr remaining economic life) results in 16% depreciation, not 33 1/3. Might be the basis of another article?
-by AgnesT
Seems like this article is trying to justify the MC Addendum. This document was flawed from day one. It was add-on from the housing crisis. When Mid-Atlantic states suffer a bad winter everything is skewed. Sellers pull their homes off the market, buyers quit looking until Spring, vacant homes don’t have driveways that are shoveled, prospects are anxious about driving in inclement weather ,etc. This extends marketing times, causes only the desperate to keep looking, etc. This skews absorption rates. The author is right about sample size. Any school teacher never wants 60 kids in one class, nor only 5 (they will get spoiled with attention). When analyzing for trends lots of items must be observed. Did the realtors have more influence in developing the asking price? Did the owners overprice and finally wise-up? This effects median DOMs. Did a government agency offer below market interest rates to home buyers with a 60 day window for application?
-by Hans G. Schaetzke II
Adjustments are simple, it’s picking the right comps that is the trick and the adjustments only apply to the right comps. Your comments show why the world needs appraisers to weed through a multitude of indicators. I love the way you pick apart the facts and demonstrate that the winter market may have no relationship at all to spring, summer and even fall. I fear that the more traction these alternate adjustment validation methods gain the more people will believe that value is a sum of the parts. When they believe they can come up with adjustments with a mass of “so-called” comp data or even a cost approach they will believe they don’t need an appraiser.
-