The first example on that page, "Regressional Goodhart", is totally wrong. If U measures V plus some noise X, assuming V and X form a bivariate normal distribution, then the conditional expectation of V is maximized by selecting the greatest U. In fact, the conditional expectation is linear in U.
Not sure how much stock I can put into the rest of the article given this. Goodhart's Law can seemingly only apply if the variables are highly non-normal, or are negatively correlated (e.g. worse doctors are more likely to refuse the difficult operations than good doctors).
> The first example on that page, "Regressional Goodhart", is totally wrong.
I don't think your statements contradict anything that it says. For reference, here are all the sentences in the "Quick Reference" part, which I'm assuming is the part you read.
Regressional Goodhart - When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal.
Model: When U is equal to V+X, where X is some noise, a point with a large U value will likely have a large V value, but also a large X value.
Thus, when U is large, you can expect V to be predictably smaller than U.
Example: height is correlated with basketball ability, and does actually directly help, but the best player is only 6'3", and a random 7' person in their 20s would probably not be as good
Is any particular one of these sentences false?
> If U measures V plus some noise X, assuming V and X form a bivariate normal distribution, then the conditional expectation of V is maximized by selecting the greatest U.
The word "maximized" carries some assumptions. If the only thing you can do is select based on U, then that statement is correct. However, if you had some means of selecting directly on V, then it's extremely likely that this would do better than selecting on U.
Suppose you're selecting the top 10 people. Suppose X ranges 0-10 chosen by a die roll, and V ranges 0-10, and it happens there are ten people with V=10, a hundred people with V=9, and a thousand with V=8 (and a lot more with lower Vs). On average, you'll have ten V=9s who score U=19, ten V=9s and a hundred V=8s who score U=18, and so on; in order for selecting on U to perform as well as selecting on V, every one of the V=10s would have to roll X=9 or X=10, which is exceedingly unlikely. It is true that taking people with high U scores yields people with higher Vs than taking people at random, and it is further true that taking the U=19s will give you better results than taking the U=18s or U=12s. It is simultaneously true that, when you take the U=19s, you'll be getting people whose X was 9 or 10, much higher than if you selected people at random or if you selected directly for high V. The first two sentences from the text state exactly this. (One consequence of this observation is, e.g., if you're doing admissions based on some test score, and you're considering raising the required score by ∆U, you should know the effect will be to raise average V and to raise average X, with ∆V=∆U-∆X, and if ∆X is large, you may be disappointed in the results. This is simple regression to the mean.)
I imagine you know all these concepts; I think you're interpreting the text as a stronger statement than it is.
Not sure how much stock I can put into the rest of the article given this. Goodhart's Law can seemingly only apply if the variables are highly non-normal, or are negatively correlated (e.g. worse doctors are more likely to refuse the difficult operations than good doctors).