When Is a Representation Mistaken for the Thing It Represents?
When Is a Representation Mistaken for the Thing It Represents?
Numbers, maps, images, models, labels, and reports allow us to understand things that would otherwise be too complex to examine directly. They simplify information so it can be compared, communicated, and used in decisions. But every simplification includes some features of reality while leaving others out.
The trouble begins when we forget the distinction between a representation and the thing it represents. An accurate summary can support an inaccurate judgment if the information it excludes is important to the question being asked.
Why Representations Leave Things Out
Consider a road map. It shows streets, connections, locations, and distances, but leaves out the weather, construction delays, noise, neighborhood culture, and countless other features of traveling through a city.
That incompleteness makes the map useful. Representing everything about a city would produce something nearly as complicated as the city itself.
Philosopher Alfred Korzybski popularized the observation that the map is not the territory. The expression captures the difference between a description of something and the thing itself.
Artist René Magritte explored a similar distinction in his painting The Treachery of Images. The painting depicts a pipe alongside the words Ceci n'est pas une pipe, meaning This is not a pipe.
The image resembles a pipe, but it cannot be used like one. It is a representation, not the physical object.
An Average Can Be Accurate but Misleading
Imagine two classrooms, each with an average test score of 85.
In one classroom, nearly every student scores close to 85. In another, half the students score 100 while the other half score 70.
Both groups have the same average, but their educational circumstances differ. The first may benefit from relatively uniform instruction, while the second has students with substantially different needs.
The average is correct in both cases. It describes overall performance but conceals the distribution of scores.
If the question is which classroom has a higher average, the number provides the answer. If the question is which students need additional support, the same number is inadequate.
The accuracy of the information has not changed. Its usefulness depends on the decision.
Accurate and Sufficient Are Different Standards
We often evaluate information by asking whether it is true.
Does a financial statement calculate revenue correctly? Does a photograph show the scene accurately? Does a model predict the particular variable it claims to predict?
Those questions matter, but they are not enough.
A representation should also be evaluated according to whether it contains the information needed for the conclusion being drawn.
Consider a company reporting 25 consecutive years of revenue growth. The figure may be accurate even if operating costs have increased faster than revenue, debt obligations have become difficult to manage, and major customers are leaving.
The growth record describes something real. It is not a complete measure of current financial health.
This creates two tests: Is the representation accurate, and is it sufficient?
The first asks whether it correctly describes what it claims to measure. The second asks whether it describes enough to support the decision.
When Metrics Begin Changing Behavior
A representation can also influence the reality it was originally created to measure.
Organizations frequently select measurable indicators to evaluate performance. Sales figures, customer accounts, productivity rates, completion percentages, and satisfaction scores make complicated processes easier to track.
The problem emerges when people begin pursuing the indicator rather than the outcome it is supposed to represent.
Wells Fargo's unauthorized-account scandal provides a real example. The bank used additional product sales and account openings as important indicators of business performance, and employees faced sales targets and incentives connected to those numbers.
In 2016, regulators penalized Wells Fargo over widespread unauthorized sales practices. An initial review identified approximately 2.1 million potentially unauthorized accounts.
An expanded review reported in August 2017 identified approximately 3.5 million potentially unauthorized accounts, while acknowledging that some flagged accounts were properly authorized.
The organization had incentives to increase account counts even when those increases did not represent genuine customer demand.
Goodhart's Law and the Target Problem
This is related to Goodhart's law, which is commonly summarized as the tendency of a measure to lose its value as an indicator when it becomes a target.
A company might originally track account openings because they provide information about customer relationships and business growth. Once account growth becomes an objective in its own right, employees have incentives to increase the number regardless of the quality of the underlying relationships.
The resulting figures can look better without the business becoming healthier or customers becoming more satisfied.
The underlying problem is that the metric captures only part of what matters. When the organization begins treating that part as the entire outcome, its behavior can change in ways that undermine the original purpose of measuring it.
Labels Can Shape Future Evidence
Not every misleading representation is numerical.
Suppose a manager describes an employee as unreliable after three missed deadlines. The label refers to genuine observations, but it excludes successful assignments, circumstances surrounding the delays, and possible improvement.
After accepting the label, the manager begins interpreting the employee's behavior through it. Another missed deadline reinforces the judgment, while completed projects receive less attention.
The label may also affect which responsibilities the employee receives. If the manager stops assigning important projects, the employee loses opportunities to demonstrate improvement or greater capability.
The representation is now affecting the conditions used to evaluate whether the representation remains accurate.
Descriptions can become self-reinforcing when they determine what evidence receives attention and which opportunities remain available.
Artificial Intelligence Can Make a Representation Convincing
AI systems introduce additional risks because they can produce detailed, fluent descriptions that resemble authoritative information.
In 2023, lawyers in Mata v. Avianca submitted fabricated judicial decisions and quotations generated using ChatGPT. The citations included case names and legal language that appeared authentic, but the underlying decisions did not exist.
A federal judge ultimately imposed a $5,000 sanction. The lawyers had relied on material that resembled legal authority without adequately verifying the underlying sources.
That case illustrates a representation that was false, rather than merely incomplete. The familiar appearance of legal citations encouraged reliance on information that did not correspond to real judicial decisions.
AI Can Also Be Accurate but Incomplete
Another risk remains even when AI systems accurately extract or summarize information.
Imagine an AI system reviewing job applicants. It might correctly identify universities attended, previous employers, technical skills, and years of experience.
Those details are useful. They do not fully describe an applicant's judgment, adaptability, creativity, interpersonal ability, or willingness to learn.
A more accurate extraction process would improve the reliability of the information collected. It would not automatically solve the problem of information that was never included in the representation.
The same limitation applies to summaries, forecasts, profiles, recommendations, and other AI-generated outputs. A model can become more fluent or more detailed without capturing the additional dimensions required by a particular decision.
Ask What Is Missing
Representations are necessary because reality is complicated. The solution is not to abandon metrics, summaries, labels, and models, but to understand what they capture and what they leave out.
Begin by identifying the representation. Are you relying on a test score, a financial metric, a professional label, a photograph, or an AI-generated analysis?
Then examine what the representation actually describes. A revenue figure measures revenue, a résumé records selected aspects of someone's experience, and an average summarizes a distribution.
The next question is whether that information is enough for the decision being made. If you are evaluating a business's financial health, revenue alone is insufficient. If you are deciding how to support students, the average score may conceal the most important differences.
Finally, seek evidence that falls outside the representation. Look at the distribution behind an average, the customer experience behind a business metric, or the actual work behind a professional label.
The Model Is Not the Reality
A useful representation simplifies something complicated while preserving enough information for a specific purpose. That is what makes maps, measurements, reports, and models valuable.
The danger comes from forgetting their boundaries. Once a number, image, or description becomes our entire understanding of a situation, omitted information becomes invisible to our judgment.
The most important question is therefore not only whether the representation is accurate. We also need to know whether it includes the information that matters to the decision.
A representation is mistaken for reality when we stop asking what is missing.
Comments
Post a Comment