If you work in user research, you'll quickly notice that what people say they'll do and what people actually do are totally different things.
User Researcher: "Would you buy product $X$?"
Subject: "Yes, I would buy product $X$."
User Researcher: "How much would you pay for product $X$?"
Subject: [Thinks for a while.] "\$100. I would pay \$100."
User Research: "Excellent, because my company is currently selling product $X$ for \$90."
Subject: [awkwardly] "No thanks."
What people say they value (through words) and what people demonstrate they value (through actions) are very different things. If you're running a business or an economy, what matters is what people demonstrate they value. In economics, the things people demonstrate they value through costly actions are called revealed preferences. The things people say they value are called declared preferences.
Political acticism and armchair philosophy tend to assume that declared preferences are representative of revealed preferences. Anything that uses a survey to measure preferences is basically meaningless because it just measures declared preferences, and not revealed preferences. This includes the democratic process of voting, since voting is just a legally-binding survey. From this perspective, democratic voting is basically just a measurement survey of Social Desireability Bias, which is why democratic countries spend too much money on education, healthcare, and infrastructure.
Does this mean we can just define human values to equal revealed preferences [and totally ignore declared preferences]? Unfortunately, no.
I live in the United States where lots of people are fat. Not just a little fat. It's not uncommon to encounter people who are dozens of pounds overweight in a way that causes massive health problems.
Is it the revealed preference of fat people to be fat? Obviously not. The modern obesity epidemic is partially the consequence of government policy and partially the consequence of technological advance.
Generally-speaking, human beings evolved to like things that are good for our reproductive fitness in our ancestral environment. The environment changed over time, but so did our genes, so human behavior was kept more-or-less in-line with adaptive behavior.
Since the invention of language approximately 50,000 years ago, technological advance has been changing the world at an exponential rate. At first this pace was glacial, and genetic evolution could keep up with it. But exponential curves don't stay glacial forever, and eventually technological advance outpaced genetic evolution by a wide margin. Important technological advances used to happen on a timescale of milennia. Then it was centuries. Now the timescale is decades at most, perhaps years.
On a long-term timescale, technology is symbiotic with reproductive fitness.[1] However, on a short-term timescale, technology optimizes revealed preferences. When technological advance is slow relative to biological evolution, revealed preferences are well-coupled to reproductive fitness, and technological advance remains coupled to reproductive fitness through the proxy of revealed preferences. But when technological advances very fast, it optimizes revealed preferences faster than reproductive fitness can keep up—at least in the short term.
Reproductive fitness is roughly analogous to human values. For most of human history, technological advance advanced human values (for those humans wielding the technology, at least). In a technologically stagnant world, technology becomes symbiotic with human values. In a world undergoing rapid technological advancement, technology and human values pull in orthogonal directions. This is a fancy way of saying that advanced technology hacks human values.
This divergence between "revealed preferences" and "human values" isn't limited to eating too much food foods and owning too much physical stuff. It infests every aspect of digital technology, including normalized pathologies like the solitary passive consumption of recorded music.
[1] The biggest religions usually got really big because they provided a small reproductive advantage that compounded exponentially over a long time. ↩