A great presentation by Kevin Hall on why we've ended up with a food system dominated by ultra-processed food, from about the 2 hr mark.
During the 20th century the world did a great a job of avoiding a Paul Ehrlich-style Malthusian disaster, using technology to increase food supply (mainly carbs + fats, less so protein) way faster than population growth; supercharged by govt policy from the 1950's to subsidise massive increases in soy and grains. => Huge amounts of cheap basic inputs for cheap, convenient, tasty UPF. But in hindsight it was overkill, solving Malthusian problems before they actually emerged, at extreme costs to population health and the environment.
He also discusses briefly his landmark 2019 experiment, showing people eating way more UPF than minimally processed, not linked to differences in carbs, protein, fat, sugar, fibre, salt, perceived palatability ...
So what is the mechanism? "A Rohrshach test", with everybody bringing in their prior opinions to answer the question, without actual evidence. He himself has no firm opinion on possible mechanisms. His group is working on a new experiment to try to get some evidence-based answers.