Broadly, meditation can be divided into two[1] kinds.
Insight is what produces permanent changes to subjective experience. Concentration practice does not produce permanent changes to subjective experience. Why even bother with concentration? Because training concentration improves your ability to do insight. Suppose you were a biologist. Insight is examining nature. Concentration is building a microscope. If you do concentration first then you'll be able to examine phenomena that remain invisible to unaugmented insight. But if you spend all your time building a microscope and never use it to examine living things, then you'll never learn how rainforest ecology works.
Insight sounds really stupid when you read the instructions. It's stuff like "pay attention to the sensation of breathing, particularly around the tip of your nose" or "label each of your thoughts with the sense (touch/taste/smell/hearing/etc.) it consists of". That's it. Do one of those activities for long enough and maybe you'll get enlightened[2].
This works. It's not easy and it takes a while, but it's almost as reproducible as "if you eat lots of protein and lift heavy weights, then your muscles will grow". However, unlike weightlifting, it's not obvious why insight meditation works. It makes sense that paying attention to the sensation of air flowing out of your nose might temporarily decrease your anxiety, but it's not obvious why doing so will permanently decrease your anxiety.
What's going on?
The first thing to understand is that insight meditation[3] produces "insight". "Insight" is a standard translation of the Pali word vipassana for which there is no English equivalent. Exactly which words we use to describe what is not really important here. What matters is that "insight" refers to a particular kind of understanding that isn't really in the Western ontological map.
"Insight" can only be acquired through direct self-experience. This is the opposite of how Western science works. If a trusted person says "I measured a rock to be 5.4 kilograms" then you know that the rock weighs 5.4 kilograms. It doesn't really matter who weighed the rock. If you weigh the rock then you'll have higher confidence that the rock weighs 5.4 kilograms because you don't have to rely on hearsay, but it's not important who weighs the rock.
"Insight" is the opposite. A guru telling you "your 'self' is not a stable, coherent entity" will not suddenly cause you to forever feel like your self is not a stable coherent entity. In order to break through to non-self awareness, you have to actually do insight practice. Listening to someone else talk about insight will increase your insight the same way listening to a gym rat will grow your muscles i.e. not at all.
Another way to think about "insight" is by comparing it to heroin. Heroin is an addictive drug that destroys your life. Taking heroin will quickly get you addicted and destroy your life. But learning about heroin abstractly, by reading about it and talking about it, will never get you addicted. You can understand heroin addiction abstractly by reading about it, but in order for heroin to permanently alter your subjective experience, you would actually have to take[4] heroin.
There's no English word for "direct experience that transcends a misconception about subjective experience", so we use the word "insight".
True understanding cannot be unlearned. A scarecrow might look like a person, but once you go up to it and feel its hay, you'll never be able to trick yourself into once again believing it's a living human being. Once you realize that someone is a bullshitter, you'll never be able to unsee it. Insight practice produces permanent changes to your subjective experience because it causes you to understand things. If you pay close attention to your sensations and you notice they're all composed of vibrations, then you will forever understand that all your sensations are composed of vibrations. Especially if you can examine those vibrations whenever you want. Insight produces permanent changes via incontrovertible direct observations that contradict misunderstandings about the true nature of phenomena. You can't believe you're blind when you can literally see anytime you want just by paying attention to the signals coming in from your eyes.
That explains why insight produces permanent changes in subjective experience. But it doesn't quite explain why you have to do insight. Isn't it enough for someone to tell you "everything is vibrations"? No, because you reading that—even if you believed it—did not cause you to feel like everything is vibrations. But why? Why isn't it enough for me to tell you "everything is vibration"?
Enlightenment isn't about believing something like "everything is vibrations" or "my 'self' isn't a stable coherent unity" the same way you "believe" that scientists have proved with $p>0.05$ that a drug works. Insight destroys your misconceptions about subjective experience by paying close attention to subjective experience. You may believe the sky is blue, but if you pay close attention, like an artist, you might notice that right now it's actually white, grey or orange.
I have a snake chasing me. You have a snake too. Everyone has a snake chasing them. Buddhas have snakes too. Normal people are scared of their snake, and run away from it in terror. Just a glance is terrifying, so we try not to look at it. If I tell you "there is no snake", then that won't do a thing, even if you parrot the words. You see the snake all the time. But if you stop running away from the snake then you might notice that the snake doesn't bite you. If you look closely and carefully, without fear, you'll notice that your snake is actually just a dirty rope tied to your ankle. Dukkha is that snake.
[1] This distinction is, of course, an artificial dichotomy. It's hard to do pure concentration practice without producing any insight, and it's hard to do pure insight practice without improving your concentration. Meanwhile, non-dual traditions refuse to distinguish between concentration and insight at all. Nevertheless, the difference between concentration and insight remains a useful distinction to make. ↩
[2] Don't actually follow my instructions here. Follow real instructions, instead. ↩
[3] Except Zen. But you can add the qualifier "except Zen" to basically everything about meditation…except Zen. ↩
[4] Do not actually take heroin. ↩