Michael Gorbachev, starting very soon after he became General Secretary of the Communist Party (the defacto head of state) in 1985, set out to establish democracy and freedom of speech "with all deliberate speed" -- i.e. as quickly as he could accomplish it without losing power or throwing the USSR into chaos.
Why did such a man get to the top of the USSR? I think part of the answer is hypocrisy. The Soviet Communist Party, like the corrupt church of the Middle Ages, preached ideals very different from their practice. Soviet children were brought up on the idea that they were on a crusade to make the world better and freer. While the doctrine was no doubt encrusted with rationalizations of the current practice, a deep thinker might go to the root of things and realize that the most central ideal is to make the world a better place, whereas in fact, they had a class of people, just like the old aristocracy, running the nation for their own benefit at the expense of the vast majority of people. Similarly Martin Luther saw a contradiction between what Jesus preached and the selling of indulgences -- essentially tickets to heaven in his time which among other practices was making the Holy See obscenely rich.
Even a leader who came up under Stalin, like Nikita Khrushchev was apt to find the gory history of the party too much at odds with some basic pull toward decency. In 1956, he gave a speech about the insane excesses of party purges under Stalin, and other inhuman policies, with the hope that the party would turn its back on that sort of thing. He allowed Alexander Solzhenitsyn to publish A Day in the Life of Alexander Denisovich, a short novel describing life in the prison camps or Gulags, and even permitted a movie to be made from it. Gorbachev was among those who, going to school at the time, found hope in the new developments. One of the problems with understanding the dynamics of a nation like the Soviet Union is the oversimplified image of a static dictatorship. Dictators such as Stalin, Mao, Saddam Hussein, and Hitler, seem to me from my reading, to have held and consolidated power only through deft orchestration of crises. They require demonic enemies out to crush the nation at the moment when extreme discipline is dropped. They have to foment the sort of hysteria that justifies trying and executing not just anyone, but people close to the top -- those who might engineer a coup against the dictator.
In the case of Stalin, he was able to keep redefining the ideology so as to marginalize whoever threatened him most at a given time. Khrushchev was playing somewhat by the same rules, but did not have Stalin's extreme lack of scruples to just execute anyone for convenience sake. The people who lost power under Khrushchev just went into obscurity; some of them outlived him. Nevertheless, Khrushchev combined Liberalization with the ejecting of his main competitors from power, and this provided for a while the sort of disorientation among the party that helped Stalin and Mao maintain one man rule.. Had he been more like them, he would probably have, as things began to stabilize and people began to feel more safe, generated a new anti-liberal hysteria and ridden that for a while.
Instead, events took on a life of their own. Rumors and excitement over liberalization spread to the Warsaw Pact nations which lead to an embarrassing degree of liberalization in Poland, and outright revolt in Hungary. Khrushchev did not have sufficient sway to survive the disintegration of the Warsaw Pact, so the pendulum started swinging the other way, and the brakes were put on liberalization. Still, there had been an awakening, and many people, like Gorbachev, never forgot it.
I suspect Khrushchev was never again so confident as to how far he could go in transforming the USSR, and this in part led to a need for various sorts of posturing such as the famous shoe pounding at the U.N. He was also barely literate and uncouth which did not serve him well on the world stage and had to be a disadvantage at home as well.
A coup was engineered by Brezhnev and others in 1964, which led to Brezhnev's rule until 1982. I think it is safe to say that Brezhnev stayed in power so long largely by staying in the center of the ideas and attitudes of the ruling class, and under him, party members lost much of their sense of insecurity, and lived a fairly comfortable life while the rest of the country scraped by. Tension with the West, in my opinion, helped maintain solid and consistent structure of the Soviet state in that period. It is often called the "period of stagnation". Peoples lives remained about as miserable as they had been, but I suspect there was a subtle change over time. The USSR "lost" the space race. Memories of the "Great Patriotic War", or World War II (in which approx one tenth of the population died in a war on their own territory) faded, and by the end of the Brezhnev era, the population seemed burnt out and cynical, and this attitude, and envy of the West's consumer culture extended into high levels of leadership - especially among those who got to travel abroad. For a few years after Brezhnev's death, there may have been a low key struggle between the "old guard" and the many people who thought that something needed to change, and there was a series of octogenarian successors.
The choice of Gorbachev signaled a strong movement of sentiment towards change. Though it is hard to say who wanted how much change, Gorbachev's ability to stay in power for several years suggests he was not so radically out of touch with the mood of the country. According to a recent Gorbachev interview, he went around to the Warsaw Pact governments and told them not to expect Russian troops to keep them in power, suggesting they had better go with the times.
In my estimation, for too long Gorbachev's principled attempts to steer his country into normalcy and modernity and away from being a police state, were treated, in much of the West, as symptoms of weakness, or some kind of ploy, certainly not as a potentially epochal change. Rather than offer help in getting through a dangerous period of change, the US mostly "kept up the pressure".
There is a lot to say about the various crises. As in 1956-7, the mood of change was seen by various ethnic groups as an opportunity to try to break free -- this time the USSR was affected while the Warsaw Pact nations were simply allowed to be what they had always pretended to be - independent nations. By 1990, there were free multiparty elections in which Gorbachev was elected the first President of the USSR, and a transition away from remote control of the government by the Communist party was in full swing if not completed.
In October 1991 some coalition of the KGB and the military tried to overthrow Gorbachev, holding him captive in his vacation dacha. Gorbachev managed to somehow face them down as demonstrations went on in Moscow lead by Boris Yeltsin, and it became clear that the highest officers of the military could not command the troops to put the demonstrations down. Gorbachev was soon released, but the facade of solidity of the USSR was shattered, and Yeltsin and others quickly engineered a splitting up of the USSR into its constituent "republics".
The end of the USSR was seen by the US at least as the end of the Cold War. In fact there was no more Cold War once Gorbachev showed his willingness to democratize the USSR and allow the Warsaw Pact nations to go their separate ways. In my opinion if the US and other Western nations had supported Gorbachev's prestige, and embraced a new era of cooperation with the USSR, and a need to help the USSR come back from the brink of mass poverty, the USSR would not have broken up, or might have partly broken up in a more orderly way, but democracy and freedom of the press would have been maintained. As it is, democracy and freedom steadily deteriated. Russia is very weak except that it has the ability to destroy civilization. Not much to cheer about in my opinion. Certainly the idea that we "won" the Cold War seems like a foolish illusion.