Cuomo, who was being sworn into his second term at the World Trade Center, said that people are "confused, disappointed and angry" over whether the justice system is fair to people of all colors and economic statuses.
But it has "devolved into a situation where everyone is talking and no one is listening," he said.
"It must stop," Cuomo said. "The truth is that the justice system does need review. The truth is that there are troubling questions that have been raised that must be answered. The truth is that police officers need more safety and need more protection.
"The truth is that law enforcement needs to respect the community as much as the community needs to respect law enforcement. It is our obligation as leaders to provide the reforms necessary to ensure safety."
Cuomo's words come at a time when relations between the administration and police are increasingly strained. In recent weeks protests erupted after a grand jury declined to indict Officer Daniel Pantaleo in the chokehold death of Garner, who was unarmed and selling loose cigarettes in Staten Island.
Many called for the details of the grand jury proceeding, which are secret under state law, to be revealed, but a judge only allowed the release of limited information, excluding witnesses and testimony.
The move and the fact that District Attorney Dan Donovan only asked for a limited release, drew criticism.
A two-hour meeting this week between Mayor Bill de Blasio and police officials in an effort to ease relations ended without a resolution.
Cuomo said the protests against police behavior signifies that people are questioning "the essence of everything we believe in."
Belief in the justice system — along with trust in the economy and public education — makes up "the primary democratic institutions of this country," he said.
Restoring confidence in the system will require New Yorkers banding together and finding commonalities, he said.
"That's the challenge for us," Cuomo said. "To be fearless enough to admit the truth. To talk about these problems, no matter how difficult."
He pointed out that the tensions in New York are being played out across the country and urged New York to be leaders in solving the justice system.
"This is a uniquely New York moment," he said. "When things are at their worst, New Yorkers are at their best."