The deal between Taliban and Trump administration is linked to the Taliban's takeover of Afghanistan, said the top US defence officials.
The agreement named Doha which set a date for US troop's withdrawal was signed in February 2020, reports BBC.
Gen Frank McKenzie said the deal had a "really pernicious effect" on the Afghan government and military.
Defence Secretary Lloyd Austin agreed, saying the agreement had helped the Taliban get "stronger".
In addition to setting a withdrawal date, the Doha agreement included broad obligations on the Taliban to take steps to prevent groups such as al-Qaeda from threatening the security of the US and its allies.
After his election, US President Joe Biden continued the plan for withdrawal but with an end date of 31 August, instead of May.
The US defence officials made the comments on Wednesday in testimony to the House of Representatives Armed Services Committee.
The hearing is taking place weeks after a chaotic withdrawal at Kabul airport as foreign powers sought to get their citizens home and thousands of desperate Afghans begged for rescue. A suicide attack killed 182 people during the operation.
As head of the US Central Command, Gen McKenzie oversaw the withdrawal from Afghanistan, which marked the end of a 20-year presence in the country and America's longest war.
Gen McKenzie told the committee the Doha agreement had a strong psychological effect on the Afghan government because it set a date for "when they could expect all assistance to end".
He said he had believed "for quite a while" that if the US reduced the number of its military advisers in Afghanistan below 2,500, the Afghan government and military would inevitably collapse.
After the Doha agreement, he said the troop reduction ordered by President Biden in April was "the other nail in the coffin".
Mr Austin said that by committing the US to ending air strikes against the Taliban, the Doha agreement meant the Islamist group "got stronger, they increased their offensive operations against Afghan security forces, and the Afghans were losing a lot of people on a weekly basis".
The defence officials previously spoke to the Senate Armed Services Committee on Tuesday, where Gen Milley and Gen McKenzie said they had recommended keeping a force of 2,500 troops in Afghanistan, ahead of the full US withdrawal in August.
Gen Milley also said the Taliban takeover would make it harder to protect Americans from terrorist attacks, as he described the group as a terrorist organisation that "still has not broken ties with al-Qaeda".
Speaking to the BBC on Wednesday, Taliban spokesman Zabihullah Mujahid said the militants had "given guarantees to the world that there will not be any threat against any country including the United States from Afghan soil".
"We are committed to the agreement which has been signed in Doha between the Islamic Emirate of Afghanistan and the United States. We also want Americans and its allies to be committed on the agreement too. Instead of making negative comments it would be better for them to choose the way of diplomacy and co-operation," he said.