US secret service agents allegedly took prostitutes back to their hotel in Cartagena, Colombia, before Obama arrived for the Summit of the Americas meeting. Eleven agents have been sent home and put on administrative leave while the scandal is investigated.
On Saturday the US Defence Department said a further five military personnel, staying at the same hotel, had been confined to their rooms after violating a curfew and "may have been involved in inappropriate conduct," the United States Southern Command said in a written statement. "The conduct is alleged to have occurred in the same hotel where the recalled United States Secret Service agents were staying."
The scandal has proved a mayor embarrassment for the White House at a meeting where Obama is discussing trade and the economy with 32 other heads of state.
The president has come under fire at the meeting over the impact of US monetary policy on the region and its attitude towards Cuba.
Peter King, chairman of the Homeland Security Committee, told NBC on Sunday: "This was wrong from the beginning to the end. If the facts are what they seem to be, it can not be tolerated."
Prostitution is legal in Colombia in designated "tolerance zones".
Local police were called to the Hotel Caribe in Cartagena's upmarket Bocagrande neighbourhood after a dispute with a woman in the room of one of the agents.
According to media reports, the agents had been drinking heavily before the president's arrival and taken women back to their hotel last Wednesday night. In the morning the woman involved in the dispute "freaked out" after she was not paid and banged on walls and doors in the hotel hallways.
King, who has been briefed on the situation, described a calmer scene to The Washington Post, which broke the story. He said that under hotel policy, overnight guests must leave photo identification at the front desk and leave the hotel by 7am the next morning.
According to King, the woman at the centre of the dispute had not left the hotel by 7am. Hotel officials called the police when the agent refused to open his room.
The woman refused to leave until she was paid, the agent disputed that he owed her any money. The incident was reported to the US Embassy and then referred to the Secret Service.
The agents were recalled and replaced before Obama's arrival in the city Friday afternoon.
King told The Washington Post that "everything they did was a violation of proper conduct".
"First of all, to be getting involved with prostitutes in a foreign country can leave yourself vulnerable to blackmail and threats," King said. "To be bringing prostitutes or almost anyone into a security zone when you're supposed to protect the president is totally wrong."
The Hotel Caribe is less than 1,000 metres from the Cartagena Hilton where Obama is staying.
Secret Service assistant director Paul Morrissey said in a statement that the situation had "had no impact on the Secret Service's ability to execute a comprehensive security plan for the president's visit to Cartagena".
"This incident is not reflective of the behaviour of our personnel as they travel every day throughout the country and the world performing their duties in a dedicated, professional manner," he said. "We regret any distraction from the Summit of the Americas this situation has caused."
The incident is the biggest embarrassment for the US secret service since 2009 when two aspiring reality TV stars gatecrashed Obama's first state dinner at the White House and were photographed with the president.
Michelle Bachmann, former Republican presidential candidate, said: "Americans should be outraged." She told NBC's Meet The Press that the White House had "clearly been embarrassed by this".