Mauricio Pochettino’s side were held to a 0-0 draw in normal time before losing 4-3 on penalties against opponents currently 10th in League Two.
Christian Eriksen’s opening spot-kick was saved and Lucas Moura hit the crossbar before Tom Lapslie scored the winning penalty to spark a pitch invasion from Colchester’s jubilant fans.
It was the first time Tottenham manager Pochettino had been knocked out of a domestic cup competition against lower-league opposition.
The shock defeat extended Tottenham’s poor run, which included a 2-1 loss at Leicester in the Premier League on Saturday.
After finishing as Champions League runners-up last season, Tottenham were expected to kick on this season but instead they have struggled to find any consistency.
Pochettino has grumbled about the distractions of the transfer window, with Eriksen and several other players linked with moves that never materialised.
As a result, Tottenham have looked a disjointed team, with two-goal leads squandered in draws at Arsenal and Olympiakos.
A surprise home defeat against lowly Newcastle was another blow and Tottenham are languishing in seventh place in the Premier League at present.
The upset was the biggest setback so far for Pochettino, who made 10 changes to his side but could still call on enough quality players to expect a win against a team 71 places below them in the football pyramid.
There were debuts for 17-year-old Troy Parrott and 20-year-old Japhet Tanganga, while Eric Dier made his first appearance of the season alongside Dele Alli in midfield.
But Tottenham laboured badly and could not find the breakthrough as Colchester followed up their second round win against Crystal Palace with an even bigger scalp on a memorable night that ranked alongside the club’s famous FA Cup success against Leeds in 1971.
With Pochettino yet to win his first trophy as Tottenham boss and the club without silverware since the 2008 League Cup, this was another blow to their flagging morale.
There were no problems for the other six Premier League teams who have all made it through to the last 16.
Manchester City started their bid for a third straight League Cup title with a comfortable 3-0 victory away to Preston North End.
There were 4-0 wins for Leicester City at Luton and for Southampton away at their local rivals Portsmouth.
Arsenal defeated Nottingham Forest 5-0. Brazilian teenager Gabriel Martinelli scored twice on his debut for the Gunners.
Watford beat Swansea 2-1. Everton got a 2-0 win at second-tier side Sheffield Wednesday. (AFP)