Dairy Crest said closing the factory would involve 164 job cuts, and has begun negotiations with trade unions.
The move was prompted by an internal review, and suggests the group is keen to consolidate its operations in order to save on costs.
A Dairy Crest spokesperson told DairyReporter.com there was little room for expansion at Totnes, where the factory is boxed in by houses and the railway.
"It became very clear that if we were going to modernise the plant, which it does need, then expansion would be necessary.
Unfortunately, that just wasn't possible."
She added the whole dairy industry continued to face "relentless cost pressures" and that "going forward we are better positioned if we produce milk in other factories" .
The group review anticipated closing the Totnes site by November.
If a deal is agreed with unions, operations there would be wound down in the two months before this, with glass bottle and non-returnable container milk production transferred to other Dairy Crest sites.
Earlier this week, Dairy Crest rival Robert Wiseman said prices for HDPE resin, used to make plastic bottles, had hit record highs.
This, said Wiseman, posed a serious threat to margin recovery over the next year.
Wiseman also said doorstep delivery milk sales had fallen over the last year.
Dairy Crest bought the loss-making Express Dairies doorstep delivery business from Arla Foods UK last autumn, and has spent the last few months integrating the business.
It is understood that moving glass bottle milk production out of Totnes would form part of this strategy.