Hoops skipper Callum McGregor curled in an early opener before Hammers striker Michail Antonio scored a quick-fire double and captain Mark Noble netted from the penalty spot.
Ryan Christie pulled a goal back for Ange Postecoglou's side five minutes after the break but Said Benrahma replied for West Ham and Celtic then made 10 substitutions before Jarrod Bowen and substitute Armstrong Okoflex, signed from Celtic this summer, added two more.
Postecoglou now has to get his players ready for the second leg of their second qualifier on Wednesday night after a 1-1 draw against the Danish side during the week, and the tightening of defence will be a priority.
West Ham, managed by former Hoops defender David Moyes, looked sharp, particularly on the counter, and will carry on preparing for their Premier League opener away to Newcastle next month.
It was not a positive start to the game for the Londoners.
Issa Diop replaced Angelo Ogbonna, who limped off injured just before McGregor easily brushed Manuel Lanzini off the ball and guided a shot from the edge of the box past goalkeeper Lukasz Fabianski.
Shortly afterwards at the other end, Celtic keeper Vasilis Barkas foiled Bowen when one-on-one.
West Ham levelled when Dane Murray's pass out from the back was intercepted by Noble who quickly moved the ball on to Antonio and he skilfully side-stepped defender Stephen Welsh and Barkas and rolled the ball into an empty net.
Pablo Fornals then sent the striker clear for a breakaway second when he split the Celtic defence with a pass and Antonio steered his shot from 12 yards past Barkas at the near post.
Noble scored a third from the spot in the 38th minute after referee John Beaton felt Antonio had been barged in the box by Celtic midfielder Ismaila Soro.
Both teams changed their goalkeepers at the start of the second half, Scott Bain replacing Barkas and Darren Randolph on for Fabianski, and the former Motherwell keeper failed to keep out Christie's close-range prod from a Liel Abada cross.
He did tip a Christie effort over the crossbar moments later before the visitors countered again and restored their two-goal lead through a composed Benrahma finish.
Postecoglou changed all of his outfield players and one of them, Leigh Griffiths, recently cleared by police over allegations of sending inappropriate messages to two schoolgirls, was initially booed by a section of the home support.
Some fans then cheered Griffiths but attention soon turned back to the game and how easily Bowen was able to flick a cross past Bain before Okoflex wrapped up the scoring with a low drive into the corner of the net.
