Odemwingie has resolved his differences with Nigeria coach Stephen Keshi and has set his sights on appearing in the tournament, which begins on 19 January in South Africa.
Midfielder Mulumbu is also set to be absent from the Baggies when the competition gets under way due to Congo's qualification.
Odemwingie said: "We have good players who can play for a whole month at Albion and they will do well without me.
"Youssouf is also a very important player for us in the middle but we have good numbers of players. We have Yassine El Ghanassy who is still waiting his chance to show something from one of the wings.
"We have (Chris) Brunt, (James) Morrison, (George) Thorne. We have players who can play at this level and Albion took into consideration when recruiting players that a few of us might go (to the African Nations Cup).
"I know it's going to be a bit of a loss, us leaving for a few weeks, but we hope the players there can follow the team's performance."
Odemwingie is relishing the prospect of trying to help his country to glory and recognises it may be his last stab at winning the tournament.
He said: "Will I play in the African Nations Cup? Yes, yes. I will put the country and the fans before anything if there are any disputes but I'm a very straightforward person.
"If things are not okay I don't pretend and say things are fine, but definitely I'll put our football fans first in any decisions.
"I know I will miss a chunk of Albion's season but it is hard not to go to this African Nations Cup. It could be my last."