Defoe was substituted in Tottenham's shock 1-0 loss to Wigan, much to the annoyance of the home fans, and the England striker went straight down the tunnel rather than taking his place on the bench.
But the Spurs boss refused to read too much into the action, insisting the team takes preference to any one player.
"For sure it is not something that is very pleasing to see," the Tottenham manager said.
"The most important thing for the players to realise is that we do things for the benefit of the team. We understand individual frustrations but we, as a team, are more important.
"The game wasn't going well for Defoe in our thought process. Maybe he could think the opposite but we don't take him off to punish him. We took him off to improve the team."
The Togolese, who was Tottenham's top scorer while on loan last season, is yet to start a Premier League game for the club this term, but it appears Villas-Boas may recall him for Sunday's clash against former team City.
There have been reports that Adebayor is unhappy at his lack of game time this season, but Villas-Boas insists there is no problem with the player's attitude, citing recent injuries as the reason why he has not started games.
"We're quite happy with his attitude," Villas-Boas said.
"He had an injury which took him out for six games and then another injury took him out for two games. We have the injury records if you want to look at them.
"From the 11 that we've played, I think he's been out for eight of them and he wasn't here at the start of the season.
"In the games when he was available he showed the quality striker that he is. We have no doubts whatsoever. We have him fit now and hopefully ready for the next upcoming games. We know his ability."