Let $p$ be an odd prime and $S$ a finite $p$-group. B.~Oliver's conjecture arises from an open problem in the theory of $p$-local finite groups and says that a certain characteristic subgroup $\mathfrak{X}(S)$ of $S$ always contains the Thompson subgroup. In previous work the first two authors and M.~Lilienthal recast Oliver's conjecture as a statement about the representation theory of the factor group $S/\mathfrak{X}(S)$. We now verify the conjecture for a wide variety of groups~$S/\mathfrak{X}(S)$.