The simplicity of Doom's monsters is good, but complex monsters can work well as part of larger fight designs too. The Maulotaur from Heretic is a good example. It chooses randomly between three attacks; one is a simple projectile spray, but the other two are more interesting. It has a charge attack where it both attacks you and changes location at the same time, and ground-crawler attack that causes area denial by creating a long, impassable, slow-moving line of damaging fire. It can be an interesting enemy to fight alone, if the space is right for it. It can be much more interesting to fight a few of them at a time, if the space is right for it. But they also can work as a piece of much bigger fights, particularly since the ground crawler can pass up or down ledges and both attacks somewhat help to break up herding/circling dynamics. Compared to the simpler Cyberdemon, it has advantages and disadvantages; it's nowhere near as useful for infighting as the Cyberdemon, and the Cyberdemon's splash damage also creates interesting spatial considerations that the Maulotaur doesn't have. Everything has pros and cons.
The Heresiarch individually is too weak to be anything more than a flavor boss, but you could imagine that its big projectile sprays, bouncing homing fireballs, and summoning could be interesting in the context of a larger fight. It's also hard to kill, so a situation where the mapper wants to keep it around for awhile as a key component of a longer endurance battle, and the player gradually whittles down its health while fighting other stuff, makes more sense. A teleporting, summoning D'Sparil could also be interesting as a piece of a bigger battle if it weren't for the fact that it kills all enemies in the map when it dies (this can be interesting too, you're just limited to only using it in the last fight of any map).
I don't think I've seen a lot of people mentioning the Arch-Vile here, but it is Doom's most complex monster and one of the best monster designs I've ever encountered. It's not that hard individually, but it can be dangerous if you're not prepared for it, and it adds a lot of complexity to any fight that combines it with any other monsters.
If you go much more boss-tier (i.e., more challenging) than that, you're probably working against yourself (if you're trying to use it as a recurring encounter that combines with other monsters). A really powerful boss enemy will quickly wreck every monster in the arena and will become the player's entire focus, or else you'll have to give them so much space that they can basically avoid it. There are ways around that too (resurrection, summoning, advanced homing attacks maybe), but the more individually deadly you make it, the less nicely it's going to play with any other fight dynamics.