For me the main purpose of a totem is to take the initial big hits from MOBs.
When it comes to trolls and ogres, the build matters more than a totem (IMHO).
If you imbue out a nice set with pyro, it is pretty easy to burn down elder trolls.
If you imbue out a nice set with cryo, orges can be nearly permanently frozen. Group recommended.
For use of the totem, it can be tricky but it can be worked around - especially if you can take aggro from the totem before it goes down.
While not ideal perhaps, using the totem to take the big hits up front can be a big help. In graveyards I use it to group mobs for AoEs. It is very useful.