Magic has to be split IMO- if you derive magic from the Valar it cannot be used to harm. If you derive magic from the song of creation it cannot directly harm (but you can make fire etc).
The Valar were not frequently worshipped directly- so, you can use local gods and spirits to customize magic a bit too. I always thoguht the Elves in particular approached the Valar a revered elders....but I digress
All depends on timeline etc too. Post WoTR there is likely more magic used, but, less powerful magic available because Sauron and his allies destroy a lot of the "arcane" knowledge in ME by over throwing places of power of Osgiliath and Fornost etc.