Interesting points across the board there.
In regards to the armor penalties, the easist fix there would be to change it to "non-Combat" skills, as that is the main intention anyways. (i.e. things like swimming, running, riding, etc.). I have done that now.
I am also removing the penalties on Bracers and Greaves. (AR and Impedance remain)
The question is whether that would be enough to counter any other issues you noticed.