Summary of Findings
1) Your first defensive item must always be a HP item.
2) When you reach 1800 ? 1900 HP this is the point where additional Health or additional Armor/Magic Resist (MR) have equal effects. Therefore you can start to alternate your items, buying more health then buying Armor/MR, or buying Armor/MR then buying health. You can also buy items that give both health and Armor/MR.
First of all we need to figure out the cost of HP, armor and magic resistance
Ruby Crystal:?180 Health,?475 Gold
Gold Cost/Health = 2.64
Giant?s Belt: 430 Health, 1110 Gold
Gold Cost/Health = ?2.58
Cloth Armor: 18 Armor, 300 Gold
Gold Cost/Armor = 16.7
Chain Vest: 45 Armor, 700 Gold
Gold Cost/Armor= 15.6
Null-Magic Mantle: 24 MR, 400 Gold
Gold Cost/MR = 16.7
Negatron Cloak: 48 MR, 740 Gold
Gold Cost/MR = 15.4
Effective HP (assuming 50% physical dmg and 50% magic dmg)
Health X (A+MR+200)/200
1) Lets say you are shen, a simple tank. at level 6 you have 850 base hp, 39 A, 30 MR
Your Effective HP is currently 850 X (39 + 30 + 200)/200
EHP = 1143.25
2) And suppose you have 500 gold to spend
If you spend it all buying HP, your stats become 1040 HP, 39 A, 30 MR
and your EHP = 1398.8
If you spend it all buying Armor, Your stats become 850 HP, 69A, 30 MR
making your EHP = 1270.75
If you spend it all buying Magic Resist, Your EHP will also be 1270.75
Conclusion here: Buying HP items is better.
What about at Higher Levels?
Lets say you are at lv 18 and you haven?t bought any defensive items yet, what should you buy?
At level 18 your base stats are hp 1873, 87 A, 30 MR
EHP = 2969
If you have 1400 gold to spend on a defensive item:
Buying just HP will give you these stats 2416 HP, 87 A, 30 MR
giving you EHP = 3829.36
Buying just Armor ?will give you these stats 1873 HP, 177 A , 30 MR)
giving you EHP = 3811.56
Interesting, both seem to give you a similar EHP.
Conclusion here: When your health is at around 1800 ? 1900 the effects of additional health and additional armor/magic resist is close to equal.
What if I already bought some HP items?
Intuitively it should be clear that if you have already bought an HP item and your health is well over 2000, it is better to buy armor/magic resist.
But lets put some numbers in.
Assuming you already bought some HP in the previous section, we are at level 18 with bonus HP giving us stats of 2416 HP, 87 A, 30 MR.?Our EHP is 3829
If we have another 1400 gold to spend,
And we buy all HP, our stats become 2959 HP, 87 A, 30 MR.
giving us EHP = 4690
If we buy all Armor, our stats become 2416 HP, 177 A, 30 MR.
giving us EHP = 4916.56
Conclusion here: Buying armor or magic resist is better when you have already bought your initial HP item and your health is around 2k or more.
Additional Finding (very mathematical, not very practical): The ideal and optimal HP to Armor+MR Ratio is 16. Example: If you have 100 Armor and 50 MR, your Health should be (100+50) X 16 = 2400 HP to achieve the optimal amount of effective hp for the lowest gold cost. Any lower and you should buy HP, any higher and you should buy Armor/MR. But of course you can?t really buy exact amounts of stuff so just follow the rough guidelines in the summary