Another thing to consider is that, in addition to having an economical F head, the 5SFE is a stroked 3SFE, meaning the engines are the same, except the 5S has a longer stroke length to make the extra .2L of displacement. Now, because of the bore of the engine, the dimensions of the engine are very odd due to the stroking. This means that 1) the redline is low, as is top end power, 2)the engine cannot be easily turboed without extensive modifications because of the longer connecting rods and other parts which can break more easily. And, 3) the engine produces a lot of torque compared to similarly powered engines. In terms of straight numbers for both economy and power, the engine is a great 1990 product, but it is not very easy to modify without rebuilding the engine's internals significantly.
It's not that any one thing makes it tough (there are plenty of stroker 22Rs out there, and the 1MZFE has an F head, but is still often supercharged), but the combination of everything makes the design very unfriendly to typical modders.
2007 Subaru WRX Wagon, 300whp
1991 Toyota Celica in my parents' garage, not running