In Raw Profit mode I set fixed dollar to 5,000. Yet my trades are well above that $5000 limit. For example, first trade is over 2400 shares at price of over $29.
If I switch to Raw profit mode > Shares/Contracts it obeys the settings.
If I create a new basic simple strategy the RP mode limit is working.
Any ideas what is wrong?

Give us a concrete example how to reproduce that includes:

+ Margin settings
+ Symbol
+ Price
+ Screenshot to proof
+ Sym Info Mgr settings

The problem was in the buy limit code. The limit was set to an indicator level amount not in the pricepane but rather a complete different scale.
Example: ShortAtLimit(bar+1, rsi[bar]);

Still, it does not make sense that the program will allow the position to go 3 times above the raw price limit. In my case the RP limit was 10,000 but the trade position was more than 30,000.

When you specify a limit price, the share size for the Position is the specified dollar size divided by the limit price. That's how it works, it's by design.

Although the mistake was yours in this case, it works that way in the real world too. Imagine a stock that closes at $50, but opens the next day at $100 for some reason. A market order for $5,000 would be sized for 100 shares. If you entered that order, went to work, and ignored the session open, you'd quickly find that you just bought $10,000 worth of stock... as long as your account had the funds for it.

In Raw Profit mode, there is no concept of an equity balance, so it always has "funds" to make the trade.

I think know what might have happened. On the signal bar* there was a significant decrease in price (x times), a large gap or simply a data glitch. The enormous position size was determined considering the penultimate bar, and on the bar of entry things went back to norm. However, there's no adjustment for this unless you switch to a Portfolio Simulation option.

* User Guide > Strategy Window > Backtesting Strategies > 100% of Equity Sizing > Basis price.

EDIT: Went AFK while preparing my reply and Cone was faster :)