I will try
Let's assume you have a very "cheap" shop, with 10% profit on sales and buying things at price.
This means that SHOP_SELLPROFIT() = 1.1 and SHOP_BUYPROFIT() = 1.
The shopkeeper is an ordinary mob, and thus has CHA = 11.
The player has high CHA, 18, with all spells and items.
The original calculation in sell_price would be like this:
sell_cost_modifier = 1.1 * (1 - (11 - 18) / 70) = 1.21
buy_cost_modifier = 1 * (1 + (11 -18) / 70) = 0.9
Now, this means, if we just stop there, that the player will sell an item with value 100 for 121, and buy it for 90. This is a goldmaking loophole - a player can sell/buy the same item over and over again, making a profit each time. With triggers, a player can be a millionaire in seconds.
So, the code makes sure that you can't sell for more than you buy for, with the next check.
So before actually calculating the price, we have sell_cost_modifier = buy_cost_modifier = 0.9. This way, the player will get the same amount of money when selling as he spends when buying.
If we at this point change the sell price based on race, we reopen the loophole:
(return) sell_price_for_humans = cost * 0.9 * 1.1 = cost * .99
buy price is lower, too.
So we need to change the sell_price_modifier BEFORE the loophole check.
And, because we've changed the calculation in buy_price(), we need to change it here as well.
I suspect this may be muddying the waters even more, but I hope it helps.