Logic of holiday accommodation prblm

when we are taking the minimum size between (left subtree) and (right subtree including the root) what will happen to the nodes that are left over since we are taking the minimum? don’t we have to compute the cost to exchange their positions since all the travellers will travel.