Relationship help? Please?

Why is it wrong to visit his brother? We cannot control what our families do, but forcing him to pick between the two of you will not end well. Support bf emotionally but don't go to visit the brother.