"I believe that she's really into you, but you're not into her," Hartsock's brother says