Yes they did annoy a lot of people and it is so easy to start rumours
I've started reading a biography on Lucrezia and Rodrigo doted on her as she i his only daughter he wanted the best for her but that didn't stop him from arranging marriages that will benefit him