Is there a way to make a tech impossible to trade to/by a specific race?
Yes, there are untradeable techs within the Yor tech tree. It has its own tag.
To have a tech impossible to acquire, you still add that tech to the tech tree without a parent tech. This makes both player and (surprisingly)AI unable to research or acquire the tech by trading.
[edit] The tag is called AICannotTrade, which is a boolean true/false.
Is there a way to make a tech gives malus/bonus to relations (diplomacy) with other civs if researched?
Probably there isn't a way. One alternative to receive a new treaty that is automatically signed by all factions that have the technology.
For race specific relations there is only the "HatesFaction" tag in FactionDefs, which was created for the Iconians, who hate the Yor. Yor probably don't have a emotion called "hate".