All praise is due to Allah. May Allah's peace and blessings be upon the Messenger of Allah, his family and companions.
To proceed:
To answer your question I say:
Firstly: I advise you to fear Allah, and to use the divorce pronouncements in issues like this. As for what happened; the way out is that you should expiate for the oath because what is apparent from your question is that you mentioned divorce in order to prevent your wife from going to her family's house. The ruling for such cases is the ruling of the oath according to the preponderant opinion of the scholars. As for a man's telling his wife, "You are prohibited to me;" this is prohibited, and Allah the Exalted criticized His Messenger for prohibiting what Allah has permitted. Allah said, " O Prophet, why do you prohibit [yourself from] what Allah has made lawful for you" [at-Tahrim: 1]. Thus the way out from this is through what was mentioned in Surah at-Tahrim, and that is the expiation for an oath because Allah said, " Allah has already ordained for you [Muslims] the dissolution of your oaths" [at-Tahrim: 2]. Thus, it is imperative for the person who utters this statement to expiate his oath, and this is what was mentioned by Allah in His statement: " Allah will not impose blame upon you for what is meaningless in your oaths, but He will impose blame upon you for [breaking] what you intended of oaths. So its expiation is the feeding of ten needy people or from the average of that which you feed your [own] families or clothing them " [al-Maaidah: