Answer
Atman, I have noted your problem. You are not alone in suffering from this problem — in the present age, a large number of people are afflicted by this kind of problem. But they are unaware that it is a "problem."
Within you, there is a strong and hopeful ray of light — you are aware that this is a "problem," and you wish to be free from it.
This very desire for liberation will certainly pull you out of it — it will most assuredly pull you out, because the wish for it has awakened in you. The fact that you are part of this group and are asking such questions is an indication that Ishvara's grace is in the process of descending upon you.
The Lord has said in the Gita:
Daivi hyesha gunamayi mama maya duratyaya | Mameva ye prapadyante mayametam taranti te || (Bhagavad Gita 7.14)
This divine Maya of Mine, constituted of the three gunas — sattva, rajas, and tamas — is exceedingly difficult to overcome. But those who take refuge in Me alone cross beyond this Maya.
Therefore, remain in the Lord's refuge — by His grace, you will overcome this crisis.
From a general standpoint, a person turns away from sinful actions for two reasons: one is fear, and the other is inner inspiration.
Fear arises when the consequences of sin become clear to the person. Maintaining a love affair with a married woman can at some point become a great cause of dishonor, enmity, and mental anguish. If this love affair turns into an illicit physical relationship, it will constitute a maha-papa (grave sin).
Its consequences in the afterlife are extremely dire. In this regard, there is a story in the Mahabharata which tells that King Nahusha, an ancestor of Shri Krishna, was an exceedingly radiant and righteous king. He even attained the throne of Indra during Indra's absence. But he became consumed with lust for Shachi, the wife of Indra. Although the queen of Devaraja Indra managed to protect herself somehow, as a consequence of this incident, King Nahusha's next birth was as a python. Now you can imagine how dire the consequence of lust can be from the perspective of the afterlife.
It is a good thing that you yourself are troubled by this. Keep one simple principle in mind: every emotion is kindled in the mind by some external stimulus. Viewing lustful images, watching lustful videos, reading lustful literature — these must be avoided because they inflame lust.
In Shrimad Bhagavad Gita, Chapter 7, Verse 11, Bhagavan Shri Krishna states clearly:
"Dharmaviruddho bhuteshu kamo'smi Bharatarshabha" — "I am that kama (desire/sexuality) in beings which is not opposed to dharma."
This verse clarifies that kama (sexual desire) is a natural and God-given power, and if it is channeled along the path of dharma, it is itself a manifestation of Ishvara.
That is to say, the Lord has prohibited only adharmic sexual desire and adharmic sexual conduct. Sexual union with one's own life partner is a natural source of joy, and no dharma shastra prohibits it. But adharmic sexual conduct is a grave sin.
Even if you wish to, never speak on the phone with your married beloved. Divert your mind and engage it elsewhere. Sexual desire is a play of the senses and the mind. The instrument the Lord has prescribed for controlling both of these is the buddhi (intellect). The way to purify and refine the intellect is the study of the shastras. Read the character of Shri Rama in the Ramcharitmanas daily. This lust will be destroyed by that very practice. Reading the Gita, the Ramayana, and the Bhagavata Purana is an excellent means of refining the intellect. Whenever you get the opportunity, read these.
While chanting the Lord's Name, also keep a picture of the Lord in front of you and keep gazing at it. If the mind wanders, refocus your eyes on the picture.
When the thought of that woman arises, behold Bhagavati Durga within her. When the minds of sannyasi-sadhaks begin to get attached to a woman, they use this method to save themselves from wrongdoing. Hari Sharanam!