Removing an accidental LoopBack

[azrael1111]

I am working on an AudioSwitcher application and I am using multiple audiochannels.

The structure of the audio devices is separated into 3 layers.
1 ) several IO audio devices (HD and/or FD channels)
2 ) several virtual audio devices
3 ) several IO audio devices. (HD and/or FD channels)

1 and 3 comunicate should communicate through 2 where 2 is recording the audio traffic. I did it this way so I would have a single point of recording.
My problem is that as soon as I would plug them into the virtual audio devices and one of the 1 or 3 would say something, He would automaticly get an loopback effect.
I could bypass it by using more virtual audio devices but the switching between them would be a bit complicated.
I thought of connecting them directly via an IO matrix but then I again loose the purpose of the virtual audio devices.

I am connecting the devices tto sinkPins directly.

Since I am hearing the "Echo" of my own voice, which I would prefer not to, Is there any suggestion from someone how to solve this problem.
Is there maybe an echo cancellation solution in audiolab or signal lab.
Any other solution from someone?