very nice that this forum is also considered competent when it comes to RME questions
but anyway - this kind of problem usually shows up with digital sync (also called master/slave) settings.
YOU have to decide which device provides the master clock, and there can be only one.
If the RME is master (most likely because otherwise you couldn't have used it) then you have to set the VM3100 to salve to Adat (or the Adat/RBus board, whatever applies).
Otherwise, if the mixer is in 'clock-master' mode, then set the RME to slave to Adat.
There may be an extra step to activate this (I only know that mixer from photos).
I'm not sure about the 8-pin cable either (I assume it's the one to be connected to a physical Adat)
I may be completely wrong, but it could even be disturbing in this context as I don't think the RME supplies (or expects) a digital clock for the optical connections on this line. As mentioned, I don't have one myself, but you'll probably get along with the master/slave setting.
cheers, tom