Hi Rakesh,
I haven't specifically looked into SEM, but in general you have a couple ways to go about this. The official, and fully supported, method would be to use the Dual-Stack Split tool (which is part of SWPM) to separate the ABAP and Java instances. Then you could use normal system copy methods with each instance.
Alternatively, it should be possible to simply copy the ABAP stack by itself using the database-dependent (i.e., backup/restore or attach/detach) method within SWPM, then afterwards just drop the Java schema from the database. However, if it were me, I would prefer to do the official dual-stack split first, even though it is more work.
I was just part of a similar discussion thread at Dual Stack system copy problems on windows.
Regards,
Matt