agctl start goldengate hangs and fails

In this post I will discuss one of the reasons why golden gate startup may hang and fail on both nodes:

Problem:

[root@primrac1 tmp]# /u01/app/grid/xag/bin/agctl start goldengate gg_replicate
^[[ACRS-2672: Attempting to start 'xag.gg_replicate.goldengate' on 'primrac2'
CRS-2674: Start of 'xag.gg_replicate.goldengate' on 'primrac2' failed
CRS-2679: Attempting to clean 'xag.gg_replicate.goldengate' on 'primrac2'
CRS-2681: Clean of 'xag.gg_replicate.goldengate' on 'primrac2' succeeded
CRS-2563: Attempt to start resource 'xag.gg_replicate.goldengate' on 'primrac2' has failed. Will re-retry on 'primrac1' now.
CRS-2672: Attempting to start 'xag.gg_replicate.goldengate' on 'primrac1'
CRS-2674: Start of 'xag.gg_replicate.goldengate' on 'primrac1' failed
CRS-2679: Attempting to clean 'xag.gg_replicate.goldengate' on 'primrac1'
CRS-2681: Clean of 'xag.gg_replicate.goldengate' on 'primrac1' succeeded
CRS-2632: There are no more servers to try to place resource 'xag.gg_replicate.goldengate' on that would satisfy its placement policy
CRS-4000: Command Start failed, or completed with errors.

Troubleshooting:

Check crsd_scriptagent_oracle.trc trace file on each database node. For my environment the full path of this file is /u01/app/grid/diag/crs/primrac2/crs/trace/crsd_scriptagent_oracle.trc

I found that the following error was repeating in the trace file:

2020-03-05 23:36:30.467 : CLSDADR:3560113920: ERRORdynamic component [R2530613] mapped to [CLSDYNAM]

On the metalink found only this PRCR-1079 : Failed to start resource ora.oc4j, the problem is not the same, but glanced a sentence containing config file.. so here I realized it could be related to the config file. And I was correct.

Solution:

Delete existing GG resource:

# /u01/app/grid/xag/bin/agctl remove goldengate gg_replicate

Readd using correct config file location:
--config_home /GG_HOME/sm/etc/conf --var_home /GG_HOME/sm/var

# /u01/app/grid/xag/bin/agctl add goldengate gg_replicate --gg_home /GG_HOME/ma --service_manager --config_home /GG_HOME/sm/etc/conf --var_home /GG_HOME/sm/var --port 9001 --adminuser oggadmin --user oracle --group oinstall --filesystems ora.ggdg.acfsgg.acfs --db_services ora.primorcl.orclservice.svc --use_local_services

Start Golden Gate, relocate and check:

[root@primrac1 ~]# /u01/app/grid/xag/bin/agctl start goldengate gg_replicate
[root@primrac1 ~]# /u01/app/grid/xag/bin/agctl status goldengate gg_replicate
Goldengate instance 'gg_replicate' is running on primrac2
[root@primrac1 ~]# /u01/app/grid/xag/bin/agctl relocate goldengate gg_replicate
[root@primrac1 ~]# /u01/app/grid/xag/bin/agctl status goldengate gg_replicate
Goldengate instance 'gg_replicate' is running on primrac1