The log shows "Received notify: PAYLOAD_MALFORMED"

The log shows "Received notify: PAYLOAD_MALFORMED"

Received notify: PAYLOAD_MALFORMED

Although indicative of payload packet malformed in transit, this error is also returned during IKE negotiation failure due to Preshared secret mismatch.

Most likely the Preshared secret does not match.

Check the SAs in both units.

Navigate to Network - IPSec VPN - Rules and Settings - Active Tunnels.

If the above is not the cause of your issue, try lowering the MTU size on the WAN interfaces and check 'Fragment outbound packets larger than WAN MTU'.

Navigate to Network - Interfaces - Configure WAN interface