Inside a Jira issue, navigate to the PagerDuty link you want to remove.
Click the menu for that link and select Delete. You will be prompted to confirm if you want to delete this PagerDuty link.