Clicking on the custom field within the grid should show the edit page.

From the edit page, we can delete it.

Remove the visual logic from the actions menu in the list page.

Do the same for Custom Field Groups.