This link is appearing under the submit button. You can see it in the footer on this page. Also there are these > appearing but only in the footer forms
The edit link only appears if you're logged in as an administrator. It's not visible to the general public when they visit your site.
The right arrows that you see in your footer are bullet images being applied to your form by your theme styles. You can refer to this FAQ item for a fix.