Accessing the Ajax registration Settings menu:-

Are you on WordPress Multisite and is BuddyPress and BuddyPress Ajax registration plugin network active? If yes, You can access the settings menu from Network Admin->Settings->BP Ajax Registration menu. In all other cases, you can access the settings from Dashboard->Settings->BuddyPress Ajax Registration Menu.

General settings:-

You can enable/disable modal feature from here. The modal allows you to enable/disable the modal for the “Login”, “Registration” or “Forget Password”.