The recovery mode should have a password lock. When the device is encrypted and u set the pattern lock to require pattern to start device, it means that the phone will not start till u don't enter the pattern lock. This security is good to prevent theft of data. But the phone can still be used my the thief by formatting it through recovery mode. So to prevent completely the use of phone , Google should have the recovery mode password protected by the Google Id password. So if the phone is stolen / lost no one else can use it but its owner.
Second scenario is where some one sells the phone so in that case that person can delete Google account from the phone which was used to register and then sell it. This was the recovery mode shall be unlocked for use by second owner.