Shared publicly  - 
Turning your Nexus 5 into a Moto X: Always-On Listening

When the Nexus 5 came around, a lot of people asked whether or not it would have a very popular feature of the Moto X: it's famous "Ok Google Now" wake-up hotword. People wondered if the MSM8974 chipset of the Nexus 5 supported Always-On Listening, considering various information from Qualcomm.

Well, here's your answer: Nexus 5 - Always-On Listening Proof of Concept

The hardware is there and the software is capable of handling it. It's far from being stable, and not releaseable in the current state (for various reasons). This is only just a proof-of-concept to show that it's possible.

I hope to be able to bring it to more people in the future, but not in the meantime.

EDIT: To further clarify, this is indeed using the dedicated audio processing chip of the S800 - that's the whole point of this PoC. The CPU is indeed turned off and isn't used at all for the audio processing.
Jeevan Ghadge's profile photoItamar goldstein's profile photoRaj Vernekar's profile photoAndrew Dodd's profile photo
Moto X has a dedicated core for that. It is impossible to achieve the same without sacrificing battery life. 
+Elison Niven Even the Snapdragon 800 has a build in core for voice interpretation when the main CPU is offline. OFC is different from the Motorola dedicated core.
+Elison Niven Indeed, as +Davide Brutti said, the S800 also has dedicated processing power. Even if it's not shown in the video, it is using the capabilities of the WCD audio codec to wake up the audio DSP when the mic reaches a certain level. The audio is then matched against a predefined sound model, which will wake up the CPU above a certain match threshold. The CPU is otherwise inactive.
Skeptical until I see how much longevity is reduced
Can't wait to try this sort of things. Will it be included as a kernel?
I've had them since nexus 4 days. Love them.
You are really taking Android to a higher level: THANK YOU!
+Blin Qipa The kernel didn't need any change, but all the audio chain needed changes (audio HAL, audio_policy, and various libs to rebuild and add). Unfortunately I can't release those changed files.
It will be great to see this working in a battery efficient manner.. Please update us with your findings.
Haha... So this is the something cool that you did the other day?
Chris P
Last time I trusted some engineers and electronic specialists from India who talked big, I got burned. Never again.
+Elison Niven true. But, Qualcomm's individual cores have independent clock speeds.... So, one of them can remain in low power mode and listen...
Looks cool. I'll wating for it.
+Moosa Mahsoom My guess (I haven't been in touch with anyone recently so haven't really chatted with +Guillaume Lesniak in a while) would be that this is not using the applications cores, but instead leveraging the QDSP6V5, a separate low-power DSP core.

In which case there may be legal issues (redistribution licensing) with some of the components.  Qualcomm is kinda notorious for that.

+Guillaume Lesniak Did you do this with the Hexagon SDK?
So this make you be special ...! Next time don't want to share keep it for your self.
+Eppu Haavisto My guess, based on some of Guillaume's comments is, "indeterminate due to lawyers".  (Think along the lines of the Nexus 4 factory image fiasco) - again this is only a personal guess.
If and when this becomes publicly available would it be easily ported to other devices with the snapdragon 800? (Running a custom ROM)
"Unfortunately I can't release those changed files." -- Interesting turn of events, indeed.
I think this would be a cool feature on our nexus 5 when it gets released. Hope it won't sacrifice our battery very much. 
Even if you do this, the functionality of a Moto X can not be replicated. Things like twisting wrist to open the camera were immediately missed for me when I had the nexus 5, it sounds trivial until you've been doing it for months without ever clicking an app but it really is a rich feature to me. Active display has apps like it but with out the accelerometer the functions are nothing near the Moto X.

It's a good thing nexus 5 owners like seeing this in my eyes. The Moto X features really are great and it would be cool to see them implemented into ASOP even though that will never happen. 
+John Burke This is one thing way beyond the capabilities of Xposed.  Based on the hints given, this goes down to the level of (most likely) modified qdsp6v5 firmware.
Listening is only part of the moto x. Contextual awareness is a big part also. 
My friend you just won all my attention. Fucking amazing hob.
Sean G
I'm not a fan of always on listening. ... unless +Guillaume Lesniak can confirm that its only listening to 1 key word? Otherwise creepy... since I don't trust the Gov
+Sean G Without entering into details, it's indeed matching audio against a prepared sound model.
Sean G
+Guillaume Lesniak Thanks! That makes me feel better. I wanna learn how to hack to read deeper into code. To know what is going on. Thanks to Dev like our backs.
When can I Has??? :) great job BTW... This will help so much with hands free driving! 
The always on thing is great.  Have been enjoying it for a long time on my droid maxx. Very useful.
This is so good to hear. I knew there was a reason I chose OMNI over everyone else.
OK Google... Turn everything off to save battery life?

+Guillaume Lesniak sounds amazing the nexus 5 community would love to be able to use this feature great work getting it to this stage and I hope this could be released eventually keep pushing the boundaries
Nice! Found this through XDA.
So no chance for Nexus4 to get this baby working?
+Matzen Schiller You need a low power processing core for this to not eat your battery, and the only mainstream processor right now supporting this is the Snapdragon 800 if I'm not mistaken, so sadly, I guess, no. The only way to get voice activation on devices without it would be having the CPU handle it, which would prevent deep sleep and really cripple battery life.
+Guillaume Lesniak Since you seem to have legal concerns, is there any way of having this in a custom ROM or distributing it somehow? I have no idea what changes this requires, but I want to know if I can look forward to something or not, heh. Thanks for doing this!
You are a luminous being. I wish I was smart enough to help, Godspeed. This is a feature that I've been wishing for for ages.
+Lars Wyness Sadly it seems like this won't ever be released, as it would require some files to be distributed which are illegal to distribute.
+Guillaume Lesniak, do you think you will be able to release this within the next month or are you thinking that the issues will make it so you can't release it at all? 
would that also be possible with a snapdragon s600 with the Hexagon, QDSP6 instead of the Hexagon QDSP6V5A in the s800?
That is useless if it can't go public.I know s800 can do this job long long ago.But I just can't find a way to enable it.This is like the pedometer.The only useable app isn't built-in or showed in recommended app.So I find the software very hard then I finally make pedometer useable.
Pleeeeaaaase give us a link to a thread where you update the progress of this! Unstable May kill your cat and burn your house down version would be so greatly appreciated! 
+Yusuf Islam For those thinking that the dedicated processor in the S800 is "disabled", understand that it isn't disabled, it's being used for other resources. Google made this feature refer to the motion sensors for things like counting steps, etc, instead of using it for voice detection purposes. 

Granted, the huge majority of us would prefer the voice detection over the (practically) useless step counter...
It's so disappointing that we'll never see this actually used because, like others have said, distributing the code would lead to legal issues...
This is awesome! I really hope you can get around the legal issues... I tried to do this for a long time, but had no luck, if you can I would love to chat about how you accomplished this even if it can never get released...
If it can never get released why cock tease the development community with it.
This is exciting.  I've tried an app that does this, but it's still in development and causes my phone to echo badly when it's active while on a call.  Looking forward to see how this development goes.  Maybe it'll work on my soon-to-be-acquired sister to the N5, my LG G2.
This is one of the key features of the Snapdragon 800 which I've yet to see get utilized by an OEM. Wonder why no one has done it yet. Good work Xplodwild, you're an asset to the community.
+Michael Bond Thanks! A formal thread to update us on how the dev plans to carry this forward was more what I was looking for though. 
Did this take a custom ROM, or can it be achieved with an app?
+Tim Martin It's a pretty complicated mod. And it won't be released, at least not like it is currently done.
+Justin Myers if this gets released and works, my guess is it will be ported to most s800 chips.
+Guillaume Lesniak  just bake it into your ROM and let us 'discover it' there. No need to release any code :P . Seriously though why won't qualcomm release APIs for it, and why do they care if you are making thier processor look good and giving the Nexus 5 a new lease on life.
Just a few words: Nice work dude!! :)
Why mention this mod if you aren't even going to share it. Makes no sense. Kind of a douche move.
Why did everyone go silent on this? I bought the nexus 5 betting the mod community would figure this out. I'm sad it has taken this long. Can't win em all.

+Chance Stad this took a lot of work in breaking down snap dragon professor code, if he gets away with releasing this to the public, it's going to take a long time to perfect! 
+Ian Thomas This has, figuratively, NOTHING to do with the Moto X... Different device, different core, etc.
+Open Mic+ for Google Now hopefully google and qualcom will knock their heads together for 4.5 builds and release this for 800 based SoC's on the nexus programn.

This coupled with Android wear will be awesome, use the android wear to do this function aswel as phone opens up many options for  customers.

not to mention an new edge over Samsung / Apple / Windows 
+Daniel Hutson This wouldn't really give them an edge on Samsung or HTC considering the fact that those devices have the same SoC. Unless this is going to be a closed-source project, which would be very unbecoming on Google's part...
it will if its based on Pure android for Nexus and GPE devices.

GPE and nexus devices are soley sold via Google play, and would intice custormers to purchase said devices rather than picking a device via their carriers subsidised devices.

its also similar software to SIRI / S-Voice e.c.t so a 300 dollar phone with the same SoC that offers the same as the competitors is a better bait than one without or a 500 dollar+ device.

I sure know what I would do if I was on a tight budget  / SIM only / Replacment handset 
+Guillaume Lesniak 
what's the propuse of this post if you're not going to share your findings or even only updates about progress? i know there is a legit issue  but at least you can update whats going on... of course your not owe nothong to anyone but it's you that share this post from first place.... it's a little bit dissapoing 
+Itamar goldstein he submitted a proof demonstrating a feature... He's said plenty since then to the effect of this possibly never reaching mainstream. He has no obligation to anyone for sharing his work or providing antsy enthusiasts with updates...
+Nicolas Finn If I’d liked to know your opinion I would write your name instead of guillaume's in my previous post. Keep your arogance to yourself. I really appreciates guillaume's work as a Dev,  there is no need to mention he doesn’t have obligations to anyone... it's funny to think otherwise.
+Itamar goldstein Arrogance* isn't the right word for what I said. 

Also, if you were looking for his opinion alone, you should probably message him directly. 
I wish we could somehow get our hands on those altered files ;) are there any other devs working on getting the feature more stable, or getting some of the legal issues worked out? 
Agreed. Maybe if the files or method of writing up the code for this was torrentable it would help the cause with less risk to litigation to the original coder. We could all just code and compile this mod individually 
If it is possible than i have to wait for touchlless control in my nexus 5
So is the Qualcomm SDK for the DSP available to developers now? There's lots of other stuff we can do besides waking up... Would be really great to have the sdk or an API. 
What will happen if you release this? Are they afraid of something being compromised? What's wrong with developing this?

Anyone know anyone else working on this?

What if we started a bounty to release this? 
+John Friend If I release this I could be sued for releasing stuff for which I don't have the rights, and the source of where it comes from could have troubles as well.
+Guillaume Lesniak , I've seen Qualcomm have an SDK available for download by registered developers. They do ask to use a 'corporate' email id to register, if you want to be considered favourably. Do you know if they would agree to small IT/android development cos?
Secondly, looks like Cyanogenmod did get the SDK right? Else they wouldn't have the Oneplus one voice wakeup stuff.
+Jeevan Ghadge I'm not sure, but it may be that to deploy in production you need to pay licensing fees.

CyanogenMod didn't get the SDK, Cyanogen Inc. did.  And it appears that only the OnePlus One is licensed.

In theory, it SHOULD be possible to get it running in AOSP derivative firmwares on devices where the vendor had it in their factory firmware, and in those cases it's less of a legal grey area.  But so far not many people have had the time to play with it.
+Jeevan Ghadge I have the Hexagon SDK right here, and so far it doesn't contain any "voice activation", Qcom explaning that it's "coming in an upcoming update". The current VoiceWakeUp SDK is a separate SDK containing only that, and licensed to corporate partners that paid the license.
Cyanogen Inc licensed the VWU SDK and integrated it into the OnePlus One in the commercial/closed release of CM.
Hey Guillame, thanks very much for sharing on this topic. I'm thinking of some features in my app, where certain words are spoken to get certain response from the app. I expect to use them when the screen is on. Does that need the voicewakeup sdk at all? Or will the hexagon sdk suffice? Or are there apis within Google voice search that can be used? I'd like to use offline recognition as far as possible. I saw your omniRom custom hotwords video. Looking for similar stuff really. 
+Jeevan Ghadge For use of Google's voice recognition API, see Omni's Launcher3 commit history.

For generic offline recognition - not sure where to look.

Qualcomm's voicewakeup is designed to do ONE thing and do it well - wake up the device in response to a voice command.  All other voice-driven features are done using different software/approaches than Snapdragon Voice Activation.

It's been kind of mentioned previously in a few other places (including, I think, earlier comments here...) that SVA is tightly integrated with a number of specific Qualcomm features:
1)  The audio chip can be put into a mode where it issues an interrupt to the host CPU when the input level exceeds a certain value.  (e.g. only process loud sounds)
2)  The hexagon DSP is very power-efficient and can process data from audio chip wakeups efficiently.  With SVA - interrupts from the audio codec don't wake the apps CPUs, they wake up the Hexagon instead.
3)  Only once the Hexagon matches a predefined wakeup pattern does it wake up the host to do any further processing.

e.g. SVA does not solve the "what did the user say?" problem, it solves the very specific "Did the user say X?  If they said anything other than X I don't care" problem.
+Guillaume Lesniak do you think that the final version of lollipop will make use of the low power core on nexus 5? (For "always listening")
Please can you release this as an app? N5 users wud surely download or even buy it :)
+Raj Vernekar This absolutely cannot be released as an app.  Implementing it requires changes to the audio HAL and a number of other low-level system components.

Also, has he indicated - he can't release this due to licensing issues.  Qualcomm owns the technology.
Add a comment...