We hacked Siri on iOS 7 to interface with Spotify, Venmo, and other third-party apps

Introduction

Imagine being able to control your Nest thermostat and unlock your car through Siri. With GoogolPlex, these are now possible.

We are Ajay Patel, Alex Sands, Ben Hsu, and Gagan Gupta, four freshmen in the M&T program at the University of Pennsylvania. Our project took 3rd place overall at PennApps (the largest university hackathon sponsored by companies like Google, Facebook, and Dropbox) out of over 220 projects. We’re writing this article to explain the technical aspects behind GoogolPlex. At the end of the article, we explain how you can install this on your iOS device and try out some proof of concept commands.

GoogolPlex is a hack on Siri (iOS 7 compatible) that allows Siri to integrate with third-party apps and hardware with custom commands like “turn on the lights.” Setup takes a minute and no jailbreaking is required. GoogolPlex already allows you to play songs in Spotify, pay friends through Venmo, Instagram selfies, and control Philips Hue lights with simple voice commands, all through Siri.

And what we’re even more excited about is that, in the future, third-party apps can integrate with GoogolPlex themselves and allow for their functionality to be accessed through Siri.

SiriProxy

Prior to our hack, there was a similar solution that existed called SiriProxy, which is as of now defunct. SiriProxy relied on intercepting network requests within Siri’s internal network. The problem with this is that it gave Apple an easy way to shut it down because Apple controls how its own internal network protocol requests are made and verified. Sure enough, in the iOS 7 update, Apple cutoff support for SiriProxy:

“SiriProxy does not (at this time) work with iOS 7. Significant changes made to the Siri protocol stack have rendered SiriProxy inoperable with iOS 7. … However, SiriProxy may never support iOS 7.”

In addition, SiriProxy required a great deal of setup. SiriProxy’s setup involved various complicated steps that required the user to mess with terminal commands and setup their own proxy server.

GoogolPlex, in addition to being much simpler, does not face this problem.

How We Hacked Siri

We named our hack “GoogolPlex,” to symbolize the near-infinite number of possibilities Siri could reach with our hack. But there is also a strategic and necessary reason we named it this. When a user says “GoogolPlex, turn off the lights,” Siri hears this as “Google ‘Plex turn off the lights,’” interpreting that the user wants to Google something. Siri then searches Google for the words “Plex turn off the lights,” and this is where GoogolPlex comes in. We are able to get the words the user was asking Siri by intercepting the request between Siri and Google with a man-in-the-middle exploit.

How DNS Spoofing Works

With a custom built proxy server, we make the phone connect to a custom DNS server. The DNS server system is essentially the system by which human-readable addresses like “google.com” are converted to IP addresses like “127.28.251.124.” By connecting the phone to our custom DNS server, we point “google.com” to our own server, redirecting requests to “google.com” to our server.

Putting it Together

Putting the vulnerability we found together with a proxy server that spoofs the DNS of “google.com,” we are able to send the string of words the user asked Siri to our server. Then, we can perform custom actions using third-party APIs to interact with various services and hardware.

Performing Actions

After receiving a string of commands like “Plex turn off the lights,” our server uses natural language processing methods to understand the command. If the user said “Plex tell me a programming joke,” and the appropriate response is text, we load a webpage with the response.

However, if the appropriate response is an action that requires interacting with an external API like Spotify, Venmo, Instagram, or Philips Hue, the webserver sends a HTTP Redirect Response to the phone with a custom URL scheme like “googolplex://.” This URL scheme then launches the GoogolPlex helper app installed on the phone that connects to the appropriate API and performs the custom action.

ActionPerform

Intelligent Proxy Connection and Scalability

As stated above, we use a remote server to proxy the connection between the phone and Siri to introduce a custom DNS server that spoofs “google.com.” In order to reduce load on our remote server and maintain the privacy of users, we only proxy requests from Siri that explicitly begin with “Google Plex.” This is done with a client-side decision making script that reads the request being made and decides whether or not to send the connection through the proxy server. Since the server then only deals with a few requests, and is only proxying a marginal amount of network traffic (kilobytes/a few megabytes) per user, it can easily scale as it is built on EC2 instances. Your network traffic only goes through our proxy server if you explicitly ask Siri for “GoogolPlex.” The final result is privacy for our users and lower server costs for us.

One-minute Installation

If you want to try this on your iOS Device, go to http://www.betterthansiri.com. Installation takes less than a minute and works on ANY Siri-enabled iOS device. However, since the GoogolPlex app is not available on the App Store as of now, you cannot perform actions that require the helper app, such as Spotify or Hue. However, you can try some proof-of-concept commands like “GoogolPlex tell me a programming joke.”

Where This is Headed

We plan on working on GoogolPlex in the near future to make it open for anyone to build custom commands and actions into the platform. For now, use the instructions above to try some proof-of-concept commands. While this hack demonstrates an awesome use of technology to get around restrictions, the only true way to get a great experience with third-party apps in Siri is if Apple builds in an open API.

Contact

To contact us with any inquiries/comments about GoogolPlex or this project email team@thefourloop.com.

  • http://mikespla.in mikesplain

    Nice work. Is your code open source? I’d love to take a look, if it is.

    • TheFourLoop

      Hey Mike, unfortunately it’s not open source as of now. However we’re planning to create an API to allow third-parties to integrate with us.

      • http://mikespla.in mikesplain

        That’s a real shame but keep up the good work.

        • *bit

          How is that a real shame? It’s their code after all. Beside, they are promising to open up the API. You can learn a lot from reading the API documentation once they release it.

          • http://mikespla.in mikesplain

            Honestly, myself and many people can’t justify piping all their data through a third party service without proper checks and balances. One way is open source… let the community see what the code does to make sure it’s not malicious (like https://github.com/plamoni/SiriProxy). Of course this isn’t always possible, and in that case a terms of service or privacy policy would be nice.

          • Oliver

            I have followed your instructions and added the proxy server on my WiFi network but when saying “google plex tell me a programming joke” safari just googles this and thats all. Any hints what I can do to fix this? Thanks!

          • http://www.digital-images.co.uk Mattrbailey25

            Hi All,nnGreat Hack, stumbled across this while trying to find a way to set home automation up with Voice control.nnAnyway looks like i’ve found a way to force google.com searches. I found the following article on a forum;nn”So, for some reason if you want to use http://www.google.com instead of nyour local Google, then you can do that easily. Just use the URL http://www.google.com/ncrn in your browseru2019s address bar. Here ncr stands for No Country Redirect.n Once you load that URL in your browser, it will create a cookie in yourn browser, which will stop redirection from Google.com to local Google domain.”nnGoogle.com doesn’t redirect to .co.uk for me anymore, so should work for non UK as well. Keep up the great work.

  • iComputerGeek101

    When will the app come out?

    • TheFourLoop

      We are working on it, 1-2 weeks most likely.

      • Vinod Tonangi

        What do you think the possibilities of Apple not approving it are?

        • TheFourLoop

          We had a team meeting and decided to scrap the app. It is going to work without an iPhone App, all built as a webapp. This should remove any influence Apple has over this. Stay tuned!

          • Vinod Tonangi

            Awesome. Can’t wait.

  • http://www.github.com/plamoni Pete Lamonica

    Very cool. Even if we never get SiriProxy working again, it’s nice to know there are other options. We’ve made a bit of progress, though. So it’s not looking as bleak as it was. But keep up the awesome work! :-)

    • TheFourLoop

      Thanks Pete! As we build our system for opening a public API I’m sure we would all be interested in your suggestions. Shoot us an email at team@thefourloop.com!

  • http://sciencedem.blogspot.com editor

    How do you integrate with Hue; said “GooglePlex turn off the lights” Got Cannot open page; address invalid.

    • TheFourLoop

      To integrate with Hue you originally needed the helper iOS app. However, we’re currently building out GoogolPlex so you can do this without the iOS app. Stay tuned!

      • http://sciencedem.blogspot.com editor

        Sounds cool; should use more natural language like “Google Please turn off the lights

        • TheFourLoop

          In the final version, this will be possible. Stay tuned!

          • http://sciencedem.blogspot.com editor

            Wemo integration too. “Turn on the fan”

  • [anonymous]

    I just set up the proxy, but I can’t set custom responses. It says I can”add more on a computer”. Whenever I open betterthansiri.com on my Mac, I just get the mobile-style start screen with no prompt to log in.

    • TheFourLoop

      We weren’t launched back then, but now you can go to http://www.betterthansiri.com, create an account, and start to develop for the platform!

  • anon

    you must select “Continue” after making your first GoogolPlex requestnnI am never presented with this option – what is going on?

    • Mark Green

      same here… where is the continue option? when i say the command it takes me to google

      • Tom

        doesn’t even take me to Google

  • hans-peter

    When is the app available on the appstore?

    • TheFourLoop

      There is no longer an app! Just go to betterthansiri.com for easy setup!

  • Tom

    not working on my side

    • TheFourLoop

      Sorry to hear that, Tom. Shoot us an email at team@thefourloop.com and we’ll see if we can help!

  • http://www.teklinksystems.com Amir Servatdjoo

    Can this work on 4G as well or does it only work with WiFi?

    • TheFourLoop

      We’re working on getting it to work on 4G, but it only works on WiFi right now. Stay tuned!

  • Rohit

    As others are reporting this simply is not working on atleast Apple iOS 6, any ideas guys?

    • hdfreak

      You realize the title of this article is “We hacked Siri on iOS 7 to interface with Spotify, Venmo, and other third-party apps”?

  • kevin nadjarian

    Hi, I cannot find the Hue integration when logged in your Web site.nnAnything I should do for that ?nnnnKind regards

  • Test

    searching google plex, tell me a joke. just googles tell me a joke

    • CD

      Same here, it doesnt seem to work for me

    • Evan Petousis

      If Google redirects you to google.com.au or anything else other than “google.com” then it won’t work.

      • Dan

        Is there not a fix for this. The iPhone in the UK is auto set to google.co.uk

        • TheFourLoop

          Unfortunately, GoogolPlex currently works only in the US. But stay tuned!

          • JWin

            In what timeframe do you think it will work outside US. Really looking forward to trying it!

  • Miguel Tavera

    Don’t work in other languages, not in spanish

  • Anuj Rajput

    It seems it only works when Google.com is opened. Doesn’t work on other Google domains

  • GeorgesVDR

    I think when you’re safari doesn’t search on the “google.COM” it does not work. I’m in belgium so it is google.be. Until now i haven’t found way to change this default setting. Does anyone have an idea??

  • maxlds

    I’d love to see IFTTT integration!

    • TheFourLoop

      The coolest part of the GoogolPlex platform is that you can develop your own commands! If you develop cool IFTTT integration, email us the link to your command, and we may feature it!

      • mobilemann

        make it opensource, include instructions on how to build a server, and people will actually do it. The majority of us who ran a siri proxy server, (no offense) would prefer to run the server side of things as well. (which would also cut down on your bills:D

      • mobilemann

        Thanks for no response at all; it is a great vote of confidence that you aren’t snooping. Fucks sake.

  • MHayes

    Does this require a wifi connection to operate?

    • TheFourLoop

      Yep, GoogolPlex, currently requires a WiFi connection, but we are working on extended setup that will work on 4G/etc.

  • A

    vulnerable to XSS

  • Mark Howard Braun

    I love it. It works like a champ. Works great with Spotify. Nest is trickier, although it does away really good. You guys are great ! Put me on your mailing list please.

    • TheFourLoop

      Thanks Mark, we appreciate the nice words!

  • Eleiu Sam Cruz

    doesn’t work for me

  • Eli

    So are you guys saving everyones searches?

    • TheFourLoop

      Nope. We’d like to stress that we ONLY look at the searches that are through Google and that contain the keyword Plex. We do not save any of your search information.

  • jimcvit

    I tried adding this. When I say it, siri comes back with Google plaques….blah blah. nothing works.

  • Deon

    This did not happened to me: “The first time you use GoogolPlex you must select “Continue” when prompted and create an account. You will only have to do this once.”

  • Sandeep

    Telling Siri googolplex tell me a programming joke only directs me to the google search results with all articles mentioning this project..does not take me to the set up page

  • Mmmm

    Google searches are geolocation specific.

  • Random

    I’m from Canada, where Siri auto uses google.ca to search. When searching with google plex, at the bottom of safari after it opens up there should be a button that says “use google.com” and if you click it the search goes to the google plex server and works (so you have to click it every time you search). Not sure if this is the case for other countries

    • cool

      now if I could integrate this with an Arduino that would be swell

  • Tushar

    I didnt get the “Continue” option. I’m trying this out on an iPhone 4S with iOS 7. Saying the command “GoogolPlex, tell me a joke” returns – “Googling “Plex tell me a joke” “

    • Tushar

      i just went through the comment chain and found that it’ll only work in US. When will this service start working in India?

  • Mat48

    hi i can’t get this to work – i’m in the UK but connected via Witopia VPN to Seattle – so Google.com works, and I’ve logged in to betterthansiri . Any suggestions ?

  • http://www.digital-images.co.uk Mattrbailey25

    Ok late night fail. Posted this as a reply at the end of the forum!!nnHi All,nnGreat Hack, stumbled across this while trying to find a way to set home automation up with Voice control.nnAnyway looks like i’ve found a way to force google.com searches. I found the following article on a forum;nn”So, for some reason if you want to use http://www.google.com instead of nyour local Google, then you can do that easily. Just use the URL http://www.google.com/ncrn in your browseru2019s address bar. Here ncr stands for No Country Redirect.n Once you load that URL in your browser, it will create a cookie in yourn browser, which will stop redirection from Google.com to local Google domain.”nnGoogle.com doesn’t redirect to .co.uk for me anymore, so should work for non UK as well. Keep up the great work.

  • Aa

    Will you open source the backend so people can set up their own (local) proxyserver?

  • Rogelio Rup

    hi i managed to get a fully working siri integration with googolplex, i must say its great, i already made some scripts to control my self-made home automation system and it works like charm, im stucked in getting it work with my 4G/LTE network, when i access the extended setup page i get this message nnAn Error Was EncounterednnUnable to load the requested file: extended_setup.php nnnni hope you guys solve this soon, cheers

    • Jelimoore

      I’d try some sort of VPN server on your computer and run the VPN server through the Googolplex proxy… Just guessing.

  • LUDDERS

    I have a custom made home automation system, it is currently voice controlled using a wireless mic on iPhone/ipad. I also have a remote app to switch things on/off manually I would love to use googolplex to voice control the remote app, is this possible?

  • Suku

    Hi,nDoes it only use free web services for creating the “commands”? Can I create a command as, “read my new email”?nnnExcellent concept .. Kudos to all involved in creating GoogolPlex.nnnCheers !

  • Mark Easy

    I have a wemo device to turn on/off the lights. Can you add it to your list? (((Please)))

  • LUDDERS

    Can the platform access an online controller via its IP address and port instead of third party app?

  • Neal Piche

    Will you be open sourcing the server code? For those of us that are REALLY paranoid to set up our own server? it seems that now it is a proxy that all traffic would hit it first? Am I mistaken in this assumption?

  • Neal Piche

    is this working for ANYONE today? it says server not responding