api | Blue is Me

Siri, the personal digital assistant

Recently I’ve been quite interested in Apple’s Siri, ever since I received my iPhone 4S and watching the WWDC Developer’s conference. At first I thought it was just a novelty, but upon seeing the upcoming features in iOS 6, my interest has been heightened. I’m interested in machine learning, AI, UI design, and aggregating content, as I’ve tried to do before. See: History of speech AI and How Siri works.

Concept and Functionality
Siri is essentially a search / AI engine thats powered by the voice.

The original Siri application relied upon a number of partners, including:
OpenTable, Gayot, CitySearch, BooRah, Yelp, Yahoo Local, ReserveTravel, Localeze for restaurant and business questions and actions;
Eventful, StubHub, and LiveKick for events and concert information;
MovieTickets, Rotten Tomatoes, and the New York Times for movie information and reviews;
Bing Answers and Wolfram Alpha for factual question answering;
Bing, Yahoo, and Google for web search.
Apple integrated it with default iOS functionality, such as contacts, calendars and text messages. It also supports search from Google, Bing, Yahoo, Wolfram Alpha and Wikipedia. Siri also works with Google Maps and Yelp! search in the United States only.

Context specific service searching
What I would like to see is Siri searching certain services depending on what keywords are being said. For example, if I say a query such as “What is the Earth’s circumference?” or “What is the area of a circle?” then it should direct me to Wolfram Alpha because Siri recognizes these queries as mathematical questions.
Asking about the news should take me to the New York Times, CNN or maybe show me some blog posts, tweets, etc.
Famous persons or companies can take me to their wikipedia page, twitter page, facebook page and website.
If I asked a knowledge question such as “Who was Rene Descartes?” or “Who invented basketball?” then it should direct me to a Wikipedia article or Yahoo/Bing Answers page. Location queries should pull up Google (or soon Apple) Maps.
Programming questions should take me to StackOverflow or some related forums.
Food queries will pull up Yelp and OpenTable to make a reservation and read reviews.
Asking about travel should pull up Priceline/Expedia/Kayak and TripAdvisor.
Movie based queries should take me to Fandango and Rottentomatoes, concert tickets to TicketMaster and event tickets to EventBrite.
Job related queries can take me to LinkedIn or Glassdoor.
In short, Siri should link all these Web 2.0 companies (that all have an API) together. Its the ultimate content aggregator.

update:
I think it would be better if you said the name of the service that you want Siri to pull from if there’s multiple sources.
So for example, if I were to say “Java, Wikipedia” – then Siri would pull from Wikipedia. But “Java, Google” would just do a google search,
and “Java, Stack Overflow” would search stack overflow for the keyword. This would make it easy and intuitive for Siri to bring up the correct response.

Integration with web and OS X
How about having Siri come to Macs and PCs? How about having Siri be accessible as a web application running on the cloud? Almost all computers come with microphones, so why not? I see the advantage of making Siri exclusive to iOS but soon I think Apple should make it widely available.

Text input
The main problem with me using Siri is that it looks awkward when I’m talking to it in a public place, or using it at a meeting can be distracting. So Siri should take text input to fix that. I’m sure some Java parser in the backend is translating natural speech into strings anyway, so it shouldn’t be too hard to make a text based option.

Expanding services
Siri should be able to integrate with countless Web 2.0 services. And it should be locale specific. If I’m in China, then Siri should integrate with Baidu, QQ, Sina, etc. If I’m looking for a deal, then Siri can pull up Groupon. Point is, Siri can be the glue that ties all these services together. And maybe pulling up Youtube when I’m looking for a funny cat video or something. Perhaps Siri can learn from users in general, in context, and know what to say in response depending on what other users have said to it (a la Cleverbot).

Social integration and Ad delivery
Siri now provides Apple with the same data as Facebook. It can learn about a user’s behavior, their location, their preferences, their hobbies, etc. Anything about a user can be learnt through Siri, so now Apple can do anything and everything that Facebook can do, including delivering targeted ads. If Siri knows you’ve been going to mexican restaurants lately, maybe it can show you an ad from another mexican restaurant nearby. Or maybe suggest you to buy Star Wars from iTunes or Netflix if you’ve been watching sci-fi movies lately. Or maybe show some ads from ESPN if you’ve been watching or asking about sports alot. Or direct me to the iTunes store for Usher’s new album or concert if I’ve been listening to a lot of his songs. Since Siri knows everything about you, it can be the ultimate ad and content delivery vessel.

Better Multilingual support and Translation
Siri will integrate with over 15 languages with iOS 6, so I would like to see quality recognition for different dialects and accents, particularly for Asian languages. And this would probably require a lot of native speakers and language specialists, but I think if Siri can do this well, then it could also be the best translator (by storing speech patterns and tonal variance). We all know Google Translate sucks for Asian languages, so I’m sure Apple can do it better!

Now that Siri is multilingual and will integrate with car manufacturers soon (potentially replacing navigation), I see lots of potential with it in the future. It has the possibility of overshadowing Google or Facebook, since it both searches by voice, and retains user information. So these are just some of my ideas for future improvements from Siri, but where it goes remains to be seen.

My company, Switchfly have events called FedEx days, which basically means we set aside 24 hours for employees to work on whatever project they wanted given that it provides some business value to our application.

Fedex Day 1:
So, I joined with a few other engineers and business people to create Facebook sharing and liking for hotels and itineraries.
So, after 24 hours of toiling, we had a few mockups to present.

Facebook Like for Hotels
Facebook’s Like feature relies heavily on the Graph API, and the most difficult part of implementing this is due to FB having to have a public, static, unique URL for every page that you want to link Like with. Since our hotels are generated with a random key, this proved quite difficult, and in addition, FB like requires heavy use of metadata tags, which have to be inserted into the body. The problem is that the info required for the metadata, such as the Hotel name, Hotel description, is available only after the page renders, and inserting meta tags dynamically into the head of the page breaks the Like feature. So, using PHP or Coldfusion, the page has to grab info from the shared database at page render time, so that the meta data exist in the source, and publicly pingable by FB. So our mockup looks like this:

Liking hotels using FB API

Meta data on the page looks like this:

<meta content="Sheraton Vistana Resort" /><meta content="hotel" /><meta content="https://airbnb.com/show_hotel_details.php?page=sheraton_vistana_resort" /><meta content="The all villa resort in Lake Buena Vista rests on 135 acres of waterfalls fountains and tropical landscaping. Each one and two-bedroom spacious vacation villa boasts a fully equipped kitchen with appliances amenities and all accommodation necessities. The resort does not charge additional person or resort fees. Guests may enjoy free parking wireless internet access and daily scheduled transportation to all four Walt Disney World® Theme Parks. Local attractions include SeaWorld Walt Disney World® Resort Universal Orlando® Resort and the Orange County Convention Center. The airport is only 13 miles away.The resort features 7 heated outdoor swimming pools 5 kids' pools 13 tennis courts 3 fitness centers spa services basketball and volleyball courts and its own Market Place where you may find souvenirs groceries Boars Head Deli and even Starbucks Coffee. Additional amenities include an 18-hole miniature golf course room service video arcades 6 restaurants including the KFC Express A&amp;W Express and Pizza Hut Express at the newly built Fountains Food Court and two bars.FL Resident Rates--Proof of residency will be required at check in. Failing to provide proper Identification at check-in will default reservation to best available rate of the day." /><meta content="8800 Vistana Centre Dr" /><meta content="Orlando" /><meta content="FL" /><meta content="32821" /><meta content="USA" /><meta content="http://d2whcypojkzby.cloudfront.net/imageRepo/1/0/24/953/468/30783_F.jpg" /><meta content="Switchfly" />

This will share this hotel or send your friends a message telling them what hotel you are looking at.

Facebook Share your Trip
So, we also have FB sharing your trip after you make a booking. This is done via simple facebook API functions, and currently sends a link on FB that leads to the same hotel/flight search that you did, but if we were to expand it, it could look something like this:

Facebook Trip Sharing

Basically, we want to be able to make custom objects and custom actions using Graph API to post information about your itinerary to your friends. You can see what friends are on your flight, and change your flight plan, book a hotel, change a reservation, or rent a car using the post. Anyways, for 24 hours, building these two proof of concepts have been pretty fun, and I someday hope to implement these features in full.

Fedex Day 2:
For Fedex day 2, we decided to make Passbook integration for our itineraries. I was the only engineer on this project this time around (and again toiled for 24 hours).

Passbook Generation
So, Passbook is an Apple application, and in order to integrate with Passbook, you need to be an Apple dev with provisioning to generate a Passbook id for integration. This part was kind of confusing because of the way how certificates and certificate requests were set up. But after looking at the Passbook documentation, I managed to generate some sample passes for our app.

Amex Switchfly Air Passbook

Amex Switchfly Hotel Passbook

Integrating with the application

Passbook and SMS Integration

Integration with the application was more difficult. Firstly, the SMS service I wrote (to send users links to their booking confirmation) integrated well with Twilio, and that API was fairly straightforward to set up (Kudos to Twilio), the limiting factor I found being the 160 char limit on SMS texts and the trial account. Secondly, the passbook service was more difficult. Basically, I had to capture the information on the page into json, and ajax it over to our coldfusion/sitebricks endpoint so that we can make use of it. Unfortunately the signing of the pass required a cryptographic algorithm which Apple doesn’t help you with. I had to use Jpasskit, a third party library to help me with that. Even then, the main issues I ran into were:

1) Passbook only contains limited space on the front. Doesn’t even have enough room to put a roundtrip flight, let alone flight + hotel.
2) Each passbook needs to be signed by a certificate which has a keystore and a password from somewhere on the server.
3) Our context and encoding filters prevent .pkpass from being distributed.
4) Even if it was working, only Mac OS and iOS users would be able to make use of it (via email or Safari).
5) Not sure how I could get the SSL Handshake with Apple’s restrictions working.

So because of these issues, not sure if this will make it to prod, but it was a good learning experience nonetheless.

Blue is Me

Tong Zou's Work and Ramblings

Tag Archives: api