App Indexing & The New Frontier Of SEO: Google Search + Deep Linking

In this article, you’ll learn how Google is surfacing deep app content and how SEOs can prepare iOS and Android deep app screens for Google’s index. Google is making significant moves to close the gap between app and Web content to make mobile interaction more seamless, and that theme will reappear throughout the analysis.

This is the second installment in a three-part series about app indexing strategies and deep linking opportunities. The first article focused on Apple’s new Search API for iOS 9, which encourages and incentivizes an app-centric mobile experience.

Today’s column, co-authored with Cindy Krum, will focus on how Google indexes deep app screens and what marketers can do to promote their app content in Google search. Google’s app indexing strategies differ significantly from Apple’s, and it’s important for marketers to understand the distinctions.

The third article in this series will focus on future app indexing challenges we will face with the growth of wearables and other non-standard device apps and device indexes.

App Indexing In Google

Historically, app landing pages on websites have been in the Google index — but actual apps and internal app screens have not. Because crawling and indexing in-app content was impossible until recently, users had to discover new apps via an app store (Google Play or iTunes), which surfaces apps according to app meta data and editorial groupings instead of in-app content. For digital marketers, internal app content has been unavailable for search — part of what Marshall Simmonds calls “dark search.”

This situation has created a two-fold problem for Google:

1. App stores had trained users away from using Google for app discovery; and
2. App developers were historically not incentivized to optimize internal app data for search. This limited Google’s mission to collect and organize the world’s data, which in turn limited its ability to make money.

Now that Google is indexing both app landing pages and deep screens in apps, Google’s app rankings fall into two basic categories, App Packs and App Deep Links. App Packs are much more like the app search results that SEOs are used to, because they link to app download pages in Google Play or the App Store, depending on the device that you are searching from. (App Packs will only show apps that are compatible with your device’s OS.)

Ranking in an App Pack (and also in the Apps Universal, under Google’s top-navigation drop-down in the mobile search results) relies heavily on the app title, description, star ratings and reviews, and it will differ greatly from the internal app store rankings, as well as in-app indexing strategies described in the rest of this article.

Deep links are different because they link to specific deep screens within an app. Google has displayed deep links in search results in a variety of ways since it started app indexing, but there are a couple of standard deep link displays (shown below) that seem more common than others. Some deep-linked results look no different from traditional blue links for websites, while other deep link search results contain more attractive visual elements like colored “install” buttons, app icons and star ratings.

[click to enlarge]

We believe that the most common deep link in the future will display the app icon and a small “open on domain.com” button because that allows users to choose between the deep app link and the Web link without an additional dialogue screen. (Currently, the dialogue screen from other types of deep links comes from the bottom of the browser window and says, “Would you like to open this in Chrome or in the [Brand Name] app?”)

It is important to note that aspects of the search context, like the mobile browser, can limit the visibility of deep links. For example, Google only supports app indexing on iOS inside the Google and Chrome apps, not in Mobile Safari, the default Web browser on iOS. It seems likely that Safari will be updated to allow for Google’s deep linking behaviors as part of the iOS 9 update, but it is not confirmed.

Similarly, Google has been experimenting with a “Basic” mobile search results view that omits rich content for searchers with slow carrier connections. “Basic” search results do not include App Packs at all (since downloading an app would not be attractive to people with slow connections), and deep link results will only show as inline blue links, without images, star ratings, icons or buttons.

These are important stipulations to keep in mind as we allocate time and budget to optimizing app indexing, but the benefits of Google app indexing are not limited to surfacing deep app screens in Google search results.

Why Is App Indexing Important For SEO?

Without apps in its index, Google was missing a huge piece of the world’s data. The new ability to index iOS and Android apps has fundamentally changed app discovery and dramatically changed mobile SEO strategies.

Now that Google’s search engine can process and surface deep app content in a similar fashion to the way it does Web content, Google search has a significant advantage over the app stores. It is still the #1 Search Engine in the world, so it can easily expose content to more potential customers than any app store could, but it can also integrate this new app content with other Google properties like Google Now, Inbox/Gmail and Google Maps.

This change has also added a whole new host of competitors to the mobile search result pages. Now, not only can app landing pages rank, but internal app screens can also compete for the same rankings.

Google’s official position at the moment is that Web parity is necessary for deep app indexing (i.e., crawlable Web content that matched the indexable app content), but at Google I/O, the company clarified that they are working on a non-parity app indexing solution. They have even started promoting an “app only interest form,” and recent live testing has reinforced the idea that apps without parity will soon be added to the index (if they haven’t been already).

This is a big deal, so SEOs should be wary of underestimating the potential market implications of Google indexing apps without Web parity. For marketers and SEOs, it means that mobile search results could soon be flooded with new and attractive competition on a massive scale — content that they never have had to compete with before.

Let’s do a bit of math to really understand the implications.

We’ll start with a broad assumption that there are roughly 24,000 travel apps, a third of which lack Web parity. If each app contains an average of just 1,000 screens (and travel apps often include many more than that), we’re looking at roughly 8,000,000 new search results with which travel websites must compete — and that’s in the travel industry alone. That is huge!

Games, the biggest app category in both stores, promises to create an even bigger disruption in mobile search results, as it is a category that has a very high instance of apps without Web parity.

Another subtle indication of the importance of app indexing is the name change from “Google Webmaster Tools” to “Google Search Console.” Historically, webmasters and SEOs have used Google Webmaster Tools to manage and submit website URLs to Google’s index. We believe the renamed Google Search Console will eventually do the same things for both Web and apps (and possibly absorb the Google Play Console, where Android apps have been managed). In light of that, removing the “Web” reference from the old “Webmaster Tools” name makes a lot of sense.

A similar sentiment by John Mueller, from Google, is noted below, and possibly hints at the larger plan:

How Does Google Rank Deep Links?

Like everything else, Google has an algorithm to determine how an indexed deep link should rank in search results. As usual, much about Google’s ranking algorithm is unknown, but we’ve pieced together some of the signals they have announced and inferred a few others. Here’s what we currently believe to be true about how Google is ranking deep links in Google Search:

Known Positive Ranking Factors

• Installation Status. Android apps are more prominently featured in Google search results when they are installed on a user’s device or have been in the past. Rather than checking the device, Google keeps track of app downloads in their cloud-based user history, so this only affects searchers when they are signed into Google.
• Proper Technical Implementation. The best way app publishers can drive rankings, according to Mariya Moeva of Google, is to “ensure that the technical implementation of App Indexing is correct and that your content is worth it.” She later elaborated in a YouTube video, explaining that app screens with technical implementation errors will not be indexed at all. (So start befriending the app development team!)
• Website Signals (title tags, description tags). Traditional SEO elements in the <head> tag of the associated Web page will display in deep link search results, and thus are also likely ranking factors for the deep links. In fact, good SEO on corresponding Web pages is critical, since Google considers the desktop Web version of the page as the canonical indexing of the content.

Known Negative Ranking Factors

• Content Mismatch. Google will not index app screens that claim to correspond with a Web page but don’t provide enough of the same information. Google will report these “mismatch errors” in Google Search Console, so you can determine which screens need to be better aligned with their corresponding Web pages.
• Interstitials. Interstitials are JavaScript banners that appear over the content of a website, similar to pop-ups but without generating a new browser window. The same experience can be included in apps (most often for advertisements), but this has been discouraged by both Apple and Google. In her recent Q&A with Stone Temple Consulting, Mariya Moeva implied that app interstitials are a negative ranking factor for deep links (and said to stay tuned for more information soon). Interstitials can also prevent Google from matching your app screen content to your Web page content, which could cause “Content Mismatch Errors” that prevent Google from indexing the app screen entirely. In either case, app and Web developers should stay away from interstitials and instead, opt for banners that just move content down on the screen. Both Apple and Google have endorsed their own form of app install banners and even offer app banner code templates that can be used to promote a particular app from the corresponding mobile website.

Apart from ranking on their own, app deep links can also provide an SEO benefit for websites. Google has said that indexed app deep links are a positive ranking factor for their associated Web pages, and preliminary studies have shown that Web pages can expect an average site-wide lift of .29 positions when deep link markup is in place.

Also, App Packs and App Carousels tend to float to the top of a mobile SERP (likely ranking as a group rather than ranking independently). Presence in these results increases exposure and eliminates a position that a competitor could occupy lower down in the organic rankings, since these “Packs” and “Carousels” take up spaces that would be previously held by websites.

Indexed Android apps will also get added exposure in the next release of the Android operating system, Android M. It includes a feature called “Now on Tap,” which represents a deeper integration of Google Now with the rest of the Android phone functionality. Android M allows Google to scan text on an Android user’s screen while in any app, then interpret a “context” from the on-screen text, infer potential queries and automatically display mobile applications that could assist the user with those inferred queries.

For example, a WhatsApp conversation about dinner plans could pull up a “Now on Tap” interface that suggests deep links to specific screens in OpenTable, Google Maps and Yelp. This only works for deep-linked app screens in Google’s index, but for those apps, it will likely drive significantly higher engagement and potentially more installs. From a strategic perspective, this adds another potential location to surface your content, beyond the mobile search results.

While Google will only surface apps they have indexed, they plan on crawling on-screen text in all apps, trying to perceive context for “Now on Tap.” Google doesn’t provide any opt-in mechanism, so Android apps that are not indexed for Google search can still be crawled to trigger a “Now on Tap” experience. This means that Google is essentially reserving the right to send users away from your app to a different app that has relevant screens in the index, but also that Google is allowing your app to “steal” users away from other apps if your app screens are in the index.

This could provide nearly limitless opportunities for “Now on Tap” to suggest apps to Android users, and the “rogue crawling” aspect of it reinforces our prediction that Google will soon be crawling, indexing and surfacing app screens that don’t have Web parity. This will make Google’s app indexing an even more important strategy for Android apps, especially once Android M is widely adopted.

The app rankings advantage is pushed to the next level when you understand that Google is intentionally giving preference to app results for certain queries. In some cases, being an indexed app may be the only way to rank at the top in mobile Google search. Keywords like “games” and “editor” are a common trigger for App Packs and App Carousels, but Google is also prominently surfacing apps for queries that seem to be associated with utilities or verbs (e.g., “flight tracker,” “restaurant finder,” or “watch tv”). And when the App Packs or Carousels appear, they often push the blue links below the fold (and sometimes way below the fold).

At the end of the day, for some queries, a blue link may not ever beat the “Packs” — in which case, the best strategy may be to focus on App Pack listings over deep links.

How Can I Get Deep App Screens Indexed For Google Search?

Setting up app indexing for Android and iOS Apps is pretty straightforward and well-documented by Google. Conceptually, it is a three-part process:

1. Enable your app to handle deep links.
2. Add code to your corresponding Web pages that references deep links.
3. Optimize for private indexing.

These steps can be taken out of order if the app is still in development, but the second step is crucial; without it, your app will be set up with deep links but will not be set up for Google indexing, so the deep links will not show up in Google Search.

NOTE: iOS app indexing is still in limited release with Google, so there is a special form submission and approval process even after you have added all the technical elements to your iOS app. That being said, the technical implementations take some time. By the time your company has finished, Google may have opened up indexing to all iOS apps, and this cumbersome approval process may be a thing of the past.

Following are the steps for Google deep-link indexing. (For a PDF version of the instructions, click here.)

Step 1: Add Code To Your App That Establishes The Deep Links

A. Pick A URL Scheme To Use When Referencing Deep Screens In Your App

App URL schemes are simply a systematic way to reference the deep linked screens within an app, much like a Web URL references a specific page on a website.

In iOS, developers are currently limited to using Custom URL Schemes, which are formatted in a way that is more natural for app design but different from Web.

In Android, you can choose from either HTTP URL schemes (which look almost exactly like Web URLs) or Custom URL Schemes, or you can use both. If you have a choice and can only support one type of URL Scheme on Android, choose HTTP.

B. Support That App’s URL Schemes In The App

Since iOS and Android apps are built in different frameworks, different code must be added to the app to enable the deep link URL Schemes to work within the specific framework.

See PDF for clickable links.

C. Set Up CocoaPods

CocoaPods is a dependency management tool for iOS. It acts as a translation layer between iOS apps and the Google SDKs, so it is only necessary in iOS apps. Google has moved all its libraries to CocoaPods, and this will now be the only supported way to source them in an iOS app.

See PDF for clickable links.

NOTE: Developers who have never worked with CocoaPods may have to rework how they currently handle all dependent libraries in the app, because once CocoaPods is installed, it is harder and more complicated to handle other non-CocoaPods libraries. There are some iOS developers who favor CocoaPods and have been using them for some time, so your app may already be working with CocoaPods. If that’s true, prepping for iOS app indexing will be much easier.

D. Enable The Back Bar

iOS devices don’t come equipped with a hardware or persistent software “back” button, so Apple and Google have built workarounds to make inter-app back navigation easier. Google requires that iOS apps recognize an additional GSD Custom URL Scheme (that was set up in Step 1B). Google only uses this to trigger a “back” bar in the iOS app.

Google will generate the GSD Custom URLs automatically when someone clicks on an iOS deep link from a search result page, so we don’t need to generate new GSD deep links for every screen; we just need to support the format in the Info.plist file and add code that will communicate with the “GoogleAppIndexing” Pod when a GSD link is received by the app.

NOTE: Google’s solution is similar to Apple’s iOS 9 “Back to Search” buttons that display in the upper left portion of the phone’s Status Bar, but when it is triggered, it appears as a blue “Back Bar” that hovers over the entire phone Status Bar. The Back Bar will disappear after a short period of time if the user does not tap on it. This “disappearing” behavior also represents a unique experience for iOS deep linking in Google, since after a certain period of time, there won’t be a way for iOS users to get back to the Google Search results without switching apps manually, by clicking through the home screen. Developers compensate by adopting more tactics that pull users deeper into the app, eat up time, and distract the user from going back to Google Search until the bar disappears.

E. Set Up Robots & Google Play/Google Search Consoles

In some cases, it may make sense to generate deep links for an app screen but prevent it from showing up in search results. In Android, Google allows us to provide instructions about which screens we would like indexed for search and which we would not, but no similar mechanism is available for iOS.

Digital marketers and SEOs should use the Google Play Console and the Google Search Console to help connect your app to your website and manage app indexation. Also, double check that your website’s robots.txt file allows access to Googlebot, since it will be looking for the Web aspect of the deep links in its normal crawls.

Step 2: Add Code To Your Website That References The URL Schemes You Set Up In The App

A. Format & Validate Web Deep Links For The Appropriate App Store

Google’s current app indexing process relies on Googlebot to discover and index deep links from a website crawl. Code must be added to each Web page that references a corresponding app screen.

When marking up your website, a special deep link format must be used to encode the app screen URL, along with all of the other information Google needs to open a deep link in your app. The required formatting varies slightly for Android and iOS apps and is slightly different from the URL Schemes used in the app code, but they do have some elements in common.

The {scheme} part of the link always refers to the URL scheme set up in your app in Step 1, and the {host_path} is the part of the deep link that identifies the specific app screen being referenced, like the tail of a URL. Other elements vary, as shown below:

See PDF for clickable links.

B. Add Web Deep Links To Web Pages With Corresponding App Screens

Internal app screens can be indexed when Googlebot finds deep app links in any of the following locations on your website:

• In a rel=”alternate” in the HTML <head>
• In a rel=”alternate” in the XML sitemap
• In Schema.org ViewAction markup

Sample code formatting for each of those indexing options is included below:

Step 3: Optimize For Private Indexing

Both Google and Apple have a “Private” indexing feature that allows individual user behaviors to be associated with specific screens in an app. App activity that is specific to one user can be indexed on that users’ phone, for private consumption only (e.g., a WhatsApp message you’ve viewed or an email you’ve opened in Mailbox).

Activities that are Privately indexed do not generate deep links that can surface in a public Google search result, but instead generate deep links that surface in other search contexts. For Android apps, this is in Chrome’s autocomplete and Google Now; for iOS, this is in Spotlight, Siri, or Safari’s Spotlight Suggest results.

See PDF for clickable links.

NOTE: Google’s documentation seems to indicate that Activities are only used for private indexing, but Google may also use them as a measurement of engagement for more global evaluations of an app (as Apple does with NSUserActivities in Apple Search). Google has not highlighted their private indexing feature as vocally as Apple, and a user’s private index can be accessed from the Phone icon in the bottom navigation of the Google Now app on Android and iOS. Currently, only Google’s apps (like Gmail) are able to surface privately indexed content in organic Google search results, but we suspect this will be opened up to third-party apps in the future.

Concluding Remarks

App indexing and deep linking are changing the digital marketing landscape and dramatically altering the makeup of organic mobile search results. They are emerging from the world of “dark search” and becoming a force to be reckoned with in SEO.

Marketers and SEOs can either look at these changes as a threat — another hurdle to overcome — or a new opportunity to get a leg up over the competition. Those who wish to stay on the cutting edge of digital marketing will take heed and learn how to optimize non-HTML content like apps in all of the formats and locations where they surface.

That being said, relying on app deep links alone to drive Google search engine traffic is still not an option. Traditional SEO and mobile SEO are still hugely important for securing a presence in Google’s mobile searches. Google still considers desktop websites the ultimate canonical for keyword crawling and indexing, and the search engine relies heavily on website parity because its strength is still crawling and indexing Web content.

The next big app indexing questions are all about apps that lack Web parity. Google does not currently use a roaming app crawler to discover deep links themselves, but we feel confident that this will change. Google’s App Indexing API currently only helps surface Android apps in autocomplete, but we believe in the future, it will help surface apps that don’t have Web parity.

Calling the system an “App Indexing API” seems to allude to a richer functionality than just adding app auto-complete functionality — and Google’s original app indexing documentation from April also indicated a more robust plan.

As shown in the diagram below, the original documentation explained that developers could use the App Indexing API (also referred to here as “Search Suggest,” which is different from the Search Suggest API) to notify Google of deep links “with or without corresponding Web pages.” That line has since been removed from the documentation, but the implication is clear: Google is paving the way for indexing apps without Web parity. Until that happens, traditional website optimization will remain a key component of optimizing app content for Google search, but when app screens can be indexed without Web parity, there will be a whole new set of ranking factors to consider and optimize for.

As we charge into this new frontier, the immediate benefits of app indexing are clear, but the newness may require a small leap of faith for more traditional marketers and SEOs.

Some may be left suspicious, with many questions: How long will Google provide a ranking benefit for deep-linked content? Will this be perceived as a “bait and switch,” like the Mobile Friendly update? Will app ranking factors evolve to include more traditional Web page ranking factors (like links and social signals)? Will Google begin to crawl app content more indiscriminately, using deep app links like Web links? Will Google develop a new app-specific crawler, or was the algorithm change on April 21 (aka “Mobilegeddon“) really this — that apps are already being crawled, rendered and evaluated by the smartphone crawler, just the same as Web?

Let us know what you think in the comments.

The post App Indexing & The New Frontier Of SEO: Google Search + Deep Linking appeared first on Search Engine Land.