Month: October 2013

Ok, I decided to look at all the Apps in the SharePoint App store last week.  This started as a little harmless click on each app, download it, install it, play with it, rate it.  Very simple.  After about the first 20, I decided…this sucks.  So I said, there has to be a way to automate the install of all the Apps!  So, two days later…I successfully built a tool that will download all the apps, their details and their app cab file.  Everything gets recorded into an App details database and the files are saved to a configurable directory.  I will tell you, it is pretty sweet to look at an app content page that has all the apps loaded on it without doing anything.

I learned a *ton* as I went down this little rabbit hole.  The first thing I learned, that you won't find anywhere else, is that each farm will initialize itself with a file from the office.microsoft.com servers.  You can find this initialization xml here:

https://office.microsoft.com/Config15

The XML file defines all the end points that the Office Apps layer will use when performing various tasks.  As you can see, the file has a lot of endpoints.

When your goal is to download and install an app, many of the steps behind the scenes are abstracted away from you.  There are many steps involved:

1. Open your site, browse to the "Add an app" link
2. Browse to the "SharePoint Store".  In reality, this is a page hosted on your sharepoint server that is simply making JSON calls to the office servers
3. The page will request the list of all Apps using (http://spservername/_layouts/15/storefront.aspx?task=GetApps&bm=US&cm=en%2DUS&category=&free=0&sort=7&query=&catalog=0).  This JSON is the list of all apps.  There are currently 240 of them in the US marketplace (at the bottom of this post is a CSV file with all the app details).  My code knows how to pull this information and save to a database, so I have a full list of all apps and all of their details.  This was helpful in my review process so that I didn't need to create some massive spreadsheet with all the details typed in!
4. Clicking on an app will pull down more specific details about the app using this url (http://spservername/_layouts/15/storefront.aspx?task=GetAppDetails&bm=US&cm=en%2DUS&appid=APPID&catalog=0").  This app detail gives you the information like if it is a paid app, whether it includes a trial license capability, etc. 
5. Once you are on a page that allows you to "Add" an app, you click the "Add It" button.  What happens here is a post back occurs on your SP Server and a redirect is generated to the office.microsoft.com marketplace.
6. When you arrive at the marketplace, cookies will be interrogated to see if you have logged in via LiveID.  If you have not, you are redirected to the LiveID servers for auth
7. Once authenticated, you are returned to the marketplace using a special cookie identifier to rebuild your session
8. You are then prompted to confirm that you really want the app, upon confirmation, you are redirected back to your sharepoint site via a HTTP GET with all the necessary license details in the querystring
9. Your SharePoint server will take these details and do a validation post behind the scenes to confirm that the app and license match – there is no tricking it here
10. Once the license is validated, it will save the information to the AppManagement database tied to your App Management Service Application
11. It will then intitiate a download of the app's cab file from the office servers using this url (https://office.microsoft.com/client/15/appinstallinfo/query?cmu=" + contentMarket + "&clientToken=" + rawXMLEntitlementToken + "&ret=0&av=" + av + "&build=" + build + "&origin=" + origin + "&corr=" + corr;).  this url contains your clientToken which is the validation that you have purchased the app from the marketplace.
12. This page will redirect you to the download page and in the body of the response will be the app cab file.
13. Once downloaded, the file is installed into the target
14. In a separate step you must trust the application. I was eventually able to automate this part. I attempted to do it via CSOM, but I get the dreaded "Sideloading of apps is not enabled on this site" error.  Bummer, no way to get an appinstance or productid from CSOM.  

So all those steps!  I didn't want to do that for every App (if you were to do this manually, it is going to take you a full day or two!), sooo…I built a tool for it.  By the way, it takes almost an hour to download and install everything!  This tools has the following features:

• Downloads all app details from the App Store into a database (this is easy via one JSON request)
• Allows you to Install an App from the App Store (you can pass it a list of the ones you want it to activate) – this is handy for building an environment that matches your production environment.  Or possibly rebuilding another environment
• Downloads all App CAB files for all installed apps
• Allows you to activate a list of apps on a web of your choosing (this would be after you have installed them)
• Tool Side-Effect – This tool allows you to bypass the validation checking of the feature requirements on the App Detail page.  In otherwards, you can purchase free apps and download their CAB file, even if you can't use it because it is O365 targeted
• Both free and trial apps are supported, did not put in support for selecting a payment method to buy an app that didn't have a trial…that would be pretty dumb!
• Supports on-prem and O365

I did try to be sneaky and hack the purchase API. It passes a special query string with the price and purchase model.  I tried to make everything a free app, but it was too smart for that.  They got lucky…

I have some clean up to do on the tool to make it something I would give out to the masses but it does work and works both for on-prem and O365. However, I'm also thinking if I do publish it, I may make it a product. Think…iOS app automated updater type of thing.  There isn't a UI that says, "Update All" in the SharePoint store interface.

Some things that suck about the Apps APIs:

• AppId – the AppId is used everywhere (on the store, in the purchase api), except in SharePoint (well it is at least saved in the databse)!  When a basic user wants to add or remove an app, it is not the AppId that you use, it is an AppInstanceId, or the completely different ProductId passed in the license.  Consistency is key here and this needs changed, see my next items…
• For automating the "Trust It" part of the process – You can't get the product id to pass into the appdownload function when you want an app to be installed in your web. You have to query the app management database *directly* to get the product id.  I think this is stupid.  You should just be able to pass the AssetId to get the app to install in the web.  This is probably one of the reasons there are no CSOM methods to install an already existing app in a web (without side-loading it).  This needs fixed!
• Adding an app – each time you add an app to a web, the content database that contains that web will download a copy of the app (from the office app store on the internet) into the database.  There is no central repository for these apps by default.  The AppCatalog site is needed for this, but I don't think it is downloaded into the catalog and I'm pretty sure it will pull it from the catalog into the local database anyway, this is a waste of space.
• Uninstalling apps – there is no easy way to get all the AppInstanceIdsproductIds.  I had to query the database directly to get them to uninstall all apps after I was done testing.  The APIs do not expose this so its a manual process without my tool.
• There is a database leak in the AppPrincipals table.  Each time you add and remove an app, a new entry is added, but not removed on removal.  not a big deal, but could cause problems later and needs to be fixed.
• App name versus App name versus app part name – this was weird, there are cases where these are all different. Makes it confusing when other people have similarly named appsapp parts.  Microsoft needs some governance here.  Also wondering what happens when you upload an app part with the same name as someone else?  Bet its not good….#badhacker
• When you have so many App parts in the UI, you really need a search feature!  Clicking the next arrow 10 times to get to the end just isn't cool.
  
• Apps that are deployed to Azure require O365 and don't work on-prem, you need special OAuth tokens to pass to the remote Azure app
  

So how do I rate all these Apps?  And what is some common things I see as issues with the App Model?  It's pretty obvious that no one has ever posted a "here's how you write an App the proper way" type of document.  Lots of technical "how to's", but no common sense "must dos".  Well, here you go on my observations:

• Business – Come on Man!  Those of you that put an app out there and are charging for it when someone put one out there for free are just stupid.  Do an analysis before you put something in the app store.  Don't waste your time or other people's time!  It's called a SWOT analysis by the way for you non-MBA types.
• Complexity – There are very few sophisticated apps in the marketplace.  Lots of simple ones, but the ones that did put in effort, really show.
• Feature Enablement – Its unfortunate that the SP Store page doesn't tell you what you need as far as feature pre-reqs to install some apps.  It just says "You can't install that app".  In the app details (which are hidden on the page and in the json), it will tell you what features are needed that you must enable (for instance, project server feature must be installed for some apps).  It would be nice if it gave you the option to enable the missing feature directly on the page
• Navigation – If you have an Azure hosted app, put the damn navigation at the top to get back to my site!
• Scalability – Most apps use an App list for data storage.  I can see these lists filling up with many many items over a long period of time.  This will be problematic.
• Ribbon – higher quality apps integrate with the ribbon in some manner.  you should think how your app will do this
• App Testing – It was painful to add a new page for every app I wanted to test and I can see trying to add many "cool" app parts to the same page can clutter the page
• Preview features – Please don't make us go to a page with your app part on it.  You should at least give us the courtesy of a preview tab!
• Centralized data storage – The biggest issue I see with apps is there is no sandbox area that is shared storage.  It is completely unacceptable to have the same app deployed on multiple sites with each one getting its own list
• Trials – People that publish an app without a trial and have a price on it…good luck with that.  I'm not buying your App and no one else should either, don't buy an app unless you can trial it first

Before we get into my rating, some interesting "awards" for the apps that I put together:

• Best overall App – this was tough, but in the end, Nintex has had that up there for a while and it is really sweet design
• Most Expensive App – Booking of Resources – really…WTF…$899.99 for that piece of #$^&$?
  
• Most over-prices apps – SONJAsAPPS
• Most Apps of same type – it was like a tie between image carousel apps and map apps…how many of these things do we need people?!?  Come on man!
• Most Fun App –
  Cheezburgers
• Most Creative App – Colour Lab
  
• Highest Quality Apps – Layer2
• Lowest Quality Apps – Point8020
• Dumbest App – Spallery
• Most Complex – Kodak Imaging App
• Most Publisher Apps – Microsoft with 22, SONJAsAPPS with 17 but they don't count, so SR1 Development Limited get the award with 11
• Notable Mention – Avior GEO Location Utility
  

And finally, my CJG rating of all apps in the Office marketplace (at least the ones that are free and with trial, 30 of the 240 do not have a trial, those are the lame ones):