Category: Uncategorized

It's time!  The clock will start on Dec 1st, which is this Sunday!  I'll take the snapshot at midnight on Dec 1st.  Here are the rules this year:

Back by popular demand AGAIN, the MSDN Forum Jam 2013 is ready to go!  What
is a forum jam?  It is an all out battle for question answering
supremacy in the Microsoft MSDN forums.  This is the third ForumJam I
have hosted.  You can see other Forum Jams here:

• https://blogs.architectingconnectedsystems.com/blogs/cjg/archive/2012/11/19/MSDN-Forum-Jam-2012.aspx
• https://blogs.architectingconnectedsystems.com/blogs/cjg/archive/2010/04/02/MSDN-Forum-Jam-2010.aspx

The rules for this Forum Jam will be the similar to the last one:

Tweet me at @givenscj or email me at chris@architectingconnectedsystems.com to say you would like to participate by November 30th, 2013 (MVPs and Non-MVPS welcome)

1. At 8am on Dec 1st, I will take a snapshot of all participants points, posts and answers
2. Contest
   will end January 31st (a full two months), the person with the following
   (as measured in difference between snapshot and 1/31 totals):
1. Highest Point total wins a $200 gift card
2. Second highest Point total wins a $75 gift card
3. Third highest Point total will win a set of ACS SharePoint 2013 training manuals
4. Fourth highest will get a copy of my SharePoint 2013 Inside Out book
   
3. Moderators are free to participate, but will be unofficially tracked
4. I'll post updates along the way to let people know where they stand
5. All
   post/answers/replies must be in the SharePoint forums.  I will audit
   these and any items that fall outside those forums will be deducted from
   your total!
6. As
   with anything, rules are always subject to change but probably won't,
   this is not endorsed by Microsoft and won't guarantee an MVP
7. Be sure to follow Forum etiquette, be nice and civil, and recognize others.  Its all about helpfulness!

If anyone else would like to contribute to the bounty let me know!

Let the community building begin!
Chris Givens
@givenscj

Ok, I decided to look at all the Apps in the SharePoint App store last week.  This started as a little harmless click on each app, download it, install it, play with it, rate it.  Very simple.  After about the first 20, I decided…this sucks.  So I said, there has to be a way to automate the install of all the Apps!  So, two days later…I successfully built a tool that will download all the apps, their details and their app cab file.  Everything gets recorded into an App details database and the files are saved to a configurable directory.  I will tell you, it is pretty sweet to look at an app content page that has all the apps loaded on it without doing anything.

I learned a *ton* as I went down this little rabbit hole.  The first thing I learned, that you won't find anywhere else, is that each farm will initialize itself with a file from the office.microsoft.com servers.  You can find this initialization xml here:

https://office.microsoft.com/Config15

The XML file defines all the end points that the Office Apps layer will use when performing various tasks.  As you can see, the file has a lot of endpoints.

When your goal is to download and install an app, many of the steps behind the scenes are abstracted away from you.  There are many steps involved:

1. Open your site, browse to the "Add an app" link
2. Browse to the "SharePoint Store".  In reality, this is a page hosted on your sharepoint server that is simply making JSON calls to the office servers
3. The page will request the list of all Apps using (http://spservername/_layouts/15/storefront.aspx?task=GetApps&bm=US&cm=en%2DUS&category=&free=0&sort=7&query=&catalog=0).  This JSON is the list of all apps.  There are currently 240 of them in the US marketplace (at the bottom of this post is a CSV file with all the app details).  My code knows how to pull this information and save to a database, so I have a full list of all apps and all of their details.  This was helpful in my review process so that I didn't need to create some massive spreadsheet with all the details typed in!
4. Clicking on an app will pull down more specific details about the app using this url (http://spservername/_layouts/15/storefront.aspx?task=GetAppDetails&bm=US&cm=en%2DUS&appid=APPID&catalog=0").  This app detail gives you the information like if it is a paid app, whether it includes a trial license capability, etc. 
5. Once you are on a page that allows you to "Add" an app, you click the "Add It" button.  What happens here is a post back occurs on your SP Server and a redirect is generated to the office.microsoft.com marketplace.
6. When you arrive at the marketplace, cookies will be interrogated to see if you have logged in via LiveID.  If you have not, you are redirected to the LiveID servers for auth
7. Once authenticated, you are returned to the marketplace using a special cookie identifier to rebuild your session
8. You are then prompted to confirm that you really want the app, upon confirmation, you are redirected back to your sharepoint site via a HTTP GET with all the necessary license details in the querystring
9. Your SharePoint server will take these details and do a validation post behind the scenes to confirm that the app and license match – there is no tricking it here
10. Once the license is validated, it will save the information to the AppManagement database tied to your App Management Service Application
11. It will then intitiate a download of the app's cab file from the office servers using this url (https://office.microsoft.com/client/15/appinstallinfo/query?cmu=" + contentMarket + "&clientToken=" + rawXMLEntitlementToken + "&ret=0&av=" + av + "&build=" + build + "&origin=" + origin + "&corr=" + corr;).  this url contains your clientToken which is the validation that you have purchased the app from the marketplace.
12. This page will redirect you to the download page and in the body of the response will be the app cab file.
13. Once downloaded, the file is installed into the target
14. In a separate step you must trust the application. I was eventually able to automate this part. I attempted to do it via CSOM, but I get the dreaded "Sideloading of apps is not enabled on this site" error.  Bummer, no way to get an appinstance or productid from CSOM.  

So all those steps!  I didn't want to do that for every App (if you were to do this manually, it is going to take you a full day or two!), sooo…I built a tool for it.  By the way, it takes almost an hour to download and install everything!  This tools has the following features:

• Downloads all app details from the App Store into a database (this is easy via one JSON request)
• Allows you to Install an App from the App Store (you can pass it a list of the ones you want it to activate) – this is handy for building an environment that matches your production environment.  Or possibly rebuilding another environment
• Downloads all App CAB files for all installed apps
• Allows you to activate a list of apps on a web of your choosing (this would be after you have installed them)
• Tool Side-Effect – This tool allows you to bypass the validation checking of the feature requirements on the App Detail page.  In otherwards, you can purchase free apps and download their CAB file, even if you can't use it because it is O365 targeted
• Both free and trial apps are supported, did not put in support for selecting a payment method to buy an app that didn't have a trial…that would be pretty dumb!
• Supports on-prem and O365

I did try to be sneaky and hack the purchase API. It passes a special query string with the price and purchase model.  I tried to make everything a free app, but it was too smart for that.  They got lucky…

I have some clean up to do on the tool to make it something I would give out to the masses but it does work and works both for on-prem and O365. However, I'm also thinking if I do publish it, I may make it a product. Think…iOS app automated updater type of thing.  There isn't a UI that says, "Update All" in the SharePoint store interface.

Some things that suck about the Apps APIs:

• AppId – the AppId is used everywhere (on the store, in the purchase api), except in SharePoint (well it is at least saved in the databse)!  When a basic user wants to add or remove an app, it is not the AppId that you use, it is an AppInstanceId, or the completely different ProductId passed in the license.  Consistency is key here and this needs changed, see my next items…
• For automating the "Trust It" part of the process – You can't get the product id to pass into the appdownload function when you want an app to be installed in your web. You have to query the app management database *directly* to get the product id.  I think this is stupid.  You should just be able to pass the AssetId to get the app to install in the web.  This is probably one of the reasons there are no CSOM methods to install an already existing app in a web (without side-loading it).  This needs fixed!
• Adding an app – each time you add an app to a web, the content database that contains that web will download a copy of the app (from the office app store on the internet) into the database.  There is no central repository for these apps by default.  The AppCatalog site is needed for this, but I don't think it is downloaded into the catalog and I'm pretty sure it will pull it from the catalog into the local database anyway, this is a waste of space.
• Uninstalling apps – there is no easy way to get all the AppInstanceIdsproductIds.  I had to query the database directly to get them to uninstall all apps after I was done testing.  The APIs do not expose this so its a manual process without my tool.
• There is a database leak in the AppPrincipals table.  Each time you add and remove an app, a new entry is added, but not removed on removal.  not a big deal, but could cause problems later and needs to be fixed.
• App name versus App name versus app part name – this was weird, there are cases where these are all different. Makes it confusing when other people have similarly named appsapp parts.  Microsoft needs some governance here.  Also wondering what happens when you upload an app part with the same name as someone else?  Bet its not good….#badhacker
• When you have so many App parts in the UI, you really need a search feature!  Clicking the next arrow 10 times to get to the end just isn't cool.
  
• Apps that are deployed to Azure require O365 and don't work on-prem, you need special OAuth tokens to pass to the remote Azure app
  

So how do I rate all these Apps?  And what is some common things I see as issues with the App Model?  It's pretty obvious that no one has ever posted a "here's how you write an App the proper way" type of document.  Lots of technical "how to's", but no common sense "must dos".  Well, here you go on my observations:

• Business – Come on Man!  Those of you that put an app out there and are charging for it when someone put one out there for free are just stupid.  Do an analysis before you put something in the app store.  Don't waste your time or other people's time!  It's called a SWOT analysis by the way for you non-MBA types.
• Complexity – There are very few sophisticated apps in the marketplace.  Lots of simple ones, but the ones that did put in effort, really show.
• Feature Enablement – Its unfortunate that the SP Store page doesn't tell you what you need as far as feature pre-reqs to install some apps.  It just says "You can't install that app".  In the app details (which are hidden on the page and in the json), it will tell you what features are needed that you must enable (for instance, project server feature must be installed for some apps).  It would be nice if it gave you the option to enable the missing feature directly on the page
• Navigation – If you have an Azure hosted app, put the damn navigation at the top to get back to my site!
• Scalability – Most apps use an App list for data storage.  I can see these lists filling up with many many items over a long period of time.  This will be problematic.
• Ribbon – higher quality apps integrate with the ribbon in some manner.  you should think how your app will do this
• App Testing – It was painful to add a new page for every app I wanted to test and I can see trying to add many "cool" app parts to the same page can clutter the page
• Preview features – Please don't make us go to a page with your app part on it.  You should at least give us the courtesy of a preview tab!
• Centralized data storage – The biggest issue I see with apps is there is no sandbox area that is shared storage.  It is completely unacceptable to have the same app deployed on multiple sites with each one getting its own list
• Trials – People that publish an app without a trial and have a price on it…good luck with that.  I'm not buying your App and no one else should either, don't buy an app unless you can trial it first

Before we get into my rating, some interesting "awards" for the apps that I put together:

• Best overall App – this was tough, but in the end, Nintex has had that up there for a while and it is really sweet design
• Most Expensive App – Booking of Resources – really…WTF…$899.99 for that piece of #$^&$?
  
• Most over-prices apps – SONJAsAPPS
• Most Apps of same type – it was like a tie between image carousel apps and map apps…how many of these things do we need people?!?  Come on man!
• Most Fun App –
  Cheezburgers
• Most Creative App – Colour Lab
  
• Highest Quality Apps – Layer2
• Lowest Quality Apps – Point8020
• Dumbest App – Spallery
• Most Complex – Kodak Imaging App
• Most Publisher Apps – Microsoft with 22, SONJAsAPPS with 17 but they don't count, so SR1 Development Limited get the award with 11
• Notable Mention – Avior GEO Location Utility
  

And finally, my CJG rating of all apps in the Office marketplace (at least the ones that are free and with trial, 30 of the 240 do not have a trial, those are the lame ones):

My good friend Fabian tweeted about a problem he was having with SharePoint 2013 workflows. 

http://fabiangwilliams.wordpress.com/2013/09/06/help-unable-to-create-list-using-sharepoint-2013-rest-api-in-spd2013/

It was an interesting problem that I have never seen anyone tackle before so the fact he was having a error wasn't surprising.  It did however peak my interest about the problem, so I asked him to send me his workflow. 

He exported his workflow as a template (solution file) and sent it over.  First issue I had was importing the workflow as a sandboxed or farm solution.  It would give me this crazy error:

svr-spwfe01 : Failed to load receiver assembly "Microsoft.SharePoint.WorkflowServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" for feature "GetItemsFromSPOListInstances" (ID: eb5838ef-cda9-42b2-9ea8-624f060d554c).: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SharePoint.WorkflowServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.SharePoint.WorkflowServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& 

So I abandoned that idea and looked to import the XAML file like I did in 2010.  I created a 2013 workflow but alas, no xaml files are to be found in the "All Files" view.  Where do they go?  Well, there is a hidden list called "wfsvc".  It has no views on it so simply unhiding it won't allow you to browse the files inside it.  Attempting to create a view is also futile.  So I turned to Windows PowerShell, navigated to the SPFile that represented the workflow xaml file and replaced it with Fabian's.  Refresh of SharePoint Designer and bingo…I had all his info.  Sweet.

The main issue of Fabian's was that when he made the Web Service call he would pass a pre-created FedAuth token (yeah I told him he would have to figure out how to generate this in the future) which established a context for him.  This context was as his SharePoint admin account, but the actual running account (the "actor") was the Workflow service.  Of course the SP Admin account can do whatever it wants, but the workflow service cannot.  This workflow service account is NOT the account that the actual workflow manager backend window service is running under.  It is the Application principal account that is created for the workflow service.  If you browse to the database, you can see there are two database tables:

• AppPrincipalPerms
• AppPrincipals

If you browse the AppPrincipals table, you will find one or more principles.  Consider these to simply be user just like any other user, yet they are applications and as such, do not have a password.  You can find the ID of the AppPrincipal for workflow by taking the second index of the NameID for the item with title "Workflow" (NOTE:  This id is different in EVERY farm, which is a bummer):

•  i:0i.t|ms.sp.ext|78cf7810-a2c9-497e-bffe-8c25368c8bc0@fcd171b6-20f6-443e-8aea-474a4b529e6e

This is the identity that will run the workflow.  If you review the permission mask in the AppPrinicpalPerms table for this appprincipal, you will see that it is set to "3" by default.  The enumerations for the masks are:

• None
• Guest
• Read
• Write
• Manage
• FullControl

"3" means "Write".  Which allows the workflow service to write out list items, but not to create lists.  This would require level 4 or higher.  You can elevate the permission mask with the following Windows PowerShell:

$web = get-spweb http://teams.contoso.com
$appworkflow = Get-SPAppPrincipal -site http://teams.contoso.com -nameidentifier 78cf7810-a2c9-497e-bffe-8c25368c8bc0@fcd171b6-20f6-443e-8aea-474a4b529e6e
Set-SPAppPrincipalPermission -AppPrincipal $appworkflow -site $web -right fullcontrol -scope site

Your workflow service app principal can now create lists!

Fabian will be researching how to add these app permissions in O365, of which he did here:

http://fabiangwilliams.wordpress.com/2013/09/08/actually-resolved-unable-to-create-list-using-sharepoint-2013-rest-api-in-spd2013/

Enjoy!
Chris