Ok, I decided to look at all the Apps in the SharePoint App store last week.  This started as a little harmless click on each app, download it, install it, play with it, rate it.  Very simple.  After about the first 20, I decided…this sucks.  So I said, there has to be a way to automate the install of all the Apps!  So, two days later…I successfully built a tool that will download all the apps, their details and their app cab file.  Everything gets recorded into an App details database and the files are saved to a configurable directory.  I will tell you, it is pretty sweet to look at an app content page that has all the apps loaded on it without doing anything.

I learned a *ton* as I went down this little rabbit hole.  The first thing I learned, that you won't find anywhere else, is that each farm will initialize itself with a file from the office.microsoft.com servers.  You can find this initialization xml here:

https://office.microsoft.com/Config15

The XML file defines all the end points that the Office Apps layer will use when performing various tasks.  As you can see, the file has a lot of endpoints.

When your goal is to download and install an app, many of the steps behind the scenes are abstracted away from you.  There are many steps involved:

1. Open your site, browse to the "Add an app" link
2. Browse to the "SharePoint Store".  In reality, this is a page hosted on your sharepoint server that is simply making JSON calls to the office servers
3. The page will request the list of all Apps using (http://spservername/_layouts/15/storefront.aspx?task=GetApps&bm=US&cm=en%2DUS&category=&free=0&sort=7&query=&catalog=0).  This JSON is the list of all apps.  There are currently 240 of them in the US marketplace (at the bottom of this post is a CSV file with all the app details).  My code knows how to pull this information and save to a database, so I have a full list of all apps and all of their details.  This was helpful in my review process so that I didn't need to create some massive spreadsheet with all the details typed in!
4. Clicking on an app will pull down more specific details about the app using this url (http://spservername/_layouts/15/storefront.aspx?task=GetAppDetails&bm=US&cm=en%2DUS&appid=APPID&catalog=0").  This app detail gives you the information like if it is a paid app, whether it includes a trial license capability, etc. 
5. Once you are on a page that allows you to "Add" an app, you click the "Add It" button.  What happens here is a post back occurs on your SP Server and a redirect is generated to the office.microsoft.com marketplace.
6. When you arrive at the marketplace, cookies will be interrogated to see if you have logged in via LiveID.  If you have not, you are redirected to the LiveID servers for auth
7. Once authenticated, you are returned to the marketplace using a special cookie identifier to rebuild your session
8. You are then prompted to confirm that you really want the app, upon confirmation, you are redirected back to your sharepoint site via a HTTP GET with all the necessary license details in the querystring
9. Your SharePoint server will take these details and do a validation post behind the scenes to confirm that the app and license match – there is no tricking it here
10. Once the license is validated, it will save the information to the AppManagement database tied to your App Management Service Application
11. It will then intitiate a download of the app's cab file from the office servers using this url (https://office.microsoft.com/client/15/appinstallinfo/query?cmu=" + contentMarket + "&clientToken=" + rawXMLEntitlementToken + "&ret=0&av=" + av + "&build=" + build + "&origin=" + origin + "&corr=" + corr;).  this url contains your clientToken which is the validation that you have purchased the app from the marketplace.
12. This page will redirect you to the download page and in the body of the response will be the app cab file.
13. Once downloaded, the file is installed into the target
14. In a separate step you must trust the application. I was eventually able to automate this part. I attempted to do it via CSOM, but I get the dreaded "Sideloading of apps is not enabled on this site" error.  Bummer, no way to get an appinstance or productid from CSOM.  

So all those steps!  I didn't want to do that for every App (if you were to do this manually, it is going to take you a full day or two!), sooo…I built a tool for it.  By the way, it takes almost an hour to download and install everything!  This tools has the following features:

• Downloads all app details from the App Store into a database (this is easy via one JSON request)
• Allows you to Install an App from the App Store (you can pass it a list of the ones you want it to activate) – this is handy for building an environment that matches your production environment.  Or possibly rebuilding another environment
• Downloads all App CAB files for all installed apps
• Allows you to activate a list of apps on a web of your choosing (this would be after you have installed them)
• Tool Side-Effect – This tool allows you to bypass the validation checking of the feature requirements on the App Detail page.  In otherwards, you can purchase free apps and download their CAB file, even if you can't use it because it is O365 targeted
• Both free and trial apps are supported, did not put in support for selecting a payment method to buy an app that didn't have a trial…that would be pretty dumb!
• Supports on-prem and O365

I did try to be sneaky and hack the purchase API. It passes a special query string with the price and purchase model.  I tried to make everything a free app, but it was too smart for that.  They got lucky…

I have some clean up to do on the tool to make it something I would give out to the masses but it does work and works both for on-prem and O365. However, I'm also thinking if I do publish it, I may make it a product. Think…iOS app automated updater type of thing.  There isn't a UI that says, "Update All" in the SharePoint store interface.

Some things that suck about the Apps APIs:

• AppId – the AppId is used everywhere (on the store, in the purchase api), except in SharePoint (well it is at least saved in the databse)!  When a basic user wants to add or remove an app, it is not the AppId that you use, it is an AppInstanceId, or the completely different ProductId passed in the license.  Consistency is key here and this needs changed, see my next items…
• For automating the "Trust It" part of the process – You can't get the product id to pass into the appdownload function when you want an app to be installed in your web. You have to query the app management database *directly* to get the product id.  I think this is stupid.  You should just be able to pass the AssetId to get the app to install in the web.  This is probably one of the reasons there are no CSOM methods to install an already existing app in a web (without side-loading it).  This needs fixed!
• Adding an app – each time you add an app to a web, the content database that contains that web will download a copy of the app (from the office app store on the internet) into the database.  There is no central repository for these apps by default.  The AppCatalog site is needed for this, but I don't think it is downloaded into the catalog and I'm pretty sure it will pull it from the catalog into the local database anyway, this is a waste of space.
• Uninstalling apps – there is no easy way to get all the AppInstanceIdsproductIds.  I had to query the database directly to get them to uninstall all apps after I was done testing.  The APIs do not expose this so its a manual process without my tool.
• There is a database leak in the AppPrincipals table.  Each time you add and remove an app, a new entry is added, but not removed on removal.  not a big deal, but could cause problems later and needs to be fixed.
• App name versus App name versus app part name – this was weird, there are cases where these are all different. Makes it confusing when other people have similarly named appsapp parts.  Microsoft needs some governance here.  Also wondering what happens when you upload an app part with the same name as someone else?  Bet its not good….#badhacker
• When you have so many App parts in the UI, you really need a search feature!  Clicking the next arrow 10 times to get to the end just isn't cool.
  
• Apps that are deployed to Azure require O365 and don't work on-prem, you need special OAuth tokens to pass to the remote Azure app
  

So how do I rate all these Apps?  And what is some common things I see as issues with the App Model?  It's pretty obvious that no one has ever posted a "here's how you write an App the proper way" type of document.  Lots of technical "how to's", but no common sense "must dos".  Well, here you go on my observations:

• Business – Come on Man!  Those of you that put an app out there and are charging for it when someone put one out there for free are just stupid.  Do an analysis before you put something in the app store.  Don't waste your time or other people's time!  It's called a SWOT analysis by the way for you non-MBA types.
• Complexity – There are very few sophisticated apps in the marketplace.  Lots of simple ones, but the ones that did put in effort, really show.
• Feature Enablement – Its unfortunate that the SP Store page doesn't tell you what you need as far as feature pre-reqs to install some apps.  It just says "You can't install that app".  In the app details (which are hidden on the page and in the json), it will tell you what features are needed that you must enable (for instance, project server feature must be installed for some apps).  It would be nice if it gave you the option to enable the missing feature directly on the page
• Navigation – If you have an Azure hosted app, put the damn navigation at the top to get back to my site!
• Scalability – Most apps use an App list for data storage.  I can see these lists filling up with many many items over a long period of time.  This will be problematic.
• Ribbon – higher quality apps integrate with the ribbon in some manner.  you should think how your app will do this
• App Testing – It was painful to add a new page for every app I wanted to test and I can see trying to add many "cool" app parts to the same page can clutter the page
• Preview features – Please don't make us go to a page with your app part on it.  You should at least give us the courtesy of a preview tab!
• Centralized data storage – The biggest issue I see with apps is there is no sandbox area that is shared storage.  It is completely unacceptable to have the same app deployed on multiple sites with each one getting its own list
• Trials – People that publish an app without a trial and have a price on it…good luck with that.  I'm not buying your App and no one else should either, don't buy an app unless you can trial it first

Before we get into my rating, some interesting "awards" for the apps that I put together:

• Best overall App – this was tough, but in the end, Nintex has had that up there for a while and it is really sweet design
• Most Expensive App – Booking of Resources – really…WTF…$899.99 for that piece of #$^&$?
  
• Most over-prices apps – SONJAsAPPS
• Most Apps of same type – it was like a tie between image carousel apps and map apps…how many of these things do we need people?!?  Come on man!
• Most Fun App –
  Cheezburgers
• Most Creative App – Colour Lab
  
• Highest Quality Apps – Layer2
• Lowest Quality Apps – Point8020
• Dumbest App – Spallery
• Most Complex – Kodak Imaging App
• Most Publisher Apps – Microsoft with 22, SONJAsAPPS with 17 but they don't count, so SR1 Development Limited get the award with 11
• Notable Mention – Avior GEO Location Utility
  

And finally, my CJG rating of all apps in the Office marketplace (at least the ones that are free and with trial, 30 of the 240 do not have a trial, those are the lame ones):

Title	Rank	Price	Comment
Corporate News App	5	Free	By far the best news app I have seen. You need this one!
Auctions Lite	5	$1.49	I can see HR or teams having fund rasiers with this. It’s a pretty sweet app!
BPMN Quick Guide	5	Free	Awesome techneic of delivering document based content.
BlackCompass Visualizer for SharePoint	5	Free	If you liked the Chart web part, you will love this stuff! Runs against list data
Nintex Workflow for Office 365	5	Free	They are crazy partiers, but damn, they build some sweet apps (this is free???)
ODATA QueryBuilder	5	Free	Pretty sweet tool. Very nice usage of SharePoint App model!
My Places	5	Free	Its simple, its sweet and it works. Very nice. Good for an HR page or two.
Layer2 Cloud Connector App for SharePoint	5	$249.99	The design and UI is amazing with this app. The auth support is even better, o365 or ADFS. This is a real app.
User Redirect Plus	5	$9.99	Very ingenious app, definitely distinguished itself from the pack.
Cheezburgers	5	Free	At first I was like WTF is Cheezburgers, then I knew. Awesome. Content is king, especially funny content.
Gimmal Drop Zone App for SharePoint 2013	5	Free	Oh yeah, this is good stuff. Needs an on-prem version
Colour Lab	5	Free	Awesome. A web based HexFinder tool…love it
HelloSign	5	Free	This has HUGE potential. App is free, but the service is not
Layer2 Term Set Glossary	5	$7.99	Layer2 knows how to build apps, they need to be encouraged to build more (oh, I guess I just did)
HelloFax	5	Free	This has HUGE potential. App is free, but the service is not
BC Site Tools	5	Free	I LOVE code generation tools, this one is far better than the single piece meal ones
SharePoint Manager 2013 Online	5	$99.99	Awesome. Works on-prem too
Avior GEO Location Utility	5	Free	OK, now we are talking! Something that really adds value. You can add GeoColumns to your lists!
ND|TagSearch	5	$3.99	I hated that we lost the vectors in SP2013 search, but this tool tried to replace our lost functionality with document tag clouds
The Needs App	4	$29.99	High performance selling was a big thing at IBM, an app that improves people thoughts about customer management is always good
BS WordPress Blog Reader	4	Free	COntent is king, any way to get your content to users is a good thing.
REST API Demo	4	Free	A nice tool to help those of you just starting out with REST APIs in SP2013
Quote of The Day	4	Free	Again, I love content push like this. As long as a bad quote doesn't show up like "Shit happens"
vCURRENCY	4	Free	Allows you to put currency rates on a page
Dropbox Import	4	Free	Allows you to move your dropbox items into SharePoint libraries
Reports and Documents Generator	4	$39.99	Sweet doc generation from sharepoint list data
AE Daily Dilbert App Part	4	Free	Fun app, the image displays so it works
SofTract Copy/Move Items	4	$1.49	Integrates into the listitem ribbon, allows you to move items. Cool
QRCode App for SharePoint	4	Free	Generates a QR Code from statis app part property. Would be nice if you could generate based on querystring or form parameter
HP ePrint	4	Free	This is sweet! Think moble device printing!
metaEngine Easy Search	4	$1.99	Shows how to work with search REST APIs. Provides some more simple UI elements than a search center.
In Out Board	4	$1.99	What a refreshing app. If you had Lync you'd already have this 🙂
Countdown Timer to Any Date	4	$1.49	Nice, simple, valuable – you could still do this with basic javascript, but…whatever
AvePoint Task & Calendar Sync for Office 365	4	Free	App is free, but you need an AvePoint account. Neat idea
My SharePoint Sites	4	$1.99	Oh yeah…this is a good one! Navigation in sharepoint is soo bad
Decision Maker	4	Free	Not an everyday business usage type of thing, but fun.
Facebook Wall	4	Free	Cool, nice documentation
ShareIt	4	Free	Not bad, only app that I have seen that does this in the app store
World Clock and Weather	4	Free	Watch out bamboo web parts…there is a new sheriff in town!. Shows weather
CarouselSlider	4	Free	This looks just like the eBay news slider we built. Only ours works no matter where you put it!
SPoodle Scheduling	4	Free	Not bad, simple and useful.
Facebook Integration	4	Free	Couldn't get this to work, but it looks pretty sweet. From what I can tell it will export your Pages library content to facebook
SharePoint File Explorer	4	Free	A simple site explorer that is better than the out of the box one
Infoware Plus Forget Not+	4	$9.99	Has some cool graphics, I can see people using this. Just need an on-premise version.
Tip of the Day	4	Free	Pushing content that helps is great!
Exam Maker	4	Free	Nice. Now if it would feed into an LMS, even better
ConnectWithUs	4	Free	Simple effective app part for showing all your social media accounts
KODAK Info Activate Solution Limited Edition	4	$9.99	Ugg…SilverLight? Really? Poor Kodak…always so slow to adapt the ONLY app that forces you to download and install something. And what is with the title? That all being said…damn this is a nice app!
Business Contact Tracker	4	Free	Lots of apps like this, but the UI really stands out with this one, very nice
Tweeting for Dollarz	4	Free	Very cool idea!
Competition Tracker	4	Free	Nicely built app. I see uses for this. Too bad its O365 only
Social Nucleus	4	Free	Cool! Shows a social graph of your social contacts
Tip Of The Day Plus	4	$1.49	Allows you to put in your own tips
InfoBus eForm	4	$49.99	Wasn't aware that infopath rendering didn't work in O365? These guys have a form editing solution
TimeOff Lite	4	$4.99	’d pay for this one. It does have some nice value to it. Worth checking out HR departments! It would get a 5 but I think most major companies have ERP systems that keep track of this sort of thing
Documents shared with me	4	Free	Allows you to see your skydrive documents that others have shared with you
VisualSP (Beta) – In-Context Help for End Users	4	Free	The fact it doesn’t simply redirect you our of your sharepoint instance is great. It integrates into the ribbon of lists to show context sensitive help items. Nice.
Poll App	4	Free	I can see this being used by orgs. A safer bet than using the Survey list.
Social Squared Beta	3	Free	A different UI thrown over discussion boards. You lose the reputation features if you use this.
CSV Uploader	3	$4.99	Very simple appUI, I can see use for this in orgs.
“Napa” Office 365 Development Tools	3	Free	SDK for development
Important Messages	3	Free	Just wish there was shared storage for cross websiteweb app display
Yammer App for SharePoint	3	Free	Until Yammer gets there act together and plays nicely with the Redmond teams, they deserver mid range grades.
Google Office Locator	3	Free	Maybe some value. Not sure. Too many map apps that do similar functianlity
Anniversary Timeline	3	Free	Most people are getting private about their lives, not sure this app has much value
Bright Banner	3	Free	Has a great documentation page and has some real world value.
News Ticker	3	Free	Would be nice if you could configure to pull from remote account to display the values across all the app parts across all webssitesweb apps
SideKick 365 xRM	3	$119.99	If you don’t already have SalesForce.com or CRM Dynamics and are a small company…why not. Go for it. Price is crazy though…
SPLatestNews	3	Free	Not sure why anyone would have this on their sharepoint pages.. Its like a google feed of the top news by category
HTML5 Video WebPart	3	$2.99	Simple HTML5 video player
Instant Consulting Practice	3	Free	Evidentially I'm ready to be a consultant – good for single person O365 lawyers, but that's it
Tab Display	3	$2.99	Different spin on a news app. Only tab app, but competes against accordians. Not sure I'd pay 2.99 for this
Accordion Display App	3	Free	Cool app I can see some people using this
Chat Web Part Touch	3	Free	If you don’t have a instant messaging platform, this could replace it, but why do this when you can do the newsfeed????
QuestionBox	3	$1.49	Probably one of the better feedback apps
Employee Of The Month	3	$1.49	Not much to say about this, not bad, not great
Commercial Real Estate Deal Tracker	3	Free	Microsoft example apps
Adlib PDF Publisher	3	Free	Simple PDF generator. Provides more support for other document types, not just Microsoft
Meal Delivery	3	Free	Microsoft example app
Order and Products	3	Free	Microsoft example apps
Calendar Alerts For SharePoint 2013	3	$1.99	Possibly some value here. You decide. Nothing that made me excited.
Infoware Plus Image Navigation+	3	$9.99	JAIC – this one you pay for pretty graphics
Web Analytics Integration	3	Free	Rather than modify your master page to add google analytics, you can add this app part to track page data
BannerMaker	3	$239.99	There are lots of image carosels, not sure I would pay 239.99 for a designer tool when the output is the same
MySiteFinest	3	$139.99	Looks like a cool app, but again to pricey
Print List	3	$1.99	it provides a simple view of a list item that can be printed. I like the PDF output one better
Site Usage	3	Free	Same data you can get from other APIs. But simple for those that don’t know how to get that data.
Dell Social Hub	3	Free	It seemed like a cool app to start, but I couldn’t get it to do anything other than show Dell feeds!
Ultimate Slider	3	$30.99	I like the Corporeate News App way better! And its free!!! Why pay so much for this app? Don’t…
List Carousel	3	Free	Another simple nes app
Bulk Edit	3	Free	The UI in Project Server isn't very intuitive, this is a slightly better UI
Firma Digitale Remota	3	Free	Another digital signature app. This space will be good, needs some work on the UI
vSlider Video Carousel	3	Free	A better spin on all the news carosels.
SharePoint 2013 Media Center App	3	Free	Only apps of its kind, allows you to connect ot Windows Azure Media Services to show remotely hosted videos
Resource planner	3	$699.99	If you don't have project server and you want something more than the out of box gantt chart, this may be an option
Project Management	3	Free	Microsoft example apps
Pingar Metadata Extractor for SharePoint 2013	3	Free	Bulk file uploader app, its ok
Contact Us Form	3	Free	Basic contact us app part
Recommendation Sharing	3	Free	Microsoft example apps
Site TreeView	3	$1.49	A cheap alternative to the SharePoint Manager Tool. I'm sure they will force the other tools prices down.
SofTract Visual CSV Uploader	3	$1.49	Allows you to upload data into a list using CSV files
TIM (Team Instant Messaging)	3	$1.99	Another chat platform. Again, only works in cnotext of the web it is isntalled on…why replace newsfeeds? It is persistenct
Ideas and Votes Tracking	3	Free	Microsoft example apps
Publish All Enterprise Projects	3	Free	Interesting app
Google Drive Import Beta	3	Free	Imports content from your google drive to sharepoint library
From files to lists	3	$24.99	There are others, you need trial there guys
Novadia Phonebook	3	Free	Phonebook search app
Collaboration Suite Quick Access	3	Free	If you don't have a breadcrumb control, which is the case in O365, then this might help you search for sites in the site collection
OneBit Quick Poll	3	Free	A polling app, I think their are better ways to do this
Export ListItem to Word	3	$599.99	Your #$&^*%& crazy to think people will pay $599 for this. This is a weeks worth of dev work at most.
Student Contracts	3	Free	Microsoft example apps
Recruiting Helper	3	Free	Microsoft example apps
Public Site Content Migration	3	Free	Microsoft example apps
SideKick365 xRM PRO	3	$159.99	This is a PRO version? SideKick is pulling the wool over people's eye with their apps. NOT cool
Feedback and Comments	3	$1.99	A page rating app.
ContentCommander™	3	$34.99	Gives you some cool "fucntion" key features when editing documents and items. Always wondering why Microsoft didn't give us that
Tool Tracker	3	Free	Microsoft example apps
SP2 Power Search	3	$2.49	Anthing that can help end users work with SP Search are good
FlashPlayer	3	$94.99	These guys again…
Training Management	3	Free	Microsoft example apps
SharePoint Object Explorer	3	Free	Not as nice as the others
Idera SharePoint Library Analyzer	3	Free	I can see some potential here with the type of app it is, but it didn't do anything for me the way it is
Computer Asset Management System	3	Free	Microsoft example apps
Student Drop in Management	3	Free	Microsoft example apps
BlueBridge List2PDF	3	$9.99	Was thinking this let you manage the columns and to edit a template, but alas, no. Simply exports the list columns to pdf. Not sure I'd pay 9.99 for this
UpdateCocktail	3	$189.99	Competes against the content search web part? Oh…and you get to pay $189 for it
AE Twitter App Part	3	Free	another simple tweet feed app
Conservazione Sostitutiva	3	Free	Good implementation of some compliance law. There should be more apps like this
ObilogicSharePoint2013RSSApp	3	Free	Rss viewer
Fast Charts	3	Free	Another replacement for the Charts web parts, not as good as the other chart app part
MozArt	3	$1.49	Free ones can accomplish the same goal
ThemeBuilder	3	$219.99	Probably their best app. Looks ok, seems pricey like all their other apps
Rdz RSSReader	3	Free	An RSS reader app, simple.
SP2 CAML Query Tester	3	Free	It only does CAML, there are other tools that bring in all the APIs into a single App
Instant Legal Practice	3	Free	Evidentially I'm ready to be a lawyer
Rackspace Cloud Storage App for Microsoft SharePoint	3	Free	Looks to be made for a very specific clientel from Rackspace. Design looks good
Infoware Plus Weather+	3	$9.99	A weather app, that you pay for, again pretty graphics
Public Site Tools For SharePoint 2013	3	$49.99	An app to expose hidden O365 features.
Sports League Manager Starter (FREE Source Code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
Yammer Manager	2	Free	couldn't get it to do anything
Image Carousel	2	Free	Its a JAIC (Just another image carosel)
SideKick 365 MPM	2	$39.99	A simple loan management tool, not worth paying for
SiteMonitor	2	$3.49
Mileage Tracker	2	Free	I'd think most large companies have an interal way of tracking this on expense reports, if your a single consultant , mmaybe you would use this
Recruitment Manager Starter Pack (Free Source Code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
ConnectToSkyDrive	2	$649.99	Seriously, I'm not amused or impressed at this point
Action Tracker	2	Free	Seems they tried to improve on a basic task list with a custom UI? Don't see value here
List data in Bing Map	2	Free	Another map app
PeopleA-Z	2	$399.99	Nooo…overpriced again…
CheckIn	2	$169.99	Are you kidding me? Really? 4Square will give this to you for free…$169…LOL
SP2 Job Listings	2	$9.99	Shows your internal job listings. Not worth $9.99
Google powered Extranet search app	2	Free	Why use google search again? The google appliance can't compete against SP2013 search
SideKick 365 LPM	2	$14.99	A simple lease management tool, not worth paying for
Test Center Starter Pack (FREE Source Code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
TalkFollowers	2	Free	Supposidly this will post to both shrepoint and yammer feeds
SimplePages	2	$449.99	Really? $499 for something that you can do via the ribbon? Wolves…or better yet…muppets!
Seers Sticky Notes	2	Free	Onenote Site Notebook? Guess if you don’t have OWA this may be helpful
DocumentsTreeView	2	$249.99	Your killin me smalls…
Analytics	2	$249.99	Too pricy, there are other apps that are cheaper and better
PublishingPages	2	$649.99	WTF…$699? Really?
YoutubeGallery	2	$9.99	Yeah no…there are free ones guys
SideKick 365 DCM	2	$29.99	A simple case management tool, not worth paying for
SimpleNews	2	$249.99	Simplenews…yeah, simple to write this app…easy to overcharge
CountDown	2	$89.99	Jesus…$89 for this?
Galleriffic App for SharePoint	2	Free	Another picture gallery app
Content Type Explorer	2	Free	Why do people need this again? And it errors out a lot
Course Designer Starter Pack (FREE Source Code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
Picture Gallery View	2	$9.99	Not worth buying, too many free ones
myFAQ	2	Free	Don’t’ get the point of this app given you can do wikis that do the same thing.
Image Slideshow	2	$2.99	Nothing to see here, don't waste your time
Bing Office Locator	2	Free	Just like all the other map apps
Instant Feed	2	$1.99	The image looks like it’s a pretty simple app. It doesn’t have a trial version. I don’t get why someone would pay for this when you can do a simply script web part to do the same thing
Smartpoint's Kanban – Easier Task Management	2	Free	An attempt to replace out of box task views. Project server can do this
Rough Cut Editor Starter Pack (FREE Source Code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
Activities Tracker	2	Free	Ehhh…
ISlider	2	$19.99	JAIC – does let a user pick the image
SlideShow	2	$199.99	another slide show app
TeamSpirit Doc Finder	2	$9.99	Not sure what the heck this does
KW Marquee	2	Free	A newsfeed ticker (this was so 1996…)
RSS Viewer	2	$2.49	Another RSS viewer
Map App	2	Free	another map app (could you tell from teh title?), at least this one provides direction features
Booking of Resources	2	$899.99	Not a very compelling app, I can see potential though, but was the first that I saw to have links to app list and site settings pages
QuickPoll	2	$199.99	Very quick to create a poll, no real features
Accordion Display	2	$2.99	Not worth the price. Especially when there is a free one.
Shreyan Sticky Notes	2	Free	Looks to be the same as the other sticky notes. Why two in the app store?
_api JSON Viewer	2	Free	simple rest json viewer for your URLsAPIs. Nothing that you couldn't do with other tools.
Google Maps for SharePoint	2	Free	There are free ones
Timer Clock	2	Free	Hmm…not sure what the point was…
Meet The Team	2	$3.49	Uses the same code as the Employee of the month
bill tracker	2	Free	Really? Manage your bills in sharePoint? I'm thinking…no
Sticky Notes	2	Free	pretty pictures…yeah
Tic Tac Toe Starter Pack (FREE source code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
Collage App	2	Free	No value app
Daily Dilbert Strip	2	Free	Fun. Not sure if they have the right to Dilbert, but hey, its funny no matter. It didn’t work though when I installed it. The image is broken link
Sports News On SharePoint	2	Free	No real world value here unless you work for a pro sports team
Content Type Manager	2	Free	You don't need this app
Locations Mapper	2	Free	Uses Google maps and looks good, but not a lot of functionality here
Lightning Chat	2	$1.99	Doesn’t keep persistence after leaving the page, why replace LyncSkypeAOLetc
Invoicing Manager Starter Pack (FREE source code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
Pipeline Manager Starter Pack (FREE source code)	2	Free	Point8020, Microsoft built example apps, you don't need to do it, quit cluttering the app store
Home and Back Buttons	2	$2.99	Because sharepoint navigation really doesn't work at all? Ouch… $2.99 for two button…lovely
SharePoint Security	2	$1.49	Funny, just shows a user and what they have access too. I wouldn't pay for this
Url Shortener Lite	1	Free	Couldn't get it to work
Location Finder	1	Free	Could fingure out how to display the map
My Locations	1	Free	Very simple app, not much value here
SharePoint Amplifier	1	$9.99	Requires a blog site in O365. Tried to work with it, but it no worky
Staff Directory	1	$7.99	The staff directory is really the local web's user information list data, not your entire staff. Do not pay for this
Quick Info	1	$1.49	Not worth downlaoding
Locations Map Plus For SharePoint 2013	1	$9.99	No real difference from the others. This guy is smoking crack for charging $9.99…dumb
SP2 Feedback	1	$24.99	Simple feedback app…why $24.99? There are free ones
Corporate News	1	$399.99	Don’t buy this…waste of money when there are free ones.
2A QuickStart	1	Free	A new window? Really? that is not sharepoint integration!
Stock Chart	1	$1.49	doesn't work
CertiViz	1	Free	Doesn't work
Spallery	1	$1.49	Might be worst of all gallery apps
My Rubrica	1	Free	A contact manager app…Outlook? iPhone? Not sure why you would use this app
RateMe Pro	1	$19.99	Have no idea what this does…
Brava Viewer		Free
BPA CRM 2013 Preview		Free
UMT Essentials Lite		$4.99
Posta Elettronica Certificata		Free
iGlobe Group Calendar		$89.99
PROJILITY ProjectWare LiveWindow		$49.99
TPG Risk Chart		$19.99
MeetSweet!		Free
Feedback Form		$1.99
AvePoint Meetings for SharePoint Online		Free
SOLVIN TrackTimesheet Go		Free
iGlobe Integrator		$14.99
TimeOFF Pro		$19.99
Newsfeed Notifier		$1.99
CS Milestone Trend Analysis		Free
OneBit Survey Master		Free
TPG MTA Chart		$29.99
ImageGallery		$9.99
RateMe Lite		$4.99
General Knowledge Quiz For SharePoint 2013		$1.99
MetaVis Information Manager for Office 365		Free
ConnectYard for Sharepoint		Free
TPG WBS Chart		$19.99

My good friend Fabian tweeted about a problem he was having with SharePoint 2013 workflows. 

http://fabiangwilliams.wordpress.com/2013/09/06/help-unable-to-create-list-using-sharepoint-2013-rest-api-in-spd2013/

It was an interesting problem that I have never seen anyone tackle before so the fact he was having a error wasn't surprising.  It did however peak my interest about the problem, so I asked him to send me his workflow. 

He exported his workflow as a template (solution file) and sent it over.  First issue I had was importing the workflow as a sandboxed or farm solution.  It would give me this crazy error:

svr-spwfe01 : Failed to load receiver assembly "Microsoft.SharePoint.WorkflowServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" for feature "GetItemsFromSPOListInstances" (ID: eb5838ef-cda9-42b2-9ea8-624f060d554c).: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SharePoint.WorkflowServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.SharePoint.WorkflowServices, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& 

So I abandoned that idea and looked to import the XAML file like I did in 2010.  I created a 2013 workflow but alas, no xaml files are to be found in the "All Files" view.  Where do they go?  Well, there is a hidden list called "wfsvc".  It has no views on it so simply unhiding it won't allow you to browse the files inside it.  Attempting to create a view is also futile.  So I turned to Windows PowerShell, navigated to the SPFile that represented the workflow xaml file and replaced it with Fabian's.  Refresh of SharePoint Designer and bingo…I had all his info.  Sweet.

The main issue of Fabian's was that when he made the Web Service call he would pass a pre-created FedAuth token (yeah I told him he would have to figure out how to generate this in the future) which established a context for him.  This context was as his SharePoint admin account, but the actual running account (the "actor") was the Workflow service.  Of course the SP Admin account can do whatever it wants, but the workflow service cannot.  This workflow service account is NOT the account that the actual workflow manager backend window service is running under.  It is the Application principal account that is created for the workflow service.  If you browse to the database, you can see there are two database tables:

• AppPrincipalPerms
• AppPrincipals

If you browse the AppPrincipals table, you will find one or more principles.  Consider these to simply be user just like any other user, yet they are applications and as such, do not have a password.  You can find the ID of the AppPrincipal for workflow by taking the second index of the NameID for the item with title "Workflow" (NOTE:  This id is different in EVERY farm, which is a bummer):

•  i:0i.t|ms.sp.ext|78cf7810-a2c9-497e-bffe-8c25368c8bc0@fcd171b6-20f6-443e-8aea-474a4b529e6e

This is the identity that will run the workflow.  If you review the permission mask in the AppPrinicpalPerms table for this appprincipal, you will see that it is set to "3" by default.  The enumerations for the masks are:

• None
• Guest
• Read
• Write
• Manage
• FullControl

"3" means "Write".  Which allows the workflow service to write out list items, but not to create lists.  This would require level 4 or higher.  You can elevate the permission mask with the following Windows PowerShell:

$web = get-spweb http://teams.contoso.com
$appworkflow = Get-SPAppPrincipal -site http://teams.contoso.com -nameidentifier 78cf7810-a2c9-497e-bffe-8c25368c8bc0@fcd171b6-20f6-443e-8aea-474a4b529e6e
Set-SPAppPrincipalPermission -AppPrincipal $appworkflow -site $web -right fullcontrol -scope site

Your workflow service app principal can now create lists!

Fabian will be researching how to add these app permissions in O365, of which he did here:

http://fabiangwilliams.wordpress.com/2013/09/08/actually-resolved-unable-to-create-list-using-sharepoint-2013-rest-api-in-spd2013/

Enjoy!
Chris

This is the data and information from the codeplex project on the Shredded Storage testing framework.  This was to be released several months ago, but things just didn't work out the way we all planned.  So I'm posting it here:

Purpose of this whitepaper

Shredded Storage has up to this point been a very elusive
feature of SharePoint 2013.  There have
been several posts in the community that have attempted to tackle this
incredibly advanced topic, and although some have been spot on in most regards,
they were far from complete when describing the intricate dance that occurs
throughout the many layers of SharePoint and Office.  Each of the authors had different goals and
motivations for creating this whitepaper. 
These ranged from simply the curiosity standpoint to the much more
directed and measurable product vendor implications on features like RBS and
Deduplication. The result of our ambitions is this highly technical white paper
and supporting code that uncovers how shredded storage works in various
scenarios and how it can be improved in the future.  We have also provided you with a guide on how
you can achieve the same conclusions as we have here.

Introduction to Shredded Storage

Shredded Storage is a new data platform improvement in
SharePoint 2013 related to the management of large binary objects.  Shredded Storage is designed to accomplish
three tasks; reduce storage, optimize bandwidth and optimize file I/O.  The idea behind Shredded Storage is if
subsequent writes back to the database are the same, then there is no need to save
those parts again and thus you save on storage, network and disk I/O.  This particular feature was originally
designed to manage heavy write scenarios such as co-authoring. 

All of these are accomplished by simply breaking apart the
blobs into smaller pieces.  It really is
as simple as that, but the details are much more complex. Think about if you
were to manually take a piece of paper and rip it into a bunch of pieces.  If the piece of paper has an image on it,
then it would make sense that you should be able to easily piece it back
together.  The difference here, is your
brain is a pretty amazing machine and can recognize the pieces.  Let’s take a blank piece of paper.  How easy do you think it would be to put it
back together if you randomly rip it apart? 
Actually, it would be quite easy as the pieces will fit nicely.  Now say you put the blank piece of paper into
a shredder that shreds them all into the same size.  How easy do you think it will be to put it
back together?  Not very easy.  You would need something to guide you in
putting it back together.  Maybe some
ultraviolet markers?  That’s where the
implementation of shredded storage comes into play.

Shredded Storage is much more than simply a set of stored
procedures in a SQL Server database.  It
is actually a tiered solution made up of three layers.  The SharePoint layer (which includes the
basic object model and other services like CelllStorage.svc and OneNote.svc
that support Office Clients and Office Web Apps), the Cobalt layer, and the
database backend.  The combination of
these three parts are what define Shredded Storage.  However, as you will learn, the Windows
Operating System has a MAJOR role to play in everything.

This white paper will dive very deeply into all three layers
and how they work with each other. 

SharePoint Layer (Object Model)

When it comes to download and upload of files, SharePoint
will utilize the SharePoint Object Model to make calls to the SPFile methods
(SaveBinary and OpenBinary).  This method
in turn calls the SPRequest class methods. 
Inside this method is where the magic happens.  This method, which is not managed code, but
in the COM portion of the SharePoint application will make a call to generate
an SPFileStreamStore based on the SPFile information.  This SPFileStreamStore is managed by the
SPFIleStreamManager class.  The
SPFileStreamManager is the key class to managing the Cobalt layer on the
SharePoint side.  You will also see some
other Microsoft.SharePoint.Cobalt* classes that help with the interaction.

Writing Files

When it comes to writing a file to the SharePoint database,
the SPFileStreamManager will execute its ComputeStreamsToWrite method.  This method is in charge of creating a
SPFIleStreamStore object which is then used in creating a
CobaltStreamRevokableContainer object. 
This object will then contain a CobaltFilePartition which in turns
contains a schema. , getting back a Schema from the SPHostBlobStore .

Reading Files

When it comes to reading a file from the database, again the
SPFileStreamStore is the main class that contains these methods.  The most important methods are the
GetBlobsById and GetBlobsAfterBsn methods. 
These two methods are where blobs are retrieved from the database.  A set of stream ids are passed to the
proc_GetStreamsById stored procedure and the first chunk of data is returned.  A data reader is opened and each blob is
converted to a SPCoordinatedStreamBuffer. 
So where does the FileReadChunkSize come into play?  It is actually used as an input parameter to
the GetStreamsById stored procedure to tell SQL Server to try to break up the
file content into smaller parts (but it will only be called once and only get
for each requested shred no more than the FileReadChunkSize.  After the first return of the data, if there
is more to be returned, subsequent calls are made using the GetStream stored
procedure and passing in the offset for the content, but yet again, only
returning up to the FileReadChunkSize. 
These parts are then aggregated by the SPCoordinatedStreamBuffer and
once all parts of the shreds have been retrieved they are put back together
into the actual shreds for consumption by the Cobalt layer.

These fully pieced together shreds are made into a Cobalt
HostBlobStore.Blob and a list of these are returned from the method.   The data of the shred is converted to a
Cobalt Atom.

The net result is the value of the FileReadChunkSize does
have an effect on the number and size of the result sets that are returned back
to the web front end.  If this setting is
several multiples smaller than the FileWriteChunkSize, you will see an almost
similar multiple in the time it takes to download the file.

SharePoint Layer (CellStorage.svc)

A separate but related feature to Shredded Storage is the
ability for Office Clients to do delta updates when working with Office XML
based documents that reside in SharePoint. 
As part of the process, Office clients will attempt to make a call to
the CellStorage WCF service.   If the
service is available the WFE will only send the requested pieces of a file to
the client.  On a save operation, only
delta changes to a file are sent back to the WFE.  Note that when this service is not available
or an error occurs, the Office client will fall back to the normal HTTP PUT
mechanisms as if it were a regular file share and the entire file will be sent
on each save. When this delta update and Shredded Storage is being utilized,
you see both an optimization from the client to the WFE and the WFE to SQL
server.   

Delta updates only support Office XML documents. The reason
they are only Office XML is that deep inside CellStorage.svc it makes a call to
create a pointer to an XmlReader object. 
Obviously, the older files are not based on this file format and an
attempt to read them is futile, therefore, you won't have any calls to
cellstorage.svc, but simply the regular HTTP PUT calls.  The CellStorage protocol (which is part of
the Cobalt assemblies) supports various different command types:

·        
GetDocMetaInfo

·        
WhoAmI

·        
ServerTime

·        
Cell (get and set)

·        
Coauth

·        
SchemaLock

·        
ReleaseLock

There is a standard process to this:

·        
First step is to send a request for the Document
MetaInfo

·        
Second step is to actually get the parts of the
document that are being viewededited at that moment (cell get)

·        
Third step is to request to start editing the
document (requesting a schema lock)

·        
Fourth step is send back any changes that a
person makes (cell set)

·        
Last step is to tell the server you are done
(release the schema lock)

During this entire process, the client will ping the
sharedaccess.asmx web service to ensure that it is the only one editing the
document.  This is done about every 20
seconds.  As part of the request, it is
looking for the ETag to change.  If it
has changed, that means someone updated the document and the version you have
is now old and you will need to refresh your copy, or overwrite what they
did.  This scenario should never happen,
but it looks like they attempt to check that it somehow does.

CellStorage.svc requests and responses are using the older
XML format as part of the messages.  This
is a costly approach because XML is a bloated format as compared to the newer
JSON format which is used in Office Web Apps communications.  We hope one day this protocol will change to
support the newer format and increase the performance even further.

SharePoint 2013’s has several WCF Services aside from the
CellStorage service.  Many of these
service reside in the ISAPI directory of the SharePoint Root (C:Program
FilesCommon FilesMicrosoft SharedWeb Server Extensions15ISAPI).  If you open the web.config file of the ISAPI
directory, you will find that all the WCF service bindings are configured with
a maxBufferSize of “4194304”.  

This will limit the size of the files from which the Office
Clients will be able to download the file shreds.  As you will see later, the performance of the
IO decreases dramatically when you set a FileWriteChunkSize higher than “4MB”.

Office Web Apps 2013 – (Onenote.ashx)

Office Web Apps has an entirely different set of multi-user
editing features.  These features are
using the end point OneNote.ashx.  This
API works hand and hand with Cobalt to “lock” sections of a document when
multiple users are using it.  This multi-user
and locking operation will generate “partitions” in the shredded storage
database store.  Partitions are sets of
shreds in the content database that correspond to what each user is changing in
a file for their editing session. 

When using regular Object Model calls, these shreds are
normally broken apart by the boundaries defined by the Cobalt schema of the
file being edited.  When using multi user
editing, an even granular break down of the shreds is performed (down to the
separate xml files in the Office XML document) and mapped to the internal xml
components being edited.  When the
editing sessions are closed, the partitions are collapsed back to their single
partition.  Having an editing session
“end”, is a very important part to the process. 
If a user doesn’t close their session, the partitions will be left in
the database until the service cleans them up. 
In our code testing, we found it possible to lock yourself out of
editing, when you had another session open. 
From a coding standpoint, this is the correct way to do this.  The default timeout for an Office Web Apps
session is 5 minutes.[A8] 

The process of starting an editing session works like this:

·        
User clicks on an Office XML document from a
library

·        
A sharepoint page called WOPIiframe.aspx is
called.  This page sets up the necessary
access tokens that OWA will use for communicating to SharePoint

·        
At this point the IFrame will redirect you to
the OWA server and initialize the session via the WordViewerFrame.aspx page and
the docdatahandler.ashx http handler

·        
A JSON object that fully describes the document
will be sent to the browser.  This JSON
notation is a very complex set of JSON objects that map to the XML elements
inside the Office XML document

·        
A user starts an editing session with OWA, which
initiates the first call to OneNote.ashx. 
From this point on, all operations are done through OneNote.ashx.

·        
Every few seconds the browser will ping
OneNote.ashx to request any changes. 
This JSON format and protocol could be the discussion of a whitepaper on
its own

As the above conversation between the client browser and
Office Web Apps is performed, the Office Web Apps server is talking to
CellStorage.svc through the WordCompanionServer class. This class will
reference the HostEnvironment class for information on where the file is
located and what request adapters are implemented to proxy the calls between
OWA and the target.  The two most
important adapters are the ICellStorageAdapter and the ICoauthAdapter.  The ICellStorageAdapter is an interface that
is implemented by the CellStorage.svc WCF service mentioned above.   

Now that you have some background of the process and
interaction of these components, it’s time to look at the Cobalt layer which is
where a majority of the code for our CellStorage.svc resides.

Cobalt

The shredding part of the process is implemented by a much
improved Cobalt API (Microsoft.Cobalt). 
Cobalt is the layer responsible for implementing a break-up schema that
dissects the files into smaller parts, commonly called shreds in
SharePoint.  Cobalt is designed to be
utilized with any type of file, not just Office file types.  The Cobalt assembly also contains the code
that handles all the FSHTTP requests discussed earlier.  Cobalt operations include:

·         TBD

Cobalt supports several different schemas that define how a
file is broken apart.  Cobalt will accept
a file, determine what type of file it is (similar with how search determine a
files format) and will then apply an algorithm based on this file type.  This means that the type of file
matters.  You simply cannot upload an
Office XML file and expect to get the same results as compared to a simple
image file of the same size.  Not only
does the algorithm change based on file type, it also changes based on the size
of the file.  Table 1 shows the paths
that are taken and the algorithm that is applies when that path is picked.

TABLE 1:

TBD

 

Each algorithm will break apart the blobs in a different
way.  Therefore the same sized file that
is a Word Office XML document will break apart differently than an older Word
document that is simply binary.  You may
also find that images (blobs within blobs in the case of Office XML Documents)
will be treated differently and may get their own shred.

As the Cobalt breaks apart the blob, there is the possibly
of two extra shreds added to the set of shreds that store configuration
information about the other shreds. 
These configuration shreds help the web front end determine if the
shreds have changed and new shreds need to be committed to the database.

 

Database

SPFarm Configuration

In SharePoint 2013, the out of box default for the maximum
size of these parts is no larger than 64K, however this can be configured to be
larger.  This FileWriteChuckSize setting
is a major focus of this whitepaper.

It is again important to note that the size of the shreds
will not equal the file size divided by the write chunk size.  It is simply a watermark that is used as a
guide for the Cobalt algorithm involved.

These configuration shreds are used by the WFE to determine
what shreds need to be saved back to the content database.  This keeps the WFE from having to download
the entire set of shreds to do a comparison. 
These configuration shreds will add a small amount to each blob added to
the database.  Note however, there are
some circumstances where there is only one shred created that includes both
configuration and data

This break-up of the blob into shreds doesn’t achieve any
storage optimization until you turn on SharePoint versioning on a document
library.  When versioning has been
enabled, each time a user saves a file, the SharePoint WFE will break apart the
blob into segments using the same method that it used before.  Then each of these shreds will be compared
with the shreds already in the database. 
If they did not change, then there is no need to send the shred to the
database for persistence.  This database
level de-duping operation has been proven to achieve 30-40% storage savings.  This is where you gain in network
optimization between the WFE and the SQL Server.  You also gain optimization from the
standpoint of file IO.  Now you are only
sending and saving a small part of the file rather than the entire file.  This is also important in the fact that if
you had done a similar operation in SharePoint 2010, but only updated the
metadata for a file, you would end up with another version of the file in the
database.  With Shredded Storage, this is
not the case and your storage is optimized greatly.  Lastly, when you have implemented a log
shipping disaster recovery strategy, you will notice smaller writes produce
smaller transaction log sizes and drive a more efficient log shipping experience.

Content Database – Overview

Once Cobalt and SharePoint have determined the Cobalt
HostBlobStore.Blob(s) that need to be saved, each one is passed to the content
database via the .

Shredded Storage optimization is document focused.  This requires a quick description of how
SharePoint saves files.  When you upload
a document to a document library or add an item with an attachment, it creates
a new list item with a unique id assigned to it. It will also get a unique
document id created for the file attached to the item when not a document library.  The two of these are tied to each other and
both are unique guids.  When you upload
the same file to a second list, a new list item and document id will be created
that have no relation to the first set. When it comes to Shredded Storage, it
is this document id that is the important part. 
So in summary, if you upload the same document in two different document
libraries, you will gain no Shredded Storage benefits. 

Content Database – Tables

There are four tables in the database that manage the shredded
storage blobs.  This includes:

·        
dbo.DocStreams

·        
dbo.DocsToStreams

·        
dbo.AllDocs

·        
dbo.AllDocVersions

In SharePoint 2010 dbo.AllDocStreams stored the document
stream and related data for documents with content streams, in SharePoint
Server 2013 dbo.DocStreams replaces dbo.AllDocStreams where each row stores a
portion of the BLOB.

The improved protocols associated with Shredded Storage
identify the rows (in the new DocStreams table) necessary to be updated to
support the change and updates the BLOB associated with that change in the
corresponding row. Several new columns are present in the DocStreams table that
represent a shredded BLOB including:

·        
BSN: The BSN of the stream binary piece.

·        
Data: Contains a subset of the binary data of
the stream binary piece unless the stream binary piece is stored in Remote BLOB
Storage.

·        
Offset: The offset into the stream binary piece
where this subset data belongs.

·        
Length: The size, in bytes, of this subset data
of the stream binary piece.

·        
RbsId: If this stream binary piece is stored in
remote BLOB storage, this value MUST contain the remote BLOB storage identifier
of a subset of the binary data of the stream binary piece. Otherwise it MUST be
NULL.

A new DocToStreams table contains a pointer to a
corresponding row in dbo.DocStreams.  The
BLOB Sequence Number (BSN) is used to manage the BLOB sequence across
dbo.AllDocVersions, dbo.DocsToStreams, and dbo.DocStreams.  NextBSN is used to manage the last BSN for
each BLOB.

dbo.AllDocs contains a single row per file similar to
SharePoint Server 2010.

dbo.AllDocVersions contains one or more rows per file and
one row per file version.

Content Database – Stored Procedures

The main stored procedures in the process include:

·         TBD

Read and Write Scenarios

Client to WFE (Read)

TBD

WFE to Database (Read)

The following table outlines some of the generic results
from our testing when utilizing the FileReadChunkSize as compared to the size
of the file (and not compared to the size of the FileWriteChunkSize):

FileReadChunksize (% of file size)	Performance Hit
>12.5%	Typical Normal read operation
6% < x > 12.5%	10% hit on read operation
3% < x > 6%	20% hit on read operation
x < 3%	50% hit on read operation

These values would point to the fact that the default
setting anticipates your average file size will be under 512KB.  It should be noted that when you install
SharePoint 2013, all files that accompany the software are under this limit.

Client to WFE (Write[A9] )

These calls can be measured using Fiddler tracing.  You can simulate an Office 2010 client by
disabling Shredded Storage and attempting to download a file from SharePoint.

WFE to Database (Write)

TBD…

Remote Blob Storage

Two quick facts; Shredded Storage does not require RBS and
RBS does not require Shredded Storage. However, you should also be aware that
when Shredded Storage is used with RBS, there are side effects to having both
Shredded Storage and RBS enabled.  RBS’s
real value is realized only when you are working with larger files. When
combined with Shredded Storage’s default max file size of 64K, the actual
implementation of RBS can have a negative impact on the performance of
retrieving a complete blob from the storage subsystem.  However, some RBS implementations presents
capabilities that Shredded Storage does not.  One such feature is the ability to actually do
the de-duplication of the shreds. 
De-duplication will monitor for when the same blob is being saved and
then creates a pointer to a single instance of the blob.  This solves the problem of uploading the same
document to multiple libraries and having it use just as much space in the disk
subsystem.  As previously mentioned, this
is something that Shredded Storage does not do in this current implementation
and depending on your RBS provider, can do.

 

Deduplication

In testing of this feature when enabled with shredded
storage, it has been found that the de-dupping feature actually causes a
performance hit when writing files with target size of 64K and loses a majority
of its efficiency when the shredded storage watermark is maxed.  Proper testing should be done to ensure that
any changes you make to the Shredded Storage setting or features you have
implemented on your disk subsystem are compatible with your performance requirements.

 

Simple Shredded Storage Facts

Shredded Storage can provide other network optimizations
other than WFE to SQL when configured with the proper set of tools.  Thus far, you have been presented with the
utilization of Cobalt on the WFE to SQL Server side of the wire.  Originally, Cobalt was designed with
multi-user editing in mind.  This goes
hand and hand with SharePoint perfectly as it is a collaboration platform.  This makes sense as a common pattern is allowing
users to be able to collaborate on documents at the same time, whether that
collaboration feature is in a viewing or editing capacity.

Disabling Shredded Storage

There are properties that allow you to attempt to turn off
shredded storage.  This attempt can be
accomplished by setting the FileWriteChuckSize so that the shreds are incredibly
large, up to 2GB, and force the likeliness that only one shred will be created.
Be warned that changing the shredded storage setting over 4MB will incur a
large I/O hit for larger files. It is not recommended that you set this value
above 4MB.

It should be noted that you CAN disable shredded
storage.  This can be accomplished NOT by
changing the FileWriteChunkSize, but the *FILTERED OUT*

This operation is not something you should proceed with
unless you are doing it in a test farm. 
If you have installed SharePoint 2013 and let users upload information
and files, then subsequently disable Shredded Storage, NONE of the files will be modifiable again unless you delete the file
and then add it back OR re-enable shredded storage. 

Shredded Storage Testing Framework

As part of this whitepaper we have published a complete
Shredded Storage testing framework (including the source code) that will allow
you to build your own tests and to validate our tests to confirm our
results.  The following section is
designed to walk you through how to use the tool and analyze the results.

Installing the Tool

Follow these steps for installing the tool:

·        
Download the tool from Http://shreddedstorage.codeplex.com

·        
Run the db script or restore the test database
to SQL Server

·        
Install SQL Server 2008 R2 or later with SQL
Profiler tools

·        
Install the Office SDK 2.0 or higher

·        
Install Fiddler

·        
Copy the SPProfiler.exe, SPProfiler.exe.config
and MonitorSharePoint.tdf to the C: empsystem32 directory

How the tool works

First and foremost. 
The tool IS NOT MULTITHREADED.  Please take this into consideration when
modifying the code as it is not possible to modify it to support multi-threaded
calls with the various logging components used.

Fiddler integration
– You can wrap any call in the StartProxy Method to record any http requests
and responses to see what kind of traffic is being generated.  These tests can help to analyze what the
different is between older non-cobalt calls and cobalt calls.

SQL Profiler
integration – You can start the SQL Profiler tool at any time by calling
StartTrace().

Time Monitoring – You
can record the time it takes for any method calls by calling StartTimer().

TestResult class – All
results should be saved to a TestResult object. 
Once the object is populated, you can simply save it to the database
with the SaveTestResult() method.

Generating content
– There are several methods that support the dynamic generation of Word and
PowerPoint files.  You can use these
methods to create a directory of a specific size, then make the call to
UploadDirectory (described below) to test the upload of the files

Reading Results

The following SQL queries provided with the tool are
provided to help you analyze your data:

·         TBD

Performance Tests

The most basic
tests would revolve around uploading and download files.  This can be easily tested by creating sets of
files of varying sizes and uploading and downloading them several times.  This is but one piece of the testing that is
needed for shredded storage. 

The following table
outlines some of the methods you can call to run various tests in your
environment.  You can build your own
calls using PowerShell or modifying the tools code

 

Method Name	Shredded Storage On?	What does it test?	What should you see?
RunDynamicUploadTest	Yes	This test uploads a file starting at a target file size and then halves the FileWriteChunkSize as many times as you tell it.	The shred sizes will increase as the FileWriteChunkSize decreases.
RunDynamicDownloadTest	Yes	This tests downloads a file starting at a target FileReadChunkSize and will half the value as many times as you tell it	The number of read stored procedures will increase and the time it takes for the file to download will take longer.
RunImageUploadTest	Yes	This test will generate a word document with images and then upload them.	You will see that the entire file is shredded and the image is not separated out into its own shred
RunOfficeDownload	YesNO	This test will use the office client to open a file stored in SharePoint.	You should see that the file size is almost exactly the same size as the file itself in the HTTP calls.
UploadDirectory	YesNo	This test method will upload a local directory to the sharepoint server.	You can use this to test a batch of file uploads for comparison between SP2010 vs SP2013.  Also a good way to test various size files with FileWriteChunkSize settings
RunOWATest	Yes	This method tests the HTTP calls between client and OWA server, you also see the SQL IO on the backend.	This method is not fully implemented as the OneNote.ashx protocol is VERY difficult to reverse engineer.
RunShreddedStorageDisableTest	Both	This test demonstrated that it is very bad to disable shredded storage using the ServerDebugFlags.	You will see that any subsequent uploads of files will break after disabling shredded storage.
AnalyzeFarm	N/A	This will analyze all the files in your Farm and give you stats on the files contained in all your content databases.	You will need these stats to pick the proper setting for your FileReadChunkSize.
TestOpenLargeFileFromOffice	Yes	This test is for analyzing the effect WCF 4MB limit has on the Cobalt layer in SharePoint.	You will see that the SQL IO has a HUGE hit when the FileWriteSChunkSize is set above 4MB.  However, for smaller files the actual amount of time it take for the client to render is not affected.
RunMaxWriteSettingTest[A10]	Yes	This test will set the FileWriteChunkSize to its maxiumum setting of 2GB and then compares how the system runs between the default setting of 64KB.	You should see that all the files are single sets of shreds versus multiple shreds with the lower value.
TBD – modify the WCF Settings and run large file test
TBD – DeDub enable, monitor file size
TBD – Enabled RBS, monitor upload speeds
TBD – Enable RBS, monitor download speeds
RunManualTest	YesNo	This test wills start recording any of the actions you perform and then save to the database when you are finished	This will assist with any custom testing you want to do without writing any code.
RunModifiedPropertyVersionTest	Yes	This test will look at when you simply change a property like “Title” and don’t change the file bytes.	You should see the entire file gets written back when a single shred exists. This is UNEXPECTED behavior.
RunModifiedVersionTest	Yes	This test will generate a word file, upload the file, then modify it by adding text to the end and upload it again.	This is the best case scenario for shredded storage.  You should see that the count and total size of the shreds don’t change drastically as in the worst case scenario test.[A11]
RunChangingVersionTest	Yes	This test will generate a file, upload the file, then generate a completely new file and upload it with the same name.	This is a worst case scenario test.  You should see the total number of shreds increase in the database with each version uploaded.  You will also see the overhead that shredded storage adds to the base file.  By the time you get to version 10, you should have 10x the number of shreds and size.

 

Comparison to SharePoint 2010

TBD

I ran into this the other day.  I thought possibly something was broken on the OWA servers, but after reviewing, I didn't find anything wrong with OWA.

Amazingly, what I found was that the site url had been changed and the link to the site notebook stays the same!  Attempting to click on the link will cause OWA to attempt to open the site notebook, but it will fail miserably.  You will need to update your navigation node to point to your new site url.

Chris

There are several questions about how to create and assign device channels programmatically.  I just attempted to do the same and found the following:

• You can create a Device Channel using code and Windows PowerShell as they are stored in a list
• Creating a master page to device channel mapping is not technically available via object model or Windows PowerShell (at least via the APIs that they have provided you)

To create a device channel, you can do this:

using (SPSite site = new SPSite("http://intranet.contoso.com"))
            using (SPWeb web = site.RootWeb)
            {
                SPList list = web.Lists.TryGetList("Device Channels");

                SPListItem li = list.AddItem();
                li["Name"] = "Windows Phone";
                li["ContentType"] = "Device Channel";
                li["Active"] = true;
                //alias can contain no spaces
                li["Alias"] = "WindowsPhone";
                li["Description"] = "The windows phone mobile channel";
                li["Device Inclusion Rules"] = "Windows Phone";
                li.Update();               
            }

If you look at how the master page settings page is laid out, it shows you all the device channels and any master pages that are tied to them.  When you look at the code, you will find that the settings are converted into a MasterPageMappingsFile object (in the Microsoft.SharePoint.Publishing.Mobile namespace). It inherits from a base class called MappingsFile<T>, both of which are marked as internal and thus you cannot use them.  When you review how it builds the list of mappings, it does so using a file called __DeviceChannelMappings.aspx that is stored in the "/_catalogs/masterpage/__DeviceChannelMappings.aspx".  It looks like this:

<%@ Reference VirtualPath="~CustomMasterUrlForMapping0" %><%@ Reference VirtualPath="~CustomMasterUrlForMapping1" %><%@ Page Language="C#" Inherits="Microsoft.SharePoint.Publishing.Internal.WebControls.MappingsFileBasePage" %><html xmlns:mso="urn:schemas-microsoft-com:office:office" xmlns:msdt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882"><%@ Register Tagprefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<head>
<meta name="WebPartPageExpansion" content="full" />
<!–[if gte mso 9]>
<SharePoint:CTFieldRefs runat=server Prefix="mso:" FieldList="FileLeafRef"><xml>
 
<mso:CustomDocumentProperties>
<mso:ContentTypeId msdt:dt="string">0x010100FDA260FD09A244B183A666F2AE2475A6</mso:ContentTypeId>
</mso:CustomDocumentProperties>
</xml></SharePoint:CTFieldRefs><![endif]–>
</head><body><mappings>
  <mapping>
    <channelAlias>WindowsPhone</channelAlias>
    <masterUrl href="/_catalogs/masterpage/windowsphone.intranet.master" token="~sitecollection/_catalogs/masterpage/windowsphone.intranet.master" />
  </mapping>
  <defaultChannelMapping>
    <siteMasterUrl token="~sitecollection/_catalogs/masterpage/seattle.master" href="/_catalogs/masterpage/seattle.master" />
    <systemMasterUrl token="~sitecollection/_catalogs/masterpage/seattle.master" href="/_catalogs/masterpage/seattle.master" />
    <alternateCssUrl token="" href="" />
    <themedCssFolderUrl token="" href="" isthemeshared="false" />
  </defaultChannelMapping>
</mappings></body></html>

Now that you know where the values are stored, you can programmatically modify the file using XML tools by downloading the files, changing it and then uploading it.  It should be noted that the file format may change in the future and its most likely why they have locked it down from an object model code standpoint.

Enjoy!
Chris