Architecting Connected Systems Blog
This is the full list of all SharePoint App store apps. I have been asked at least 3 times over the past month for this list. Not sure why the interest lately, but here it is!
Some quick stats:
Source CSV File is attached below…
Enjoy!
Chris
Gotta love cross team interaction sometimes. In trying to get PowerPivot and PowerView to work on SP2013, I ran into many many issues. One of which has no items on the internet when you search for it! So here's my steps to resolve getting this whole thing to work right.
Once this is done, theoretically you should be good to go, but alas…shit happens.
The biggest issue is the snapshots of the files. I have seen many of these errors before, but this time things were a bit different.
When it comes to the actual GallerySnapshot.exe tool, it is a very interesting beast, some details are found here:
And for all other issues, you have this great post:
Which one part that I had not seen before was the Group Policy being applied for the Trusted sites. Because I had set this up in my image, the code could not set its own policy and would error. This post was helpful to find that issue (which was to remove the policy):
For my particular situation, I had what this poor soul had:
This error is "NullReferenceException in Microsoft.AnalysisServices.SPAddin.ReportGallery.SnapshotHandler.RegisterGlobalExtensionHandlers()" Ugly. So of course, I went in and found the assembly just as he did and sure enough, the section is missing in my web.config file of my BI Center site. What does it need to look like? Here ya go (answer in bold):
<?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
<configuration>
<configSections>
<sectionGroup name="SharePoint">
<section name="SafeControls"
type="Microsoft.SharePoint.ApplicationRuntime.SafeControlsConfigurationHandler,
Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" />
<section name="RuntimeFilter"
type="System.Configuration.SingleTagSectionHandler, System,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section name="WebPartLimits"
type="System.Configuration.SingleTagSectionHandler, System,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section name="WebPartCache"
type="System.Configuration.SingleTagSectionHandler, System, Version=4.0.0.0,
Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section name="WebPartWorkItem"
type="System.Configuration.SingleTagSectionHandler, System,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section name="WebPartControls"
type="System.Configuration.SingleTagSectionHandler, System,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section name="SafeMode"
type="Microsoft.SharePoint.ApplicationRuntime.SafeModeConfigurationHandler,
Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" />
<section name="MergedActions"
type="System.Configuration.SingleTagSectionHandler, System,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section
name="PeoplePickerWildcards"
type="System.Configuration.NameValueSectionHandler, System,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<section name="WorkflowServices"
type="Microsoft.SharePoint.Workflow.ServiceConfigurationSection,
Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" />
<section name="BlobCache"
type="System.Configuration.SingleTagSectionHandler, System,
Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
/>
<section name="OutputCacheProfiles"
type="System.Configuration.SingleTagSectionHandler, System,
Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
/>
<section name="ObjectCache"
type="System.Configuration.SingleTagSectionHandler, System,
Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
/>
<section name="MediaAssets"
type="System.Configuration.SingleTagSectionHandler, System,
Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
/>
<section
name="ApplicationAuthentication"
type="Microsoft.SharePoint.IdentityModel.ApplicationAuthenticationConfigurationSection,
Microsoft.SharePoint.IdentityModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
/>
<section
name="CustomCaptureSection"
type="Microsoft.AnalysisServices.SPAddin.ReportGallery.CustomCaptureSection"/>
</sectionGroup>
<sectionGroup
name="System.Workflow.ComponentModel.WorkflowCompiler"
type="System.Workflow.ComponentModel.Compiler.WorkflowCompilerConfigurationSectionGroup,
System.Workflow.ComponentModel, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35">
<section name="authorizedTypes"
type="System.Workflow.ComponentModel.Compiler.AuthorizedTypesSectionHandler,
System.Workflow.ComponentModel, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35" />
<section name="authorizedRuleTypes"
type="System.Workflow.ComponentModel.Compiler.AuthorizedTypesSectionHandler,
System.Workflow.ComponentModel, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35" />
</sectionGroup>
<sectionGroup
name="microsoft.sharepoint.client">
<section name="serverRuntime"
type="Microsoft.SharePoint.Client.ClientServiceServerRuntimeSection,
Microsoft.SharePoint.Client.ServerRuntime, Version=15.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" />
</sectionGroup>
<sectionGroup name="ReportingServices">
<section name="DataExtensions"
type="System.Configuration.DictionarySectionHandler" />
</sectionGroup>
<sectionGroup name="Bpm">
<section name="FCODaoProviders"
type="System.Configuration.DictionarySectionHandler" />
</sectionGroup>
<sectionGroup name="reportserver">
<section name="redirection" type="Microsoft.ReportingServices.SharePoint.Configuration.RSRedirectConfigSection,
RSSharePointSoapProxy, Version=11.0.0.0, Culture=neutral,
PublicKeyToken=89845dcd8080cc91" />
</sectionGroup>
<section name="microsoft.identityModel"
type="Microsoft.IdentityModel.Configuration.MicrosoftIdentityModelSection,
Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35" />
</configSections>
<microsoft.sharepoint.client>
<serverRuntime>
<hostTypes>
<add
type="Microsoft.SharePoint.Client.SPClientServiceHost,
Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" />
</hostTypes>
</serverRuntime>
</microsoft.sharepoint.client>
<SharePoint>
<CustomCaptureSection createProcessMethodForSnapShot="0"
snapshotCaptureTimeoutSecods="300" maxSnapshotsCount="19"
>
<Handlers>
</Handlers>
</CustomCaptureSection>
…
</configuration>
After adding these entries…your PowerPivot and PowerView snapshots will work.
Enjoy,
Chris
So what the heck does that title mean? Well, for those of you that are not familiar with Search (which is a majority of you out there). The actual engine is called "Ceres". As in the dwarf planet in our solar system (Wikipedia). Keeping with universe terms, there is also a constellation of nodes in the search engine that make up the universe of bodies in the engine. If you take a minute, you will find several references to Constellation in the various classes inside the assemblies, but enough about the universe, what about extending the Ceres engine?
When it comes to search, many of you are already familiar with the various nodes types that make up the system. This includes:
But that's the easy part. and so are the architecture design aspects! This post will take you into a rabbit hole that you may never come out of! For the purpose of this post, we are interested in the Content Processing component AND the Query component. If you dive into the core of the Content Processing component you will find that it is made up of a series of flows. You can find the registered flows in the "C:Program FilesMicrosoft Office Servers15.0SearchResourcesBundles" directory, I will describe what these dlls are and how they get generated later in the post. Here is the full list (in the future I will update this post with what each of these flows purpose is):
The most important flow is the Microsoft.Crawlerflow. This flow is the master flow and defines the order of how all the other flows will be executed. A flow is simply an xml document that defines the flows and operators that should be executed on an item that is processed in the engine. The xml makes up an OperatorGraph. Each Operator has a name and a type attribute. The type attribute is made up of the namespace where the class lives that contains the code for the flow and then the name property of a special attribute added to the class. Each operator is deserialized into an instance of a class as the flow is "parsed". As you review the xml, you should see that the flow of the flow is determined by the "operatorMonkier" that has the name of the next operator that should be executed. The first part of this file looks like the following:
|
<?xml version="1.0" encoding="utf-8"?> <OperatorGraph dslVersion="1.0.0.0" name="" xmlns="http://schemas.microsoft.com/ceres/studio/2009/10/flow"> <Operators> <Operator name="FlowInput" type="Microsoft.Ceres.Evaluation.Operators.Core.Input"> <Operator name="Init" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.Mapper"> |
As you can see the first operator that is executed is of the type "Microsoft.Ceres.Evaluation.Operators.Core.Input". This means that if you look in the "Microsoft.Ceres.Evaluation.Operators" namespace, you will find a class that is decorated like the following:
|
[Serializable, Operator("Input", MinInputCount=0, MaxInputCount=0)] public class InputOperator : TypedOperatorBase<InputOperator>, IMemoryUsingOperator, IOutputTypeConfigurableOperator { … |
You should note that the class is marked as serializable and that the Operator attribute has been added with the name "Input". Again, the combination of the namespace of the class and the name of the attribute are used to find the operator when the flow is executed.
One of the flows that I am most interested in is the Microsoft.CrawlerContentEnrichmentSubFlow. As some of you are aware, you can "extend", really don't like that word used in context of Content Enrichment now that I know how to do flow insertion, using a web service to add your own logic to create new crawled properties on items that pass through the engine. You can find more information about content enrichment and examples of using it at http://msdn.microsoft.com/en-us/library/jj163968.aspx. Now, Microsoft is going to tell you that this is the only supported way to extend the Ceres engine. And that is correct. What I am about to show you has never been done outside of Microsoft and if you venture down this path, you do so on your own. Anyway, the problem with the CES is that it is not flexible and it uses stupid old technology called web services. that means it is sending this big ugly xml around on the wire…not JSON. Bummer. That's not the only thing. When you look at the pipeline and all the things you are indexing, if you do not put a trigger on the CES, EVERY single item will be passed to your service. You would then need to have all kinds of logic to determine the type of the item, what properties exist on it by looping through them all and so many other weird bad things it just makes me cringe. Now, if you do put a trigger on it, you are now limiting yourself to implementing a very targeted set of logic. You have no ability to add more than one CES with different triggers with different logic. Huh? Big feature gap here. I'm not a fan. So for people that just don't want to multiple the time it takes to do a crawl by 100 to 1000x over because you implemented CES, you need a better option. A faster option. A more reliable and performant option. One that lives in the engine, not outside of it. If you want to know how to do this…keep reading!
Ok, so this is all simple so far. But how does one add a new flow to the Ceres engine and then implement your own Operators? Well, this is much more difficult than you think!
The first step is to create an operator class that inherits from TypedOperatorBase<T>. Where T is the class name. This is an abstract class and you must implement the method called ValidateAndType. You can see most of this in the operator example above. The next step is to add the Serializable and Operator attributes to the class. Ok, fair enough, now what do we do? If you look at the XML of an operator, you will see that you can implement properties and that those properties are simply deseriabled to the properties in the class. Ok, so add some properties. In my example, I create a class with one property:
|
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using Microsoft.Ceres.Evaluation; namespace CustomOperator public CustomOperator() [Property(Name="custom")] protected override void ValidateAndType(OperatorStatus status, IList<Microsoft.Ceres.Evaluation.Operators.Graphs.Edge> inputEdges) |
Ok, great. So now what do we do? Well, I wasn't sure if the system would just pick up the assembly from the GAC dynamically so I figured, let's just deploy the solution and try to add a flow with the operator in it. Here's how you do that:
Deploy the assembly to the GAC…easy, right-click the project, select "Deploy"
Next, create a new flow (xml file) that uses the operator:
|
<?xml version="1.0" encoding="utf-8" ?> <OperatorGraph dslVersion="1.0.0.0" name="CustomFlow" xmlns=" http://schemas.microsoft.com/ceres/studio/2009/10/flow"> <Operators> <Operator name="SubFlowInput" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.SubFlow.SubFlowInput"> |
Connect to the ceres engine and try to deploy the flow:
|
Add-PsSnapin Microsoft.SharePoint.Powershell & "C:Program FilesMicrosoft Office Servers15.0SearchScriptsceresshell.ps1" Connect-System -Uri (Get-SPEnterpriseSearchServiceApplication).SystemManagerLocations[0] -ServiceIdentity contososp_farm Connect-Engine -NodeTypes InterActionEngine $flowname = "CustomFlow" Remove-Flow $flowname Get-Content C:CustomOperatorCustomOperator$flowname.xml | Out-String | Add-Flow $flowname Stop-Flow –FlowName $flowname –ForceAll |
You will get the following error that the system cannot find the Operator called CustomOperator.CustomOperator. Bummer. So that didn't work. So how do I "register" my operator with the engine? Well, it turns out that their is so much more that needs to be done than simply creating an operator class. You also need to create several other classes with special attributes attached to them. Sooo…here we go!
First off, you will need to create a Producer class. This producer is really the class that does all the work. The operator is really just a way to get some parameters into the producer. As you can see the Producer inherits from SingleOutputProducer<T>, where T is your operator class. Here is an example of the producer:
|
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using Microsoft.Ceres;
namespace CustomOperator private IUpdateableRecord holder; public override void ProcessRecord(IRecord record) base.SetNextRecord(record); |
Next up is to create a NamedPlugInSource. Operators are also called "PlugIns". These plugins must be registered with the system in order for you to use them. If you review all the operator assembiles, you will see that there is always some kind of *PlugInSource class that has the role of adding plugins to the Ceres core system. For my pluginsource, I only have one operator and that is my CustomOperator:
using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Ceres.CoreServices.Services;
using Microsoft.Ceres.CoreServices.Services.Node;
using Microsoft.Ceres.Evaluation.Operators;
using Microsoft.Ceres.Evaluation.Operators.PlugIns;
namespace CustomOperator
{
[DynamicComponent]
public class CustomPlugInSource : NamedPlugInSource<OperatorBase>
{
public static OperatorBase PlugIn1()
{
File.AppendAllText(@"c: empsearch.txt", "PlugIn1");
return new CustomOperator();
}
protected override void AddPlugIns()
{
File.AppendAllText(@"c: empsearch.txt", "AddPlugIns");
Func<OperatorBase> f = PlugIn1;
base.Add(typeof(CustomOperator),f);
}
}
}
Now that you have the plugin built. You will notice that is has been decorated with the "DynamicComponent" attribute. This is where the "Ah-ha" moment kicks in. By adding this attribute to the assembly, Ceres knows that is must start this as a managed component in the system. However, just simply deploying this to the GAC, will not get Ceres to recognize the assembly and load the components. We'll get to that soon, we still have lots more to talk about!
Next up is an Evaluator. An Evaluator is responsible for actually making the call to the producer. In my example I create a class that inherits from ProductEvaluator<T> where T is my CustomOperator. ProductEvaluator is again an abstract class with one method called GetProducer. You must instatiate your producer here and return it. There are many types of producers, but I have not had the time to document all of them as of yet. Soon though!
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Ceres.CoreServices.Services;
using Microsoft.Ceres.CoreServices.Services.Node;
using Microsoft.Ceres.CoreServices.Services.Container;
using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Processing.Producers;
using Microsoft.Ceres.Evaluation.Operators;
using Microsoft.Ceres.Evaluation.DataModel;
using Microsoft.Ceres.Evaluation.Operators.Graphs;
namespace CustomOperator
{
public class CustomEvaluator : ProducerEvaluator<CustomOperator>
{
/*
protected override IRecordSet SetupOutput(Edge outputEdge, IList<IRecordSet> inputs)
{
CustomProducer cp = new CustomProducer();
return null ;
}
*/
protected override IRecordProducer GetProducer(CustomOperator op, Microsoft.Ceres.Evaluation.DataModel.Types.IRecordSetTypeDescriptor type, IEvaluationContext context)
{
return new CustomProducer(op, type, context);
}
}
}
Next on the list is an EvaluatorBinder. The evaluator binder is responsible for registering an operator with an evaluator. This class will inherit from AbstractEvaluatorBinder and need to implement the AddBoundOperators and BindEvaluator methods:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Operators;
namespace CustomOperator
{
public class CustomEvaluatorBinder : AbstractEvaluatorBinder
{
protected override void AddBoundOperators()
{
base.Add(typeof(CustomOperator));
}
public override Evaluator BindEvaluator(OperatorBase op, IEvaluationContext context)
{
if (op is CustomOperator)
{
return new CustomEvaluator();
}
return null;
}
}
}
Last on the list is the EvaluatorBinderSource. Similar to a PlugInSource, this will also be decorated with the DynamicComponent attribute which will instantiate and register the evaluators. Here is the binder source:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Ceres.CoreServices.Services;
using Microsoft.Ceres.CoreServices.Services.DependencyInjection;
using Microsoft.Ceres.CoreServices.Services.Node;
using Microsoft.Ceres.CoreServices.Services.Container;
using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Operators;
namespace CustomOperator
{
[DynamicComponent]
public class CustomEvaluatorBinderSource : AbstractContainerManaged
{
[Exposed]
public IEvaluatorBinder CieEvaluatorBinder
{
get
{
this.exampleBinder = new CustomEvaluatorBinder();
return this.exampleBinder;
}
}
private CustomEvaluatorBinder exampleBinder;
}
}
You now have everything you need to add a new flow and operator to the Ceres engine! Kinda. If you deploy the code at this point, you will notice if you try to run the above install script, you will still get the same error! This is because the assemblies only get loaded when you restart the Host Controller service. NOTE: You can read more about the Host controller service in Randy Williams and I's MSPress book on SharePoint due out very soon. Ok, so re-start the service. Try the commands…NO GO…bummer. But I did everything you said Chris! Why doesn't it recognize my operator? Well…going back to my previous statement, Ceres nodes don't look at the entire GAC and analyze every class. That would be WAAAY to expensive. So it only does the one that it is told to do. This was the final magic step that I stumbled upon very luckily.
For each node that is started (via the NodeRunner.exe process), each one is fed its own configuration file that drives the WCF configuration. This file is stored in C:Program FilesMicrosoft Office Servers15.0SearchRuntime1.0
oderunner.exe.config. It is a very generic file, not much going on here. As part of the NodeController code, it will look for another file and feed some special values into the process in addition to the regular app.config file. These files are stored in the Ceres node directory which is in C:Program FilesMicrosoft Office Servers15.0DataOffice ServerApplicationsSearchNodes<RandomNodeID>. Each role that has been assigned to the server will get a directory under this path. Since most of what we are doing is related to the ContentProcessingComponent, let's look there first. If you open and explore this directory, what you will find is a nodeprofile.xml file. It looks like this…tell me if you notice anything interesting:
<?xml version="1.0" encoding="utf-8"?>
<NodeProfile xmlns="http://schemas.microsoft.com/ceres/hostcontroller/2011/08/nodeprofile">
<AutoStart xmlns="">true</AutoStart>
<Stopped xmlns="">false</Stopped>
<Modules xmlns="" />
<Properties xmlns="">
<Property Key="Managed.Node.Name" Type="string" Value="ContentProcessingComponent1" />
<Property Key="Managed.SystemManager.ConstellationName" Type="string" Value="A99B1A" />
<Property Key="Managed.Node.SystemName" Type="string" Value="A99B1A" />
<Property Key="Managed.SystemManager.ConstellationVersion" Type="int" Value="-1" />
<Property Key="Managed.Runtime.Version" Type="string" Value="1.0" />
<Property Key="Managed.Node.LocalSystemManager" Type="bool" Value="False" />
<Property Key="Managed.Node.ShutdownOnComponentFailed" Type="bool" Value="True" />
<Property Key="Managed.Node.ProcessPriorityClass" Type="string" Value="BelowNormal" />
<Property Key="Managed.Node.DynamicAssemblies" Type="string" Value="Microsoft.Ceres.ContentEngine.AnnotationPrimitives, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Bundles, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Component, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.DataModel.RecordSerializer, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Fields, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.LiveEvaluators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.NlpEvaluators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.NlpOperators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Operators.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Parsing.Component, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Parsing.Evaluators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Parsing.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Processing, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Processing.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Properties, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.AliasLookup, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.RecordCache, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.RecordType, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Repository, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Services, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.SubmitterComponent, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Types, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Util, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Processing.Mars, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.DataModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.DataModel.Types, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Engine, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Engine.WcfTransport, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators.Core, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators.Parsing, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Processing, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Processing.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Services, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.DocumentModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Admin, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.ContentRouter, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Services, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Utils, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Schema.SchemaCatalogProxy, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Query.MarsLookupComponent, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.FastServerMessages, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Schema.SchemaCatalog, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.AnnotationStore, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Automata, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Dictionaries, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.DictionaryInterface, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Ese.Interop, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.RichFields, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.RichTypes, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.StringDistance, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Transformers, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.IndexTokenizer, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.AnalysisEngine.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchAnalytics.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.UsageAnalytics.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;CustomOperator, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7d300eac1b9f50c2" />
<Property Key="Managed.Node.SearchServiceApplicationName" Type="string" Value="14087e61-67e2-4245-b23d-0e52c6dcf704" />
<Property Key="Managed.Node.SystemDisplayName" Type="string" Value="0a1ee46f-59f2-49b7-bfca-bb4d20adaf1a" />
<Property Key="Managed.Node.BasePort" Type="int" Value="17042" />
<Property Key="Managed.Node.BasePort.4" Type="int" Value="17046" />
<Property Key="PortShared" Type="bool" Value="True" />
</Properties>
</NodeProfile>
If you guessed the "Managed.Node.DynamicAssemblies" property…then you are very smart! [:D] Yep…that is what we are looking for. Those are the only assemblies that will be loaded into the AppDomain. Only these assemblies will be interrogated for the DynamicComponent attribute. Great! So as you can see, I have added my CustomOperater assembly to the list. Let's try again and run the script. Dang it! NO GO! It still doesn't like my CustomOperator.CustomOperator operator! Grrr….so at this point, I'm really wondering if my assembly is getting laoded…after a browse in the ULS logs…I see these:
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Inactive] to [Configuring and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiyyo Verbose Microsoft.Ceres.CoreServices.Management.ManagementServer : Registered agent CustomOperator.CustomEvaluatorBinderSource.ComponentManager of type Microsoft.Ceres.CoreServices.Services.Container.IComponentManagerManagementAgent
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Configuring] to [Configured and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Configured] to [Resolving and eventSent=True]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Resolving] to [Readying and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Readying] to [Ready and eventSent=True]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Ready] to [Activating and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Activating] to [Active and eventSent=True]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiywj Medium ComponentManager(CustomOperator.CustomPlugInSource) : CustomOperator.CustomPlugInSource [Active] started
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search Search Platform Services aiyv9 Verbose ComponentManager(CustomOperator.CustomPlugInSource) : ***** QUEUESENTINEL finished task for CustomOperator.CustomPlugInSource: CustomOperator.CustomPlugInSource[Active]state Active
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Inactive] to [Configuring and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Configuring] to [Configured and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Configured] to [Resolving and eventSent=True]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Resolving] to [Readying and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Readying] to [Ready and eventSent=True]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Ready] to [Activating and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywq High ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Activating] to [Active and eventSent=True]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiywj Medium ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomOperator.CustomEvaluatorBinderSource [Active] started
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search Search Platform Services aiyv9 Verbose ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : ***** QUEUESENTINEL finished task for CustomOperator.CustomEvaluatorBinderSource: CustomOperator.CustomEvaluatorBinderSource[Active]state Active
Ok…they ARE being loaded. So what the hell is going on? Well…the clue WAS in the logs files. After running the PowerShell to attempt to add the flow, I noticed something. It was the name of the component that is actually being used to register a flow. Its called QueryProcessingComponent1. Well ok, so it seems that even though the content processing node does all the work, the query processing component manages all the registration of the plugins and operators. After going back to the node directory, I find the QueryProcessingComponent1 directory and find that it too has a NodeProfile.xml file. Bingo. Adding the assembly to the property and restarting the host controller one more time, I again attempt to add a custom flow, with a custom operator.
YYYYYEEEESSSS!!!! NO ERROR…………..I successfully inserted my flow and operator into the Ceres engine! Now, what part is missing? Well, even though the flow is now installed and working, it is not a part of the main flow ("Microsoft.CrawlerFlow"). I would need to insert the flow into that main file and then redeploy it. The main issue with that, is that not all of the operators are recognized by the system. Yeah, weird I know. This is part of the installationdeployment of the search service application and is there by default. if you ever want to make changes, you would need to add all the possible assemblies to the query processing component and then update the main flow.
In terms of debugging, you can attach to the NodeRunner.exe processes and debug your operator and evaluators. Easy.
Now for some clean up. All those bundles of flows at the top of this post. How did they get there? Well, what happens is each time you upload a flow, it will generate a new assembly with the flow added to it as a resource. If you were to reflect on any of the assemblies above, you can get the flow xml out of the assembly. But this is also easily done using the Windows PowerShell commands above.
I will be posted all the code for this project on code.msdn.microsoft.com. You can use it as a starting point for implementing your own flows and operators. But you are probably asking, why would I do something that is not supported. Well, its the same reason you want to keep your job. The customer wants high performance and needs to implement way more than the Content Enrichment Service can provide and saying no will stop any chance you have of completing an incredibly cool and awesome project. Now, why is this not supported if you CAN do it? Well, as you can see, it is VERY complex. Only a few people in the world are going to be able to build these, deploy them and successfully use them. So you are still asking yourself…why did you post this if we can't really do it…great question!
BECAUSE I WANT IT SUPPORTED. If we band together and find various use cases for doing this, the product team will have no choice but to train the Level I, II, and III support people on how to troubleshoot these. As of right now, it is simply unsupported from the fact that the support people don't even know what a flow and operator is when it comes to supporting SharePoint Search (update: met with the search team and the COE support team *is* familiar with flows so were one step closer to support). It would be my goal to get some ISVs to start playing around with creating custom flows and operators to make SharePoint Search a BEAST (not that is already isn't cuz its the best on the market right now, sorry Google Appliance but you suck big time)! So…there you have it. Do with it what you will, have fun, be smart and as always…enjoy!
Chris
So what have I been working on for the past two weeks? Well, other than consulting clients, books and working on my garden, I have also been involved with the Microsoft Learning SharePoint 2013 Advanced Development Microsoft Official Curriculum (MOC) course 20489 that will be available later in the year (sorry no link just yet but it will be here when it is released). I was able to finish up two chapters on Search quickly as that is one of my main fortes, but then decided to take what I though was the middle of two Business Connectivity Services (BCS) chapters. For those of you not familiar with BCS, you can find a great overview here. It turns out, the module was the hardest one! Why? Because it covers things that no one has ever done before (outside of the product team that is).
So what is the big deal about what I worked on? You are probably saying to yourself…BCS has been around for a while right? Well, yes this is very true, and there are several great posts about how to expose external data using SharePoint Designer and Visual Studio using the various BDC model types (Database, WCF, .NET Connectivity and Custom). You can also find how to implement stereotyped methods that support CRUD methods and search indexing (link here). Given all that content, there were a game changing set of features that were added to BCS in SharePoint 2013 that add a whole new level of complexity. These features include:
There are plenty of posts on OData in general (this one from MSDN is pretty awesome if you are just getting started) and a few posts on how to setup a BDC OData model. And although my fellow SharePoint MVP Scot Hillier did a presentation on the subscriber model at the last SharePoint Conference it was only in context of a database model. When it comes to integrating the two features (OData and the subscriber methods) together, that is where a massive black hole exists and is the focus of this blog post.
The first step to getting this whole thing to work is to create an OData service. This is very simple with the tools provided by Visual Studio and steps to do this are provided in this MSDN post
The next step is to build your basic BCS model using the new item template wizard provided in Visual Studio 2012. This has also been nicely blogged about by several of my colleagues and does have an article on MSDN. The important thing to note about the MSDN article I reference is that it is using an OData feed that is hosted by http://services.odata.org. Since you do not own this service, you will not be able to extend it to implement the subscribe and unsubscribe methods that I discuss later in this post. Therefore, you can follow the steps in the article, but use a local instance of your OData service.
Once the service has been generated, you must add some supporting methods to your OData service to accept data from SharePoint when a subscription occurs. There are some gotchas to this. Currently there is no real guidance on how to set this up properly. The little that does exist will point you to mixed signals as to how to successfully setup the communication layers. In my example below, you will see that I am using a GET for the http method. This was the only successful way that I was able to get the method parameters to populate in the web method in the OData service. As you will see later, there are also some very important BDC method properties that must be set in order for all of this to work:
|
[WebGet] // Generate a new Guid that will function as the subscriptionId. if (DeliveryURL == null || EventType == null || EntityName == null || SelectColumns == null) // This sproc will be used to create the subscription in the database. string sqlConn = "Data Source=.;Initial Catalog=Northwind;uid=sa;pwd=Pa$$w0rd"; // Create connection to database. cmd.Parameters.Add(new SqlParameter("@SubscriptionId", subscriptionId)); try return subscriptionId; [WebGet] string sqlConn = "Data Source=.;Initial Catalog=Northwind;uid=sa;pwd=Pa$$w0rd"; // Create connection to database. cmd.Parameters.Add(new SqlParameter("@SubscriptionId", subscriptionId)); try |
On the BCS side, you need to add the stereotyped methods that will send the data to the web methods you just created in the last step. This includes the EntitySubscriber and EntityUnsubscriber methods. First the let's review the EntitySubscriber method. In the table below, you will notice that I am sending the OData web method parameters in the querystring. You can use the '@' parameter notation just like in regular BDC Models to token replace the values. You should also use HTML encoded '&' to signify the ampersand (this was one of the things that took me a while to figure out). Notice the various method parameters. They include:
Here is the working method XML for the Subscribe method:
| <Method Name="SubscribeCustomer" DefaultDisplayName="Customer Subscribe" IsStatic="true"> <Properties> <Property Name="ODataEntityUrl" Type="System.String">/Subscribe?DeliveryURL='@DeliveryURL'&EventType=@EventType&EntityName='@EntityName'&SelectColumns='@SelectColumns'</Property> <Property Name="ODataHttpMethod" Type="System.String">GET</Property> <Property Name="ODataPayloadKind" Type="System.String">Property</Property> <Property Name="ODataFormat" Type="System.String">application/json;odata=verbose</Property> <Property Name="ODataServiceOperation" Type="System.Boolean">false</Property> </Properties> <AccessControlList> <AccessControlEntry Principal="NT AuthorityAuthenticated Users"> <Right BdcRight="Edit" /> <Right BdcRight="Execute" /> <Right BdcRight="SetPermissions" /> <Right BdcRight="SelectableInClients" /> </AccessControlEntry> </AccessControlList> <Parameters> <Parameter Direction="In" Name="@DeliveryURL"> <TypeDescriptor TypeName="System.String" Name="DeliveryURL" > <Properties> <Property Name="IsDeliveryAddress" Type="System.Boolean">true</Property> </Properties> </TypeDescriptor> </Parameter> <Parameter Direction="In" Name="@EventType"> <TypeDescriptor TypeName="System.Int32" Name="EventType" > <Properties> <Property Name="IsEventType" Type="System.Boolean">true</Property> </Properties> </TypeDescriptor> </Parameter> <Parameter Direction="In" Name="@EntityName"> <TypeDescriptor TypeName="System.String" Name="EntityName" > <DefaultValues> <DefaultValue MethodInstanceName="SubscribeCustomer" Type="System.String">Customers</DefaultValue> </DefaultValues> </TypeDescriptor> </Parameter> <Parameter Direction="In" Name="@SelectColumns"> <TypeDescriptor TypeName="System.String" Name="SelectColumns" > <DefaultValues> <DefaultValue MethodInstanceName="SubscribeCustomer" Type="System.String">*</DefaultValue> </DefaultValues> </TypeDescriptor> </Parameter> <Parameter Direction="Return" Name="SubscribeReturn"> <TypeDescriptor Name="SubscriptionId" TypeName="System.String" > <Properties> <Property Name="SubscriptionIdName" Type="System.String">SubscriptionId</Property> </Properties> </TypeDescriptor> </Parameter> </Parameters> <MethodInstances> <MethodInstance Type="EventSubscriber" ReturnParameterName="SubscribeReturn" ReturnTypeDescriptorPath="SubscriptionId" Default="true" Name="SubscribeCustomer" DefaultDisplayName="Customer Subscribe"> <AccessControlList> <AccessControlEntry Principal="NT AuthorityAuthenticated Users"> <Right BdcRight="Edit" /> <Right BdcRight="Execute" /> <Right BdcRight="SetPermissions" /> <Right BdcRight="SelectableInClients" /> </AccessControlEntry> </AccessControlList> </MethodInstance> </MethodInstances> </Method> |
Next is the unsubscribe method, notice how SharePoint must pass back the subscription id that lives in the external system. The name of the SubscriptionIdName property will always be SubscriptionId. This subscription id must be saved somewhere, but the question is…where?:
| <Method Name="UnSubscribeCustomer" DefaultDisplayName="Customer Unsubscribe"> <Properties> <Property Name="ODataEntityUrl" Type="System.String">/UnSubscribe?SubscriptionId='@SubscriptionId'</Property> <Property Name="ODataHttpMethod" Type="System.String">GET</Property> <Property Name="ODataPayloadKind" Type="System.String">Property</Property> <Property Name="ODataServiceOperation" Type="System.Boolean">false</Property> </Properties> <AccessControlList> <AccessControlEntry Principal="NT AuthorityAuthenticated Users"> <Right BdcRight="Edit" /> <Right BdcRight="Execute" /> <Right BdcRight="SetPermissions" /> <Right BdcRight="SelectableInClients" /> </AccessControlEntry> <AccessControlEntry Principal="Contosodomain users"> <Right BdcRight="Edit" /> <Right BdcRight="Execute" /> <Right BdcRight="SetPermissions" /> <Right BdcRight="SelectableInClients" /> </AccessControlEntry> </AccessControlList> <Parameters> <Parameter Name="@SubscriptionId" Direction="In"> <TypeDescriptor Name="SubscriptionId" TypeName="System.String"> <Properties> <Property Name="SubscriptionIdName" Type="System.String">SubscriptionId</Property> </Properties> </TypeDescriptor> </Parameter> </Parameters> <MethodInstances> <MethodInstance Name="UnSubscribeCustomer" DefaultDisplayName="Customer Unsubscribe" Type="EventUnsubscriber" Default="true"> <AccessControlList> <AccessControlEntry Principal="NT AuthorityAuthenticated Users"> <Right BdcRight="Edit" /> <Right BdcRight="Execute" /> <Right BdcRight="SetPermissions" /> <Right BdcRight="SelectableInClients" /> </AccessControlEntry> </AccessControlList> </MethodInstance> </MethodInstances> </Method> |
Now that those items are setup, you need to deploy your BCS model and set permissions. This is very common activity so I'll skip the details in this blog post, however I will say that it is annoying that the user that uploads the model is not automatically added (or have an option somewhere to add them on the import page) as a admin with permissions to the model and methods [:(]
Now that the model is deployed, the next step is to enable a feature that enables subscription support, which brings us back to the question brought up before…where does SharePoint store the subscription id of the external system? A list of course! To create this list, there are two features of which you can enable. One is called BCSEvents, the other is called ExternalSubscription. The funny thing about these two features and their relationship is that the BCSEvents feature is made up of a feature activation receiver. That receiver has only one goal: To activate the ExternalSubscription feature. In addition to this interesting design, you will find that the BCSEvents is a hidden feature whereas the ExternalSubscription feature is actually visible in the web features settings page. What does the ExternalSubscription feature do? It creates our list of course! This list is called "External Subscriptions Store". This is a hidden list and can be unhidden using PowerShell, but it exists in the "_private/ExtSubs" folder and has no views from which you can view the data, so again Windows PowerShell is the way to go if you want to see what lives in the list. Here is a screen shot of the columns of the list:
Next you need to create a subscription. This can be done explicitly or implicitly. The explicit way is to make a call to the entity's subscribe method as shown here (as previously pointed out above, the notification callback url is ignored in an OData Model):
|
function SubscribeEntity() { //these are the helper methods and variables var context; // This code runs when the DOM is ready and creates a context object which is needed to use the SharePoint object model entity = web.getAppBdcCatalog().getEntity("NorthwindModel", "Customers"); lob = entity.getLobSystem(); lobSystemInstances = lob.getLobSystemInstances(); context.executeQueryAsync(GetLobSubscribesystemInstance, failmethod); // Initialize the LobSystemInstance. |
Subscriptions can be one of three types (you can learn more about event types here):
Note that there is no event type that supports ItemAdding, ItemUpdating or ItemDeleting. This means you cannot cancel the insertion, update or deletion in the external source, you can only expect to receive notification after the event has occurred.
The implicit way is to create an alert or to setup an event receiver. This means you should setup an external list pointing to your OData model. You can then use the ribbon to create an alert which will in turn execute the web method call to create the subscription. Note that you must setup your outgoing email settings on your farm, or the alert ribbon button will not display! If an error occurs when creating an event receiver, you will be passed the web method exception from your OData service. This can be very helpful for troubleshooting.
NOTE: When you create an external list, there are several items that seems to get cached in the external list's properties that will require you to delete the list and then re-create it. This means that as you are testing your solution, you should create a Windows PowerShell script that will remove your BDC model, re-deploy it, remove the external list and then add it back.
Once this has all been completed, you can now start telling SharePoint that things have changed. As much work as we have done to this point, it is really rather simple compared to the amount of work needed for this component of the eco-system. There are several approaches you could take to do this:
Each of these have advantages and drawbacks in terms of time and complexity. No matter what, you need some component that will tell SharePoint that something has changed. In the code samples I provide, you have a simple console application that will allow you to send the notification to SharePoint for testing purposes.
So now that you have something that can send a message to SharePoint, what does that message look like? This process of communication is un-documented anywhere, until now, and is the real meat of this post! It turns out that there are two message parsers that come out of the box with SharePoint. These include an IdentityParser and an ODataEntryContentNotificationParser. The difference between the two is that one only tells SharePoint that a set of identities has changed and the other actually can pass the changed properties of the item to SharePoint. Both requires a completely different style of ATOM message to be sent.
In the case of the IdentityParser, it is looking for a message that looks like the code snippet below. This particular piece of XML must have a valid XPath to "/a:feed/a:entry/a:content/m:properties/b:BcsItemIdentity". If it does not, then any call to "retrieve the item" in your event receiver will fail. The message will be received and the event receiver will execute as long as you don't make calls to the various properties that will not be available without the id. You should also be aware that none of the other items that live outside of the XPath are ever looked at and can be anything you like as they are not validated or used:
| <?xml version="1.0" encoding="utf-8" standalone="yes"?> <feed xml:base="http://services.odata.org/OData/OData.svc/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:b="http://schemas.microsoft.com/bcs/2012/" xmlns="http://www.w3.org/2005/Atom"> <entry> <title type="text">Customers</title> <id>http://www.northwind.com/customers</id> <author> <name>External System</name> </author> <content type="application/xml"> <m:properties> <b:BcsItemIdentity m:type="Edm.String"><CustomerID>ALFKI</CustomerID></b:BcsItemIdentity> <d:Name>Customer</d:Name> </m:properties> </content> </entry> </feed> |
In the case of the ODataEntryContentNotificationParser, you must pass an XML message that has a valid XPath to "/a:entry/a:link/m:inline/a:entry". The XML node in this XPath must itself be a valid ATOM message. Again, everything that is outside the XPath seems to be ignored and only the embedded ATOM Message is used:
| <?xml version="1.0" encoding="utf-8" standalone="yes"?> <atom:entry xml:base="http://sphvm-92723:90/WcfDataService2.svc" xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:atom="http://www.w3.org/2005/Atom"> <atom:category term="NorthwindModel.EntitySubscribe" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme"/> <content type="application/xml"> <m:properties> <d:SubscriptionId m:type="Edm.Int32">1</d:SubscriptionId> <d:EntityName>Customers</d:EntityName> <d:DeliveryURL>{11}</d:DeliveryURL> <d:EventType m:type="Edm.Int32">{12}</d:EventType> <d:UserId m:null="true" /> <d:SubscribeTime m:type="Edm.Binary">AAAAAAAABE4=</d:SubscribeTime> <d:SelectColumns>*</d:SelectColumns> </m:properties> </content> <id>OuterId</id> <atom:id>http://sphvm-92723:90/WcfDataService2.svc/EntitySubscribes(1)</atom:id> <atom:link href="EntitySubscribe(1)" rel="self" title="EntitySubscribe"/> <atom:link href="Customers(2147483647)" rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/customers" type="application/atom+xml;type=entry"> <m:inline> <entry xml:base="http://sphvm-92723:90/WcfDataService2.svc/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom"> <id>http://sphvm-92723:90/WcfDataService2.svc/Customers('57849')</id> <title type="text" /> <updated>2012-04-30T11:50:20Z</updated> <author> <name /> </author> <link rel="edit" title="Customer" href="Customers('57849')" /> <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Orders" type="application/atom+xml;type=feed" title="Orders" href="Customers('57849')/Orders" /> <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/CustomerDemographics" type="application/atom+xml;type=feed" title="CustomerDemographics" href="Customers('57849')/CustomerDemographics" /> <category term="NorthwindModel.Customer" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <content type="application/xml"> <m:properties> <d:CustomerID>{0}</d:CustomerID> <d:CompanyName>{1}</d:CompanyName> <d:ContactName>{2}</d:ContactName> <d:ContactTitle>{3}</d:ContactTitle> <d:Address>{4}</d:Address> <d:City>{5}</d:City> <d:Region>{6}</d:Region> <d:PostalCode>{7}</d:PostalCode> <d:Country>{8}</d:Country> <d:Phone>{9}</d:Phone> <d:Fax>{10}</d:Fax> </m:properties> </content> </entry> </m:inline> </atom:link> <title>New Customer entry is added</title> <updated>2011-07-12T09:21:53Z</updated> </atom:entry> |
In addition to the two out of the box parsers, there is a setting that specifies "Custom". By implementing our own NotificationParser, we can format the message in a much more simple and efficient way such as JSON. The main method to implement is the ChangedEntityInstance method. As part of this parser, you will be passed the message byte array in the initialization and it would be your responsibility to parse the message and pass back the entity instance.
| public abstract class NotificationParser { // Methods protected NotificationParser() { } public void Initialize(NameValueCollection headers, byte[] message, IEntity entity, ILobSystemInstance lobSystemInstance) // Properties public abstract Identity ChangedItemIdentity { get; } |
Summary:
Now that you have all the pieces, you can download the code I have placed on the code.msdn.microsoft.com site here. This code has a BCS OData model fully working with the subscriber methods. As code generation techniques have become more common place, OData layers generated via Visual Studio are more common as well. It will be well worth implementing these new BDC method stereotypes in your OData model and in your OData services to provide the ability to be notified when data changes in your remote systems!
Office Web Apps and SharePoint are integrated in several very cool ways. For example, you can see the callout menu on a document library to create new documents:
You can view document previews in the callout of a document:
You can see the document preview in the callout of a search result:
One of the things I noticed right off the bat with Office Web Apps 2013 was that the call out menu has the same options for all office content types of a certain type. This causes some issues when users try to click on the links.
An example of this is the "Follow" link. If you click on the "Follow" link of a document that is stored in a file share, it will error with the following:
I'm sure you could get into the display template and remove that action, but it wasn't as important as some of the other items on my plate when it comes to OWA and SharePoint Search integration. For example, you will also notice that you do not get the ability to view a document that lives in the file share in the thumbnail previewer. I find this unacceptable and so did my customer! So…I started out on the path to figure out how to get it working! Here's what I came up with.
The first thing I thought was cool about Office Web Apps is the ability to "embed" documents in your web pages that live pretty much anywhere. If you open the "http://svr-owa/op/generate.aspx" page, you will see you have the ability to create an embedded link:
Once you create the link to a fileshare document you can put it on any of your HTMLSharePoint pages. However, after creating the link, if you try to open it, you will typically get a "File not found" error:
Turns out this is the main wrapper error around just about anything error that happens in OWA. The main reason that the files won't display is because of permission issues on the Office Web App server. You see, it doesn't open the file as you, it opens it as the Office Web Apps service identity! You can find out what your identity is by opening the IIS Manager on the OWA server and looking for the OpenFromUrlWeb app pool:
By default, this account is set to "Network Service". This account can't do much with secured network based resources unless you assign it those permissions. This means doing the whole give DOMAINCOMPUTERNAME$ access to your file share and all the OWA servers in your OWA farm. This then implies that any software that is running on your OWA servers will then have access to your file share. If only OWA is installed, then you should be ok, but don't forget about all the other service that are running as NETWORK SERVICE on the services applet. I'm not a big fan of this and in my case, I made it a specifc OWA domain account that has at least read access to the shares that contain your data. NOTE: This change is not supported by Microsoft, but let's be clear about what unsupported means. There are two types of unsupported features, ones they don't have the scripts for at the first level of support, and the ones they do have at second and third levels of support. In my eyes, this falls into the first category and is a simple change but they have not trained anyone on how to do this or troubleshoot it, so this is at your own discretion. So why does OWA work like this? The reason lies in the way that Office Web Apps must access a file
in order to render it. When it is stored in SharePoint, SharePoint
will pass an OAuth access token that OWA can use to access the file as the
user. This will always ensure that OWA accesses a file that the user can access. When accessing through the OpenFromUrl means, it has to access it
as itself using regular windows auth. This has some security implications. Search does only show files the user has access too, so that isn't really a security hole when using the method below, but where things do get interesting if a savvy user figures out how to construct the web page viewer url to a file they don't have access too. This can elevate their privileges and allow them to look at a file but not change it. In this case, you should place a "Deny" ACL on the OWA account (whether domain or network service) to prevent it from reading the document.
If you aren't comfortable in making these changes, then don't. You can take the also not so great approach of simply copying your file share (all 5TB of it) into SharePoint to gain the functionality.
Once that is done, you should now be able to see your documents open in the embedded link generated:
So now that, that works. How do I get SharePoint Search to open the file as a preview? Hmmm…tricky. Let's look at how it does files that live in SharePoint. In order to do this, we have to open the search display templates for Office documents. The first one is for Word documents and it is called "Item_Word_HoverPanel.html" (yeah, they all have a different display template so you'll have to repeat the steps for each). You will notice some JavaScript that looks for a specific property ("ServerRedirectedEmbedURL"):
If this property exists (it is only populated by search if the content lives in SharePoint and is of a specific file type), then it will render the Office Web Apps preview area in the callout. In order to get OWA to work for fileshare files, we have to add the url that renders the link to the browser view or the embedded iframe. I wasn't able to get the iframe to work, but I did get a link that users can click on to get another browser to open with the file. You can do this by modifying the file to have the following:
<!–#_
var i = 0;
var wacurlExist = !Srch.U.e(ctx.CurrentItem.ServerRedirectedURL) && !Srch.U.e(ctx.CurrentItem.ServerRedirectedEmbedURL);
var id = ctx.CurrentItem.csr_id;
ctx.CurrentItem.csr_FileType = Srch.Res.file_Word;
ctx.CurrentItem.csr_ShowFollowLink = true;
ctx.CurrentItem.csr_ShowViewLibrary = true;
ctx.currentItem_IsOfficeDocument = true;
var find = '/';
var re = new RegExp(find, 'g');
function replaceAll(find, replace, str) {
return str.replace(new RegExp(find, 'g'), replace);
}
_#–>
<div class="ms-srch-hover-innerContainer ms-srch-hover-wacSize" id="_#= $htmlEncode(id + HP.ids.inner) =#_">
<div class="ms-srch-hover-arrowBorder" id="_#= $htmlEncode(id + HP.ids.arrowBorder) =#_"></div>
<div class="ms-srch-hover-arrow" id="_#= $htmlEncode(id + HP.ids.arrow) =#_"></div>
<div class="ms-srch-hover-content" id="_#= $htmlEncode(id + HP.ids.content) =#_" data-displaytemplate="WordHoverPanel">
<div id="_#= $htmlEncode(id + HP.ids.header) =#_" class="ms-srch-hover-header">
_#= ctx.RenderHeader(ctx) =#_
</div>
<div id="_#= $htmlEncode(id + HP.ids.body) =#_" class="ms-srch-hover-body">
<!–#_
if ((ctx.CurrentItem.FileType == "docx") && Srch.U.n(ctx.CurrentItem.ServerRedirectedEmbedURL))
{
ctx.CurrentItem.csr_DataShown = true;
ctx.currentItem_ShowChangedBySnippet = true;
_#–>
<a href="https://svr-owa.contosocom/op/view.aspx?src=_#= $urlHtmlEncode(ctx.CurrentItem.Path.replace('file:','').replace(re,'%5C')) =#_" target="_blank">View File</a>
<!–#_
}
_#–>
<!–#_
if(!Srch.U.n(ctx.CurrentItem.ServerRedirectedEmbedURL))
{
ctx.CurrentItem.csr_DataShown = true;
ctx.currentItem_ShowChangedBySnippet = true;
_#–>
<div class="ms-srch-hover-viewerContainer">
<iframe id="_#= $htmlEncode(id + HP.ids.viewer) =#_" src="_#= $urlHtmlEncode(ctx.CurrentItem.ServerRedirectedEmbedURL) =#_" scrolling="no" frameborder="0px" class="ms-srch-hover-viewer"></iframe>
</div>
<div class="ms-srch-hover-wacImageContainer">
<img id="_#= $htmlEncode(id + HP
.ids.preview) =#_" alt="_#= $htmlEncode(Srch.Res.item_Alt_Preview) =#_" onload="this.parentNode.style.display='block';" />
</div>
<!–#_
}
else
{
ctx.CurrentItem.csr_ShowLastModifiedTime = true;
ctx.CurrentItem.csr_ShowAuthors = true;
}
if(!Srch.U.e(ctx.CurrentItem.SectionNames))
{
ctx.CurrentItem.csr_DataShown = true;
_#–>
<div class="ms-srch-hover-subTitle"><h3 class="ms-soften">_#= $htmlEncode(Srch.Res.hp_SectionHeadings) =#_</h3></div>
<!–#_
var sectionNames = Srch.U.getArray(ctx.CurrentItem.SectionNames);
var sectionIndexes = Srch.U.getArray(ctx.CurrentItem.SectionIndexes);
if(!Srch.U.n(sectionIndexes) && sectionIndexes.length != sectionNames.length)
{
sectionIndexes = null;
}
var hitHighlightedSectionNames = Srch.U.getHighlightedProperty(id, ctx.CurrentItem, "sectionnames");
if(!Srch.U.n(hitHighlightedSectionNames) && hitHighlightedSectionNames.length != sectionNames.length)
{
hitHighlightedSectionNames = null;
}
var numberOfSectionsToDisplay = Math.min(Srch.SU.maxLinesForMultiValuedProperty, sectionNames.length);
var sectionsToDisplay = new Array();
var usingHitHighlightedSectionNames = Srch.SU.getSectionsForDisplay(
hitHighlightedSectionNames,
numberOfSectionsToDisplay,
sectionsToDisplay);
for(i = 0; i < sectionsToDisplay.length; ++i)
{
var index = sectionsToDisplay[i];
if(Srch.U.n(index))
{
continue;
}
var tooltipEncoded = $htmlEncode(sectionNames[index]);
var htmlEncodedSectionName = "";
if(usingHitHighlightedSectionNames)
{
htmlEncodedSectionName = hitHighlightedSectionNames[index];
}
else
&nbs
p; {
htmlEncodedSectionName = tooltipEncoded;
}
_#–>
<div class="ms-srch-hover-text ms-srch-ellipsis" id="_#= $htmlEncode(id + HP.ids.sectionName + i) =#_" title="_#= tooltipEncoded =#_">
<!–#_
if(!Srch.U.n(sectionIndexes) && sectionIndexes.length >= i && !Srch.U.e(sectionIndexes[index]) && wacurlExist)
{
var encodedSectionIndex = "&wdparaid=" + $urlKeyValueEncode(sectionIndexes[index]);
_#–>
<a clicktype="HoverSection" linkIndex="_#= $htmlEncode(i) =#_" href="_#= $urlHtmlEncode(ctx.CurrentItem.ServerRedirectedURL + encodedSectionIndex) =#_" target="_blank">
_#= htmlEncodedSectionName =#_
</a>
<!–#_
}
else
{
_#–>
_#= htmlEncodedSectionName =#_
<!–#_
}
_#–>
</div>
<!–#_
}
}
_#–>
_#= ctx.RenderBody(ctx) =#_
</div>
<div id="_#= $htmlEncode(id + HP.ids.actions) =#_" class="ms-srch-hover-actions">
_#= ctx.RenderFooter(ctx) =#_
</div>
</div>
<!–#_
if(!Srch.U.n(ctx.CurrentItem.ServerRedirectedEmbedURL)){
AddPostRenderCallback(ctx, function(){
HP.loadViewer(ctx.CurrentItem.id, ctx.CurrentItem.id + HP.ids.inner, ctx.CurrentItem.id + HP.ids.viewer, ctx.CurrentItem.id + HP.ids.preview, ctx.CurrentItem.ServerRedirectedEmbedURL, ctx.CurrentItem.ServerRedirectedPreviewURL);
});
}
_#–>
</div>
This will get the "View File" link to display for the file. You can then click on it to have the file open in a new browser window and the users can now view the files!
Enjoy!
Chris
In one of my previous posts, I came to an unlikely conclusion that Office Clients don't support delta updates as has been so widely marketed. This was because on every single client I had tested it on, never did it make such an intelligent call. Since I have two of four chapters done for our upcoming book, I had some time and decided to install windows 8 on a 120GB SSD drive on my laptop. After doing so, I also decided to install Office 2013 to see what the experience would be like. Not bad, a lot of re-learning to do, but hey, I'm an MCT, we love learning new stuff. Sooo, I thought, let's try my previous tests that have subsequently failed so many times on this freshly installed OS and Office Client. To my surprise, something different happened when saving and editing SharePoint stored Office XML documents.
The reason the previous Office clients I was working didn't do the delta updates is because they failed to make the proper calls to cellstorage.svc. The HTTP request headers appear to be correct, but it doesn't send a body, which is the most important part of the request. When this happens, it falls back to "FULL" update mode and every time you make a change, the full file is sent using an HTTP PUT request (no call to cellstorage.svc is made after the first failure). Delta updates only support Office XML documents. The reason they are only Office XML is that deep inside cellstorage it makes a call to create a pointer to an XmlReader object. Obviously, the older files are not based on this file format and an attempt to read them is futile, therefore, you won't have any calls to cellstorage.svc, but simply the regular HTTP PUT calls.
Again, until today I was not able to get it to work. But now is a different story. As far as what happens when it does work, this POST will help you. I am working with several others to figure out why my other clients did not work (more to come later).
CellStorage supports various different command types (not sure the entire set, but this is a lot of them):
There is a standard process to this:
During this entire process, the client will ping the sharedaccess.asmx web service to ensure that it is the only one editing the document. This is done about every 20 seconds. As part of the request, it is looking for the ETag to change. If it has changed, that means someone updated the document and the version you have is now old and you will need to refresh your copy, or overwrite what they did. This scenario should never happen, but it looks like they attempt to check that it somehow does (it is MicrosoftSharePoint right?).
Some interesting facts:
The following APPENDIX is a series of CellStorage calls for each of the steps above:
APPENDIX:
The first request (Doc MetaInfo) looks like this:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<RequestVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<RequestCollection CorrelationId="{5E1AF699-7B07-4FF3-ADCA-16E1E40201CC}" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Request Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" UserAgent="{1984108C-4B93-4EEB-B320-919432D6E593}" UserAgentClient="msword" UserAgentPlatform="win" Build="15.0.4420.1017" MetaData="1031" RequestToken="1">
<SubRequest Type="GetDocMetaInfo" SubRequestToken="1"/>
<SubRequest Type="WhoAmI" SubRequestToken="2"/>
</Request>
</RequestCollection>
</s:Body>
</s:Envelope>
The response is this:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<ResponseVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<ResponseCollection WebUrl="http://www.sanspug.org" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Response Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" RequestToken="1" HealthScore="0">
<SubResponse SubRequestToken="1" ErrorCode="Success" HResult="0">
<SubResponseData>
<DocProps>
<Property Key="vti_internalversion" Value="513"/>
…
</DocProps>
<FolderProps>
<Property Key="vti_hassubdirs" Value="true"/>
…</FolderProps>
</SubResponseData>
</SubResponse>
<SubResponse SubRequestToken="2" ErrorCode="Success" HResult="0">
<SubResponseData UserName="Chris Givens" UserLogin="i:0#.w|chrisgivensadministrator" UserEmailAddress="givenscj@hotmail.com" UserSIPAddress=""/>
</SubResponse>
</Response>
</ResponseCollection>
</s:Body>
</s:Envelope>
The second request (get doc parts) is like this:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<RequestVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<RequestCollection CorrelationId="{459EA7C7-7B07-4FF3-ADCA-16E1E40201CC}" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Request Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" UserAgent="{1984108C-4B93-4EEB-B320-919432D6E593}" UserAgentClient="msword" UserAgentPlatform="win" Build="15.0.4420.1017" MetaData="1031" RequestToken="1">
<SubRequest Type="ServerTime" SubRequestToken="1"/>
<SubRequest Type="Cell" SubRequestToken="2">
<SubRequestData PartitionID="383adc0b-e66e-4438-95e6-e39ef9720122" BinaryDataSize="103">DQALAJzPKfM5lAabBgIAAO4CAACqAiAAjBCEGZNL606zIJGUMtblk1oEFgANbXN3b3JkB3dpbnoCCABFFOUPdwEWAgYAAwUAigICAADaAgYAAwAAygIIAAgAgAOEAEELAawCAFUDAQ==</SubRequestData>
</SubRequest>
<SubRequest Type="Cell" SubRequestToken="3">
<SubRequestData GetFileProps="true" BinaryDataSize="103">DQALAJzPKfM5lAabBgIAAO4CAACqAiAAjBCEGZNL606zIJGUMtblk1oEFgANbXN3b3JkB3dpbnoCCABFFOUPdwEWAgYAAwUAigICAADaAgYAAwAAygIIAAgAgAOEAEELAawCAFUDAQ==</SubRequestData>
</SubRequest>
<SubRequest Type="Cell&quo
t; SubRequestToken="4">
<SubRequestData PartitionID="7808f4dd-2385-49d6-b7ce-37aca5e43602" BinaryDataSize="103">DQALAJzPKfM5lAabBgIAAO4CAACqAiAAjBCEGZNL606zIJGUMtblk1oEFgANbXN3b3JkB3dpbnoCCABFFOUPdwEWAgYAAwUAigICAADaAgYAAwAAygIIAAgAgAOEAEELAawCAFUDAQ==</SubRequestData>
</SubRequest>
<SubRequest Type="Cell" SubRequestToken="5">
<SubRequestData BinaryDataSize="88">DQALAJzPKfM5lAabBgIAAO4CAACqAiAAjBCEGZNL606zIJGUMtblk1oEFgANbXN3b3JkB3dpbnoCCABFFOUPdwEWAgYAAxcAAgQIAAQ1DAALAawCAFUDAQ==</SubRequestData>
</SubRequest>
<SubRequest Type="GetVersions" SubRequestToken="6"/>
</Request>
</RequestCollection>
</s:Body>
</s:Envelope>
The response is this (but also included the binary that was requested in the request after the XML part of the response):
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<ResponseVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<ResponseCollection WebUrl="http://www.sanspug.org" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Response Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" RequestToken="1" HealthScore="0">
<SubResponse SubRequestToken="1" ErrorCode="Success" HResult="0">
<SubResponseData ServerTime="634934067280000000"/>
</SubResponse>
<SubResponse SubRequestToken="2" ErrorCode="Success" HResult="0">
<SubResponseData>DAALAJ3PKfM5lAabFgMCAACsAgAMWkCfInndLk8SpahJoDsL3N5J72OAed0uTxKlqEmgOwvc3knvYz9FAAAAAAAAA4gEAAAFVQ4CBgADBQD6AigAQJ8ied0uTxKlqEmgOwvc3knvYwCEACYCIAATHwkQgsj7QJiGZTP5NMIdbAFwWQwcuPLQK0r+TVOn9MDyqn7BNTEAEFAFZRBVAGgpfUBUooP7XAJQAFAAUAAAcLkMWcPYYK6I7UOVZMRDNVJSnZUxADtQMYDEgGKDDF05g8hQM1MOXDSAwFADUwxcMHzIZDZmzFw4iMSEYoONXTF90Ggmc01bYYWEXdZSzWhico1lY1ZNaTh1wVAAALUTASYCIAAO6XY6MoAMTbnd88ZQKUM+TAEgJgwcuPLQK0r+TVOn9MDyqn7BAwClEwFBBwGLAQ==</SubResponseData></SubResponse><SubResponse SubRequestToken="3" ErrorCode="Success" HResult="0">
<SubResponseData Etag=""{2EC13881-6761-4F26-A5AC-4CBBC5F4E8E0},2"" CreateTime="129677043240000000" HaveOnlyDemotionChanges="False" LastModifiedTime="129677044800000000" ModifiedBy="System Account">
<xop:Include href="cid:http://tempuri.org/1/634933491321512406" xmlns:xop="http://www.w3.org/2004/08/xop/include"/>
</SubResponseData>
</SubResponse>
<SubResponse SubRequestToken="4" ErrorCode="Success" HResult="0">
<SubResponseData>DAALAJ3PKfM5lAabFgMCAACsAgAMWkCjInndLk8SpahJoDsL3N5J72OAed0uTxKlqEmgOwvc3knvY0dFAAAAAAAAA4gEAAAFVQ4CBgADBQD6AigAQKMied0uTxKlqEmgOwvc3knvYwCEACYCIAAO6XY6MoAMTbnd88ZQKUM+TAEgJgwcuPLQK0r+TVOn9MDyqn7BAwAgJgyZIgShWw0pQpGd0oV6Kr7gAAClEwEmAiAAEx8JEILI+0CYhmUz+TTCHWwBcFkMHLjy0CtK/k1Tp/TA8qp+wTUxABBQBWUQVQBoKX1AVKKD+1wCUABQAFAAAHC5DFnD2GCuiO1DlWTEQzVSUp2VMQA7UDGAxIBigwxdOYPIUDNTDlw0gMBQA1MMXDB8yGQ2ZsxcOIjEhGKDjV0xfdBoJnNNW2GFhF3WUs1oYnKNZWNWTWk4dcFQAAC1EwFBBwGLAQ==</SubResponseData></SubResponse><SubResponse SubRequestToken="5" ErrorCode="Success" HResult="0">
<SubResponseData HaveOnlyDemotionChanges="False">DAALAJ3PKfM5lAabFgMCAACsAgBVDgIGAAMXAAoEKgBUqbSHwoLARZ4MOEP3qCkZ6gQENQwHAYsB</SubResponseData></SubResponse><SubResponse SubRequestToken="6" ErrorCode="Success" HResult="0">
<GetVersionsResponse>
<GetVersionsResult>
<results>
<list id="{713BA73C-EFCD-49C9-B23D-8A2106868919}"/>
<versioning enabled="0"/>
<settings url="http://www.sanspug.org/_layouts/15/LstSetng.aspx?List={713BA73C-EFCD-49C9-B23D-8A2106868919}"/>
<result version="@1.0" url="http://www.sanspug.org/Presentations/Safety Plan Template.docx" created="12/6/2011 8:08 PM" createdRaw="2011-12-07T04:08:00Z" createdBy="SHAREPOINTsystem" createdByName="System Account" size="30106" comments=""/>
</results>
</GetVersionsResult>
</GetVersionsResponse>
</SubResponse>
</Response>
</ResponseCollection>
</s:Body>
</s:Envelope>
The request that drives the saving of a part back is (also included is a multi part binary that has the changes):
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<RequestVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<RequestCollection CorrelationId="{71E42242-7B07-4FF3-ADCA-16E1E40201CC}" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Request Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" UserAgent="{1984108C-4B93-4EEB-B320-919432D6E593}" UserAgentClient="msword" UserAgentPlatform="win" Build="15.0.4420.1017" MetaData="7" RequestToken="1">
<SubRequest Type="Coauth" SubRequestToken="1">
<SubRequestData CoauthRequestType="RefreshCoauthoring" SchemaLockID="29358EC1-E813-4793-8E70-ED0344E7B73C" ClientID="{7D99133D-52B2-44E4-BCFC-A56DBD9BE639}" Timeout="3600"/>
</SubRequest>
<SubRequest Type="SchemaLock" SubRequestToken="2" DependsOn="1" DependencyType="OnNotSupported">
<SubRequestData SchemaLockRequestType="RefreshLock" SchemaLockID="29358EC1-E813-4793-8E70-ED0344E7B73C" ClientID="{7D99133D-52B2-44E4-BCFC-A56DBD9BE639}" Timeout="3600"/>
</SubRequest>
<SubRequest Type="Cell" SubRequestToken="3" DependsOn="2" DependencyType="OnSuccessOrNotSupported">
<SubRequestData Coalesce="true" CoauthVersioning="true" GetFileProps="true" BypassLockID="29358EC1-E813-4793-8E70-ED0344E7B73C" SchemaLockID="29358EC1-E813-4793-8E70-ED0344E7B73C" BinaryDataSize="32021">
<i:Include xmlns:i="http://www.w3.org/2004/08/xop/include" href="cid:820fe252-7a34-4b40-802f-e666a88a8941-0@tempuri.org"/>
</SubRequestData>
</SubRequest>
</Request>
</RequestCollection>
</s:Body>
</s:Envelope>
The file update response is:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<ResponseVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<ResponseCollection WebUrl="http://www.sanspug.org" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Response Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" RequestToken="1" HealthScore="0">
<SubResponse SubRequestToken="1" ErrorCode="Success" HResult="0">
<SubResponseData LockType="SchemaLock" CoauthStatus="Alone"/>
</SubResponse>
<SubResponse SubRequestToken="2" ErrorCode="DependentOnlyOnNotSupportedRe
questGetSupported" HResult="2147500037">
<SubResponseData/>
</SubResponse>
<SubResponse SubRequestToken="3" ErrorCode="Success" HResult="0">
<SubResponseData Etag=""{2EC13881-6761-4F26-A5AC-4CBBC5F4E8E0},3"" HaveOnlyDemotionChanges="False">DAALAJ3PKfM5lAabFgMCAACsAgBVDgIGAAMLADoEBAAAAIQAJgIgAPY1ejJhBxREloZR6QBmek2kAHgmYSrmvvwoD0GvMdRy1rRs+wDyCHgmntUZQQPX8L5QziuNKUuTBADyCFETASYCIAAO6XY6MoAMTbnd88ZQKUM+TAEgKAx+xz7RnpjZsFpTs0Q6CxcfpgwAICgUfsc+0Z6Y2bBaU7NEOgsXH6YMAKUTASYCIAATHwkQgsj7QJiGZTP5NMIdbAFwSQykwL9CeUV1ACf0G+m6m5XCJTEACVAFdAJw13VWUwNQAFAAALUTAUFKBAIAAAcBiwE=
</SubResponseData>
</SubResponse>
</Response>
</ResponseCollection>
</s:Body>
</s:Envelope>
Closing the session request is:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<RequestVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<RequestCollection CorrelationId="{C2ADFD14-7B07-4FF3-ADCA-16E1E40201CC}" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Request Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" UserAgent="{1984108C-4B93-4EEB-B320-919432D6E593}" UserAgentClient="msword" UserAgentPlatform="win" Build="15.0.4420.1017" MetaData="1031" RequestToken="1">
<SubRequest Type="Coauth" SubRequestToken="1">
<SubRequestData CoauthRequestType="ExitCoauthoring" SchemaLockID="29358EC1-E813-4793-8E70-ED0344E7B73C" ClientID="{7D99133D-52B2-44E4-BCFC-A56DBD9BE639}"/>
</SubRequest>
<SubRequest Type="SchemaLock" SubRequestToken="2" DependsOn="1" DependencyType="OnNotSupported">
<SubRequestData SchemaLockRequestType="ReleaseLock" SchemaLockID="29358EC1-E813-4793-8E70-ED0344E7B73C" ClientID="{7D99133D-52B2-44E4-BCFC-A56DBD9BE639}"/>
</SubRequest>
</Request>
</RequestCollection>
</s:Body>
</s:Envelope>
The response is:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><ResponseVersion Version="2" MinorVersion="2" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
<ResponseCollection WebUrl="http://www.sanspug.org" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<Response Url="http://www.sanspug.org/Presentations/Safety%20Plan%20Template.docx" RequestToken="1" HealthScore="0">
<SubResponse SubRequestToken="1" ErrorCode="Success" HResult="0">
<SubResponseData/>
</SubResponse>
<SubResponse SubRequestToken="2" ErrorCode="DependentOnlyOnNotSupportedRequestGetSupported" HResult="2147500037">
<SubResponseData/>
</SubResponse>
</Response>
</ResponseCollection>
</s:Body>
</s:Envelope>
So as some of you know, I'm on each of the MSPress IT Pro books that will be released on SharePoint 2013. In one of the books, I'm talking about the architecture and how the content databases have changed. One of those changes is Shredded Storage. Yes, I have read Bill Baer's post here and he has a second one here, and I agree with everything he says (even though he doesn't really use the internal architecture's terminology in several cases), except for one point which I bring up at the end. I have also read Dan Holme's blog here. He pretty much comes to the same conclusions that I do. He even nailed it with the max 64K chuck size as the default (but the files don't max at the 64K limit, they end up somewhere below it).
So, even after reading all this technet and SPC12 mumbo jumbo. I really wanted to see this stuff working for myself and that required digging in deep to learn how it really works given all the blogs that are incorrect. In that exploration of the assemblies, classes, tables and stored procedures I have had a few ah ha moments. Some facts:
UPDATE: See this blog post for my more recent experiences with editing with Office Clients and intelligent updating (it does work)…
Back to one of the above points. I was not able to get the Office clients (2010 or 2013) to do any type of smart updating as mentioned in Bill's post (with Word and PowerPoint with a particular client build). In other wards, I only wanted to send the changes I had made to SharePoint. I had never really tested this before and was looking forward to seeing it in action, but alas, it doesn't seem to do it at all. I'm guessing only in a multi-user editing mode (OWA?) will you see this type of feature being utilized (UPDATE: and this guess was correct! See below!). Just as an FYI, here's what I did with Office and SharePoint OM calls:
If anyone knows how to get this working reliably with Office Clients and SharePoint OM, please let me know. Otherwise, I'm going to have to say we are getting dupped on this whole "delta" changes from office client non-sense and there is no network optimization going on between Office Client and WFE. In this scenario, Shredded storage is really just saving us a few bytes here and there (if versioning is turned on) which does reduce the number of writes, but at the cost of more "reads" and CPU to rebuild the files.
UPDATE: Office Web Apps and Shredded
Storage – this is where you will see the wire optimization between the client (OWA) and the WFE. It works like this. When a request for a file is made from SharePoint, OWA will ask the WFE to give it the file. The file will be built by the WFE from the shreds. When two users open the file for editing, a new type of shred container is created called a "partition". This partition contains shreds that each users is working on. This is where the shreds get broken down to their smaller XML pieces. As each part of a document is changed, new partitions are created. When someone wants to see what another person has done, they will request this new partition and it (and only it) will get sent to the client. Any changes that are made are also sent singly and the entire file is not.
So where does that put us? Here is the reality and correct details about Shredded Storage (as every blog on Shredded Storage is wrong on the internet). You have to test shredded storage with 3 things in mind to see if you really are getting any benefits (whether storage or network based).
If you want more information at a super deep level, buy our MS Press book in October!
Enjoy!
Chris
Lovely error right? You can spend a lifetime trying to figure out why you can't get rid of this error. But that's what community is for! After writing my own OAuth apps to hit the SharePoint AppsSTS (appssts.svc), I finally know why this error occurs!
Turns out that O365 SharePoint Farms have a special Service Application installed. This service application is not available to us mere mortals with Beta version of SP2013 running on-premise. The service application is called "Azure Access Control Service Application". It has a corresponding Proxy that goes with it. When your SharePoint App code executes, it makes a call to this ACS proxy. If it doesn't find the existance of one of them in your Farm, it fails with a null reference, and then you get the "The Azure Access Control service is unavailable." error.
What does the proxy do? It actually will grab the Azure EndPoint for OAuth2 and do the delegation calls. You can see this xml document here:
https://accounts.accesscontrol.windows.net/metadata/json/1
Now you may be thinking…can I trick it? Well, even if you could (which you can't), notice the "Location" fields in the SharePoint endpoint below. Doing the same endpoint call in SharePoint, you will see that those "location" fields are not populated so you couldn't do it anyway!
http://intranet.contoso.com/_layouts/15/metadata/json/1
Nothing you can do about it with an onpremise installation of SharePoint 2013. Whatever app you built, something in it is wanting to be deployed to O365. This tells you that Microsoft is driving you to it by leaving out real OAuth endpoints in SP2013 in Beta. You can find several differnt ways to get an access token in the beta but none of them really support the notion of a clientid and secret when working with on-premise. Just FYI, one way (that is as close to OAuth as you can get) is particuarlly bad as it doesn't have any auth on it and I can get auth tokens all day long!!!
All that being said, there are hints of the ablity to do this in RTM. You can find several layouts pages (AppRegNew.aspx) dedicated to creating applications that live on-premise and use real OAuth calls (clientid and secret). However, sad to say, nothing really works at the moment in the beta (believe me…I have tried). The fact that the code is even exposed about the ACS Service Application is because there is a generic interface that can be replaced in later code to make calls to other services, however, it looks like they just didn't get around to putting all the plumbing in yet.
So, don't fret about the above error, you can't do anything about it with on-premise apps.
Chris
Yeah, lots of people are using OAuth. Its evolution from WRAP->OAuth 1.0 has in my opinion been mostly good. However, there are always the details of how something is built that makes you realize, not everything is what it seems to be:
http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/
After reading the article…go to the last part…I tweeted that all not too long ago. I did the same for REST in SharePoint 2010.
You have to wonder. Who is the "Enterprise" in this case? Hint…My guess is the name starts with an "M" and ends in "T"
Chris
If you run across this error you will find it in the event log when you try to execute a call to the PowerPoint Automation Service Application for a conversion request. The error occurs because the application pool account that the service is running under doesn't have the needed permissions (ie, they weren't setup by SharePoint properly in Beta) to create directories here:
C:ProgramDataMicrosoftSharePoint
If you look at the permissions on the directory, you see that Machine Administrators can do anything they like on this directory. Bummer. You now have to add the service account (or change the application pool to some other account that you already have given this access…like the Farm Account) for the PowerPoint Automation Service Application into the local machine administrators group. That seems a bit much don't you think?
Hopefully it will be fixed in RTM,
Chris